SizeTieredCompactionStrategy for General Write-Oriented Workloads

Learning outcomes

AtlasMart inherits a large table created years ago with SizeTieredCompactionStrategy (STCS). Writes are smooth, but the team sees bursts of disk use and a growing number of similarly sized SSTables during peak ingestion. Before migrating anything, they need to understand exactly how STCS groups files and why its read/space behavior emerges.

Compaction terms before changing a table

An SSTable (Sorted String Table) is immutable on-disk data. A memtable flush creates a new SSTable; updates and deletes therefore create newer versions instead of modifying older files in place. Compaction selects SSTables, reads their sorted contents, reconciles newer cells and tombstones, writes replacement SSTables, and retires input files after active readers no longer need them. Read amplification is the extra storage work a logical read performs across files/versions. Write amplification is the number of bytes rewritten in the background relative to application bytes written. Space amplification is temporary or steady-state disk usage above live logical data, including overlapping SSTables, replacement outputs, snapshots, streaming, repair and backup staging. A tombstone is a timestamped deletion/expiration marker; compaction can purge it only when Cassandra can do so safely. Compaction debt means eligible/pending work is accumulating faster than it is completed. UCS means UnifiedCompactionStrategy; STCS, LCS and TWCS mean SizeTiered, Leveled and TimeWindow compaction strategies respectively.

1. STCS mechanism

STCS groups SSTables into size buckets. When enough similarly sized SSTables are available—commonly governed by min_threshold and max_threshold—it merges a subset into a larger SSTable. The strategy tends to write a generation a relatively small number of times compared with leveled approaches, which is attractive for heavy writes, but reads can face several overlapping files. Temporary disk demand can also be high because input SSTables remain until the output is safely installed.

2. Build a deterministic STCS queue

docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 java -versiondocker exec atlasmart-cass-1 nodetool getcompactionthroughputdocker exec atlasmart-cass-1 nodetool getconcurrentcompactorsdocker exec atlasmart-cass-1 nodetool compactionstatsdocker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra"

CREATE KEYSPACE IF NOT EXISTS atlasmart_compactionWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CONSISTENCY LOCAL_QUORUM;

CREATE TABLE IF NOT EXISTS atlasmart_compaction.compaction_stcs (  bucket text, event_time timestamp, event_id uuid, payload text,  PRIMARY KEY (bucket,event_time,event_id)) WITH compaction = {'class':'SizeTieredCompactionStrategy','min_threshold':'4','max_threshold':'32'};

docker exec atlasmart-cass-1 nodetool disableautocompaction atlasmart_compaction compaction_stcsdocker exec atlasmart-cass-2 nodetool disableautocompaction atlasmart_compaction compaction_stcsdocker exec atlasmart-cass-3 nodetool disableautocompaction atlasmart_compaction compaction_stcs

INSERT INTO atlasmart_compaction.compaction_stcs VALUES ('b','2026-09-07T18:00:00Z',00000000-0000-0000-0000-000000000001,'generation-1');

for i in 1 2 3 4; do  ID=$(printf "%012d" $i)  docker exec atlasmart-cass-1 cqlsh -e "INSERT INTO atlasmart_compaction.compaction_stcs (bucket,event_time,event_id,payload) VALUES ('b','2026-09-07T18:0${i}:00Z',00000000-0000-0000-0000-${ID},'g${i}');"  for n in 1 2 3; do docker exec atlasmart-cass-$n nodetool flush atlasmart_compaction compaction_stcs; donedonedocker exec atlasmart-cass-1 nodetool tablestats atlasmart_compaction.compaction_stcsdocker exec atlasmart-cass-1 nodetool compactionstats

3. Re-enable and observe selection

for n in 1 2 3; do docker exec atlasmart-cass-$n nodetool enableautocompaction atlasmart_compaction compaction_stcs; donefor i in 1 2 3 4 5; do  docker exec atlasmart-cass-1 nodetool compactionstats  sleep 2donedocker exec atlasmart-cass-1 nodetool compactionhistorydocker exec atlasmart-cass-1 nodetool tablestats atlasmart_compaction.compaction_stcsdocker exec atlasmart-cass-1 nodetool tablehistograms atlasmart_compaction compaction_stcs

On a tiny cluster, background compaction may finish before the first poll. That is not a failed lab: compare history and SSTable counts. On production-scale files, selection and rewrite can be long-running, and the critical signal is whether pending work grows faster than throughput clears it.

4. STCS tradeoffs versus UCS

5. Verification and cleanup

• Auto-compaction is re-enabled on all three nodes.
• The table schema still explicitly says STCS.
• Capture before/after SSTable counts, compaction history and disk free space.
• Do not change global thresholds/throughput as part of this lesson.

Production judgment

Compaction tuning is capacity engineering, not a table-property beauty contest. Evaluate application write rate, overwrite/delete/TTL rate, partition and clustering distribution, read/write p50/p95/p99 latency, SSTables per read, compression ratio, pending compactions, bytes compacted, compaction throughput, disk queue/throughput, CPU, JVM/GC, network, repair and streaming load, snapshots, SAI/vector index amplification, and free-space trajectory. RF and consistency level change how many replicas incur the physical work; a locally fast compaction configuration can still violate a cluster SLO under repair, node replacement, backup, or failure.

Every production change needs an acceptance window and rollback plan. Record the old schema/options, node-by-node rollout order, expected rewrite volume, required free space, compaction backlog limit, latency/error SLOs, and stop conditions. Managed Cassandra services may hide throughput/concurrency controls or select strategies for you; map the same concepts to the provider's exposed metrics rather than assuming identical knobs. Lesson 3 examines LCS, where predictable overlap above L0 is purchased with repeated leveling rewrites and different space/I/O characteristics.

Summary and next bridge

STCS explains many inherited Cassandra fleets: bucket similar sizes, merge opportunistically, accept more overlap in exchange for a write-oriented profile. The right response is measurement, not automatic migration or automatic retention. Next, contrast this with LCS.

Authoritative references

Use these as the version-sensitive source of truth when regenerating the lesson; compaction recommendations and options can evolve between Cassandra releases.

• Apache Cassandra releases
• Compaction overview and strategies
• Compaction CQL subproperties including UCS
• ALTER TABLE and compaction-strategy rewrite warning
• cassandra.yaml compaction throughput/concurrency/defaults
• Storage engine / LSM write amplification
• Repair and replica convergence