LeveledCompactionStrategy for Read Predictability and Its Write / Space Amplification Tradeoffs

Learning outcomes

AtlasMart's product lookup table has a strict p99 read target. An older design selected LeveledCompactionStrategy (LCS) to bound overlapping SSTables, but storage writes are much higher than application writes. The team needs to understand that trade instead of assuming “read-optimized” means universally efficient.

Compaction terms before changing a table

An SSTable (Sorted String Table) is immutable on-disk data. A memtable flush creates a new SSTable; updates and deletes therefore create newer versions instead of modifying older files in place. Compaction selects SSTables, reads their sorted contents, reconciles newer cells and tombstones, writes replacement SSTables, and retires input files after active readers no longer need them. Read amplification is the extra storage work a logical read performs across files/versions. Write amplification is the number of bytes rewritten in the background relative to application bytes written. Space amplification is temporary or steady-state disk usage above live logical data, including overlapping SSTables, replacement outputs, snapshots, streaming, repair and backup staging. A tombstone is a timestamped deletion/expiration marker; compaction can purge it only when Cassandra can do so safely. Compaction debt means eligible/pending work is accumulating faster than it is completed. UCS means UnifiedCompactionStrategy; STCS, LCS and TWCS mean SizeTiered, Leveled and TimeWindow compaction strategies respectively.

1. LCS mechanism: pay writes to bound overlap

Newly flushed SSTables enter level 0 (L0), where overlap is allowed. Compaction moves data into higher levels. Within a level above L0, SSTables are non-overlapping by token range, so a read generally needs at most one SSTable per level for a partition key, plus possible L0 candidates. Level capacity grows by a fanout factor. Maintaining this invariant means data can be rewritten repeatedly as it moves through levels: lower read amplification is purchased with higher write amplification and sustained compaction I/O.

A very large partition is an important boundary: Cassandra does not split one partition across arbitrary SSTables just to honor the configured LCS target size, so simplistic “every SSTable is exactly 160 MB” diagrams are wrong.

2. Create an LCS fixture

docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 java -versiondocker exec atlasmart-cass-1 nodetool getcompactionthroughputdocker exec atlasmart-cass-1 nodetool getconcurrentcompactorsdocker exec atlasmart-cass-1 nodetool compactionstatsdocker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra"

CREATE KEYSPACE IF NOT EXISTS atlasmart_compactionWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CONSISTENCY LOCAL_QUORUM;

CREATE TABLE IF NOT EXISTS atlasmart_compaction.compaction_lcs (  sku text, version_time timestamp, version_id uuid, price decimal, payload text,  PRIMARY KEY (sku,version_time,version_id)) WITH compaction = {'class':'LeveledCompactionStrategy','sstable_size_in_mb':'160'};

for i in 1 2 3 4 5; do  ID=$(printf "%012d" $i)  docker exec atlasmart-cass-1 cqlsh -e "INSERT INTO atlasmart_compaction.compaction_lcs (sku,version_time,version_id,price,payload) VALUES ('sku-42','2026-09-07T18:0${i}:00Z',00000000-0000-0000-0000-${ID},${i}.0,'g${i}');"  for n in 1 2 3; do docker exec atlasmart-cass-$n nodetool flush atlasmart_compaction compaction_lcs; donedonefor i in 1 2 3 4 5; do docker exec atlasmart-cass-1 nodetool compactionstats; sleep 2; donedocker exec atlasmart-cass-1 nodetool compactionhistorydocker exec atlasmart-cass-1 nodetool tablestats atlasmart_compaction.compaction_lcsdocker exec atlasmart-cass-1 nodetool tablehistograms atlasmart_compaction compaction_lcs

3. Read predictability versus write/space amplification

Do not benchmark LCS versus UCS with only five tiny rows. The lab proves mechanics and observability. A meaningful comparison needs representative partition sizes, update rate, read distribution, compression, concurrency and enough duration for the compaction steady state to emerge.

4. Current UCS comparison

UCS can express more tiered or more leveled behavior using scaling parameters and is recommended for most new Cassandra 5.0 workloads. That makes it the first comparison point for a new read-sensitive table. Existing LCS remains valid when its measured behavior, migration rewrite cost and operational history are favorable. Do not translate “UCS recommended” into “ALTER every table now”: current documentation warns that changing the compaction strategy of an existing table can rewrite existing SSTables and take hours.

5. Verification

docker exec atlasmart-cass-1 nodetool tablestats atlasmart_compaction.compaction_lcsdocker exec atlasmart-cass-1 nodetool tablehistograms atlasmart_compaction compaction_lcsdocker exec atlasmart-cass-1 nodetool compactionstatsdocker exec atlasmart-cass-1 nodetool compactionhistorydocker exec atlasmart-cass-1 nodetool getcompactionthroughputdocker exec atlasmart-cass-1 nodetool getconcurrentcompactorsdocker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra && find /var/lib/cassandra/data/atlasmart_compaction -type f -name '*Data.db' -printf '%s %p\n' | sort -n | tail -20"

Production judgment

Compaction tuning is capacity engineering, not a table-property beauty contest. Evaluate application write rate, overwrite/delete/TTL rate, partition and clustering distribution, read/write p50/p95/p99 latency, SSTables per read, compression ratio, pending compactions, bytes compacted, compaction throughput, disk queue/throughput, CPU, JVM/GC, network, repair and streaming load, snapshots, SAI/vector index amplification, and free-space trajectory. RF and consistency level change how many replicas incur the physical work; a locally fast compaction configuration can still violate a cluster SLO under repair, node replacement, backup, or failure.

Every production change needs an acceptance window and rollback plan. Record the old schema/options, node-by-node rollout order, expected rewrite volume, required free space, compaction backlog limit, latency/error SLOs, and stop conditions. Managed Cassandra services may hide throughput/concurrency controls or select strategies for you; map the same concepts to the provider's exposed metrics rather than assuming identical knobs. Lesson 4 shifts to TWCS, where time windows and TTL can make whole expired SSTables disposable—but only if data is mostly immutable and timestamp/retention discipline is strong.

Summary and next bridge

LCS is a deliberate exchange: more background rewrites for constrained file overlap and more predictable reads. It remains an important inherited/specialized strategy, but UCS is the current new-table comparison. Next, examine the very different retention-driven logic of TWCS.

Authoritative references

Use these as the version-sensitive source of truth when regenerating the lesson; compaction recommendations and options can evolve between Cassandra releases.

• Apache Cassandra releases
• Compaction overview and strategies
• Compaction CQL subproperties including UCS
• ALTER TABLE and compaction-strategy rewrite warning
• cassandra.yaml compaction throughput/concurrency/defaults
• Storage engine / LSM write amplification
• Repair and replica convergence
• LeveledCompactionStrategy