Connect SSTable Count / Size to Compaction Strategy, Read Amplification, and Disk Headroom

Learning outcomes

AtlasMart stores 1.2 TiB of live Cassandra data on a node with a 1.5 TiB volume and concludes that 300 GiB free is “25% headroom, so enough.” That percentage is meaningless without knowing SSTable size distribution, compaction strategy, pending compactions, snapshot/backup retention, streaming plans, restore staging, and failure mode. This lesson turns file-level evidence into a capacity decision.

Storage terms before touching the filesystem

An SSTable (Sorted String Table) is Cassandra's immutable on-disk representation produced by a memtable flush, compaction, streaming, bulk load, or related storage workflow. A component is one file belonging to an SSTable generation; component sets and names depend on SSTable format/version. A partition is the rows sharing a partition key and is sorted with other partitions by token order inside SSTable data; rows inside a partition follow clustering order. Compression chunks are independently compressed blocks of the Data component, letting Cassandra read/decompress only relevant chunks instead of the full file. Compaction reads SSTables and writes replacement SSTables, then retires old ones when safe. Streaming transfers replica data between nodes for bootstrap, rebuild, repair, replacement, and topology movement. Disk headroom is free capacity reserved not only for live data but for temporary overlap during these operations. Offline SSTable tools inspect or transform SSTables outside normal CQL/native-protocol execution; many explicitly require Cassandra to be stopped and therefore must never be pointed casually at live production paths.

1. SSTable count is a symptom; overlap and workload determine cost

More SSTables can increase read amplification because a read may need to consider data/tombstones from several immutable generations. But “20 SSTables is bad” is not a universal rule: compaction strategy, overlap, query key presence, Bloom filters, partition shape and repaired state matter. Likewise one huge SSTable can create different concerns around streaming/rewrite duration and large partitions. Measure SSTables per Read, table latency, compaction backlog, file sizes and partition distributions together.

docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 java -versiondocker exec atlasmart-cass-1 sh -lc "grep -n -A8 -B2 '^sstable:' /etc/cassandra/cassandra.yaml || true"docker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra && df -i /var/lib/cassandra"

CREATE KEYSPACE IF NOT EXISTS atlasmart_storageWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_storage.orders_by_customer_day (    customer_id text,    order_day date,    order_time timestamp,    order_id uuid,    status text,    total decimal,    note text,    PRIMARY KEY ((customer_id, order_day), order_time, order_id)) WITH CLUSTERING ORDER BY (order_time DESC, order_id ASC)  AND compaction = {'class':'UnifiedCompactionStrategy'}  AND compression = {'class':'LZ4Compressor','chunk_length_in_kb':'16'};CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_storage.orders_by_customer_day(customer_id,order_day,order_time,order_id,status,total,note)VALUES ('cust-42','2026-09-07','2026-09-07T18:00:00Z',00000000-0000-0000-0000-000000000001,'PAID',129.90,'first');INSERT INTO atlasmart_storage.orders_by_customer_day(customer_id,order_day,order_time,order_id,status,total,note)VALUES ('cust-42','2026-09-07','2026-09-07T18:01:00Z',00000000-0000-0000-0000-000000000002,'PACKING',89.50,'second');INSERT INTO atlasmart_storage.orders_by_customer_day(customer_id,order_day,order_time,order_id,status,total,note)VALUES ('cust-77','2026-09-07','2026-09-07T18:02:00Z',00000000-0000-0000-0000-000000000003,'CREATED',44.00,'third');DESCRIBE TABLE atlasmart_storage.orders_by_customer_day;SELECT * FROM atlasmart_storage.orders_by_customer_dayWHERE customer_id='cust-42' AND order_day='2026-09-07';

docker exec atlasmart-cass-1 nodetool flush atlasmart_storage orders_by_customer_dayfor i in 1 2 3; do  docker exec atlasmart-cass-1 cqlsh -e "CONSISTENCY LOCAL_QUORUM; UPDATE atlasmart_storage.orders_by_customer_day SET note='generation-$i' WHERE customer_id='cust-42' AND order_day='2026-09-07' AND order_time='2026-09-07T18:00:00Z' AND order_id=00000000-0000-0000-0000-000000000001;"  docker exec atlasmart-cass-1 nodetool flush atlasmart_storage orders_by_customer_daydonedocker exec atlasmart-cass-1 nodetool tablestats atlasmart_storage.orders_by_customer_daydocker exec atlasmart-cass-1 nodetool tablehistograms atlasmart_storage orders_by_customer_daydocker exec atlasmart-cass-1 nodetool compactionstatsdocker exec atlasmart-cass-1 sh -lc "find /var/lib/cassandra/data/atlasmart_storage -type f -path '*orders_by_customer_day*' -name '*Data.db' -printf '%s %f\n' | sort -n"

2. Disk accounting has multiple meanings

tablestats reports table/SSTable metrics; du reports filesystem blocks reachable from paths; snapshots often use hard links so “snapshot size” and “additional physical bytes” are not the same; nodetool listsnapshots distinguishes size-on-disk and true disk-space concepts. During compaction, old SSTables coexist with newly written replacements until the operation commits. During streaming or restore, incoming/staged files can coexist with existing data. Therefore a capacity model must state which byte concept it uses.

3. A transparent headroom worksheet

Use a scenario model rather than a magic “keep X% free” rule. Define measured current bytes, plausible simultaneous operations, and a safety/uncertainty reserve. Avoid summing impossible worst cases if they are operationally mutually exclusive; conversely, do not omit combinations your runbook actually permits.

Measured current filesystem used on data volume:          1,200 GiBMeasured free capacity:                                      600 GiBPlanned operation envelope (example only):  largest eligible compaction rewrite still in progress:     180 GiB  snapshot-retained old SSTables during window:               90 GiB  expected bootstrap/repair incoming data on this node:       120 GiB  restore/backup staging allowed concurrently:                  0 GiB  (runbook forbids overlap)  logs/commitlog/other growth allowance during window:         25 GiB  uncertainty / measurement error reserve:                     60 GiB                                                           ---------Scenario temporary requirement:                              475 GiBRemaining free if all allowed overlap occurs:                125 GiBDecision is NOT “125 GiB is enough” by itself.Validate disk throughput, inode space, operation duration, abort/rollback behavior,compaction free-space constraints, and whether emergency growth can be added.

docker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra; df -i /var/lib/cassandra"docker exec atlasmart-cass-1 nodetool tablestats atlasmart_storage.orders_by_customer_daydocker exec atlasmart-cass-1 nodetool compactionstatsdocker exec atlasmart-cass-1 nodetool listsnapshotsdocker exec atlasmart-cass-1 nodetool netstats -Hdocker exec atlasmart-cass-1 sh -lc "du -sh /var/lib/cassandra/data /var/lib/cassandra/commitlog /var/lib/cassandra/saved_caches 2>/dev/null"

4. Controlled before/after compaction evidence

Force compaction only on the disposable fixture to demonstrate temporary rewrite and resulting file consolidation. Capture disk/free space immediately before, during if possible, and after. Tiny data may complete too quickly to observe temporary overlap; report that limitation rather than inventing a spike.

docker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra"docker exec atlasmart-cass-1 nodetool tablestats atlasmart_storage.orders_by_customer_day# Lab only; production manual compaction requires an explicit reason and headroom plan.docker exec atlasmart-cass-1 nodetool compact atlasmart_storage orders_by_customer_daydocker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra"docker exec atlasmart-cass-1 nodetool tablestats atlasmart_storage.orders_by_customer_daydocker exec atlasmart-cass-1 nodetool tablehistograms atlasmart_storage orders_by_customer_day

5. Pre-operation acceptance and rollback gates

Production judgment

SSTable files expose valuable operational evidence, but they are not an application contract. Production decisions must combine logical data shape with physical storage state: partition rows/bytes, mutation and TTL/delete rates, RF/CL, read/write p95/p99, SSTables per read, compaction strategy and backlog, repaired/unrepaired state, compression ratio, CPU/decompression cost, disk throughput/latency, temporary compaction/streaming space, snapshot/backup retention, topology changes, repair cadence, and restore objectives. Include JVM/GC, page cache/off-heap use, driver timeouts/retries/idempotency, network bandwidth, tenant isolation, encryption-at-rest expectations, filesystem/device behavior, and managed-service restrictions.

Do not plan disk capacity as “live dataset bytes × RF” only. Flush, compaction, repair, streaming, snapshots, incremental backups, anti-compaction, restore staging, and operational safety margins can temporarily retain additional SSTables. Do not manually delete or edit component files to recover space. If space is critical, first stop unsafe automation, measure ownership/snapshots/compaction/streaming, and choose a documented recovery path with rollback. Chapter 12 now focuses on compaction itself: why merges exist, how UCS/STCS/LCS/TWCS differ, and how to measure read/write/space amplification without freezing old strategy folklore into defaults.

Summary and next bridge

SSTable count, size and temporary copies translate directly into latency and capacity risk only when combined with compaction, snapshots, streaming, repair and workload evidence. Chapter 12 takes the next step: choose and operate compaction strategies by measured amplification and workload fit.

Authoritative references

These are version-sensitive sources of truth. Re-check them when regenerating the lesson because SSTable formats, utilities, defaults, and topology procedures evolve.

• Apache Cassandra downloads / current GA baseline
• Apache Cassandra storage engine and SSTable components
• Cassandra 5.0 cassandra.yaml SSTable format and streaming settings
• Apache Cassandra compression guidance
• Apache Cassandra SSTable tools safety overview
• sstablemetadata
• sstabledump
• sstablepartitions
• nodetool netstats
• Repair and streaming differences
• nodetool listsnapshots
• Compaction overview