Streaming SSTables During Bootstrap, Repair, Rebuild, and Topology Changes

Learning outcomes

AtlasMart plans to add capacity and assumes “streaming just copies the current dataset once.” In reality bootstrap, rebuild, repair, replacement, and range movement can stream different token ranges, interact with compaction, create temporary disk/network pressure, and leave cleanup work behind. This lesson follows SSTables through those topology workflows without performing a risky full production-like rebalance.

Storage terms before touching the filesystem

An SSTable (Sorted String Table) is Cassandra's immutable on-disk representation produced by a memtable flush, compaction, streaming, bulk load, or related storage workflow. A component is one file belonging to an SSTable generation; component sets and names depend on SSTable format/version. A partition is the rows sharing a partition key and is sorted with other partitions by token order inside SSTable data; rows inside a partition follow clustering order. Compression chunks are independently compressed blocks of the Data component, letting Cassandra read/decompress only relevant chunks instead of the full file. Compaction reads SSTables and writes replacement SSTables, then retires old ones when safe. Streaming transfers replica data between nodes for bootstrap, rebuild, repair, replacement, and topology movement. Disk headroom is free capacity reserved not only for live data but for temporary overlap during these operations. Offline SSTable tools inspect or transform SSTables outside normal CQL/native-protocol execution; many explicitly require Cassandra to be stopped and therefore must never be pointed casually at live production paths.

1. Same transport family, different operational intent

Do not equate “streaming completed” with “cluster is fully repaired.” Streaming is a transport mechanism used by several workflows. Each workflow has its own consistency, ownership, cleanup, and validation semantics.

2. Observe current stream state and configuration before creating traffic

docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 java -versiondocker exec atlasmart-cass-1 sh -lc "grep -n -A8 -B2 '^sstable:' /etc/cassandra/cassandra.yaml || true"docker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra && df -i /var/lib/cassandra"

CREATE KEYSPACE IF NOT EXISTS atlasmart_storageWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_storage.orders_by_customer_day (    customer_id text,    order_day date,    order_time timestamp,    order_id uuid,    status text,    total decimal,    note text,    PRIMARY KEY ((customer_id, order_day), order_time, order_id)) WITH CLUSTERING ORDER BY (order_time DESC, order_id ASC)  AND compaction = {'class':'UnifiedCompactionStrategy'}  AND compression = {'class':'LZ4Compressor','chunk_length_in_kb':'16'};CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_storage.orders_by_customer_day(customer_id,order_day,order_time,order_id,status,total,note)VALUES ('cust-42','2026-09-07','2026-09-07T18:00:00Z',00000000-0000-0000-0000-000000000001,'PAID',129.90,'first');INSERT INTO atlasmart_storage.orders_by_customer_day(customer_id,order_day,order_time,order_id,status,total,note)VALUES ('cust-42','2026-09-07','2026-09-07T18:01:00Z',00000000-0000-0000-0000-000000000002,'PACKING',89.50,'second');INSERT INTO atlasmart_storage.orders_by_customer_day(customer_id,order_day,order_time,order_id,status,total,note)VALUES ('cust-77','2026-09-07','2026-09-07T18:02:00Z',00000000-0000-0000-0000-000000000003,'CREATED',44.00,'third');DESCRIBE TABLE atlasmart_storage.orders_by_customer_day;SELECT * FROM atlasmart_storage.orders_by_customer_dayWHERE customer_id='cust-42' AND order_day='2026-09-07';

docker exec atlasmart-cass-1 nodetool netstats -Hdocker exec atlasmart-cass-1 sh -lc "grep -n -E 'stream_entire_sstables|stream_throughput|inter_dc_stream' /etc/cassandra/cassandra.yaml || true"docker exec atlasmart-cass-1 nodetool getstreamthroughput 2>/dev/null || truedocker exec atlasmart-cass-1 sh -lc "df -h /var/lib/cassandra"docker exec atlasmart-cass-1 nodetool compactionstats

If no topology/repair operation is active, netstats should show no current sending/receiving sessions. This is a baseline, not a guarantee that background network traffic is zero. Current Cassandra configuration allows entire-SSTable streaming when eligible; otherwise Cassandra can stream selected ranges/sections and materialize appropriate SSTables on the receiver. Eligibility depends on format, ownership boundaries, operation, and configuration.

3. Safe preview: estimate repair streaming before transferring data

Rather than forcing a large bootstrap in a three-node laptop cluster, use repair preview to build Merkle trees and estimate the differences that a repair would stream without performing the stream. On a freshly consistent tiny fixture, the estimate may be zero. To make the mechanism visible without unsafe network/firewall manipulation, create a controlled stale replica exactly as in Chapter 10, then preview a full repair. Exact byte estimates depend on timing and replica participation.

docker exec atlasmart-cass-1 nodetool disablehandoffdocker exec atlasmart-cass-2 nodetool disablehandoffdocker pause atlasmart-cass-3docker exec atlasmart-cass-1 cqlsh -e "CONSISTENCY LOCAL_QUORUM; UPDATE atlasmart_storage.orders_by_customer_day SET note='missed-by-node3' WHERE customer_id='cust-42' AND order_day='2026-09-07' AND order_time='2026-09-07T18:00:00Z' AND order_id=00000000-0000-0000-0000-000000000001;"docker exec atlasmart-cass-1 nodetool enablehandoffdocker exec atlasmart-cass-2 nodetool enablehandoffdocker unpause atlasmart-cass-3docker exec atlasmart-cass-1 nodetool status

# Preview estimates streaming; it does not stream the differences.docker exec atlasmart-cass-1 nodetool repair --preview --full atlasmart_storage orders_by_customer_day# In a second terminal while an actual repair runs, watch netstats.docker exec atlasmart-cass-1 nodetool repair --full atlasmart_storage orders_by_customer_daydocker exec atlasmart-cass-1 nodetool netstats -H

4. Entire-SSTable streaming and disk-headroom coupling

Cassandra can stream eligible entire SSTables, avoiding serialization of individual partitions and reducing CPU/GC pressure. But the receiver still needs space for incoming data and later compaction. If range boundaries require only sections of an SSTable, or eligibility conditions are not met, data can be streamed more granularly. In either case, planning only network bandwidth is insufficient: source read I/O, receiver write I/O, compaction, repair/anti-compaction, and snapshots can overlap.

A dangerous approach is to set streaming throughput to unlimited because a maintenance window is short. That can starve foreground traffic or disks. Throughput values are environment-specific; measure source/target saturation and adjust in a controlled window with rollback.

Production judgment

SSTable files expose valuable operational evidence, but they are not an application contract. Production decisions must combine logical data shape with physical storage state: partition rows/bytes, mutation and TTL/delete rates, RF/CL, read/write p95/p99, SSTables per read, compaction strategy and backlog, repaired/unrepaired state, compression ratio, CPU/decompression cost, disk throughput/latency, temporary compaction/streaming space, snapshot/backup retention, topology changes, repair cadence, and restore objectives. Include JVM/GC, page cache/off-heap use, driver timeouts/retries/idempotency, network bandwidth, tenant isolation, encryption-at-rest expectations, filesystem/device behavior, and managed-service restrictions.

Do not plan disk capacity as “live dataset bytes × RF” only. Flush, compaction, repair, streaming, snapshots, incremental backups, anti-compaction, restore staging, and operational safety margins can temporarily retain additional SSTables. Do not manually delete or edit component files to recover space. If space is critical, first stop unsafe automation, measure ownership/snapshots/compaction/streaming, and choose a documented recovery path with rollback. Lesson 5 combines component size, SSTable count, compaction/read amplification and operation-specific temporary copies into a practical disk-headroom model.

Summary and next bridge

Streaming moves replica ownership or repairs differences; the SSTable transport path is only one part of the operation. Monitor sessions, disk/network pressure and follow-up cleanup/repair separately. The final lesson turns those physical files and transfers into a disk-headroom/capacity model.

Authoritative references

These are version-sensitive sources of truth. Re-check them when regenerating the lesson because SSTable formats, utilities, defaults, and topology procedures evolve.

• Apache Cassandra downloads / current GA baseline
• Apache Cassandra storage engine and SSTable components
• Cassandra 5.0 cassandra.yaml SSTable format and streaming settings
• Apache Cassandra compression guidance
• Apache Cassandra SSTable tools safety overview
• sstablemetadata
• sstabledump
• sstablepartitions
• nodetool netstats
• Repair and streaming differences
• Topology changes and netstats monitoring
• nodetool bootstrap