Run cleanup After Topology Changes and Verify Ownership, Disk Use, and Replica Coverage

Learning outcomes

AtlasMart has successfully added capacity. The new node is healthy, but old nodes still report more disk usage than expected because Cassandra deliberately keeps data for ranges they used to own. Deleting SSTable files by hand would be unsafe. The supported finalization step is cleanup, performed only after ownership and replica coverage are accepted.

Terms and lifecycle mental model

A Cassandra node is one member of a peer-to-peer cluster. A datacenter (DC) and rack are logical topology labels used by replication placement to represent failure domains. A partition key is hashed by the partitioner to a token; with virtual nodes (vnodes), one physical node owns many token positions. A replica stores a copy of a token range according to the keyspace replication strategy. A coordinator is whichever node handles one request, not a permanent leader. A consistency level (CL) defines the replica acknowledgments/responses required for an operation.

A topology transition changes cluster membership or token ownership. Bootstrap is the join process in which a new node receives ranges by streaming from current replicas before becoming a normal owner. Pending ranges are ownership changes that are being prepared during a transition: Cassandra must keep writes safe while the future replica set is not yet fully ready. Streaming transfers SSTable data over the internode network. Decommission removes a healthy node and streams its ranges away. Replacement gives the logical ownership of a dead member to a fresh node identified by the dead endpoint. removenode removes an unreachable member from another live node and re-replicates from survivors. rebuild repopulates data on the current node by streaming from selected source nodes/DCs without changing that node's ownership. cleanup is a compaction-style rewrite that discards ranges a node no longer owns after a completed range movement. Repair is anti-entropy reconciliation between replicas; bootstrap/rebuild/cleanup do not make pre-existing replica inconsistencies disappear.

Node identity includes endpoint/broadcast address, host ID and token ownership recorded by cluster metadata. Headroom is spare disk, network, CPU, memory, compaction and replica availability capacity required to perform the transition without violating service objectives. A rollback/abort path is the documented action for a stalled or failed lifecycle sequence; it must be chosen from the current Cassandra version's supported topology state, not improvised by deleting system tables or reusing data directories.

1. Why Cassandra keeps old ranges

During a range movement Cassandra prioritizes availability and recoverability. Once node 4 bootstraps and assumes some replica ranges, the former owners do not immediately delete the SSTable sections for ranges they lost. Current topology guidance explicitly requires operators to run nodetool cleanup on nodes that lost ranges after the new node is known-good. Cleanup is implemented as compaction-style rewriting: SSTables containing mixed owned/unowned ranges may be rewritten so no-longer-owned keys can be discarded. That consumes disk I/O, CPU and temporary space.

2. Build the before-state: bootstrap node 4 and do not clean yet

# These names are dedicated to Chapter 23; the reset block never targets the shared course cluster.docker network inspect atlasmart-cassandra-life >/dev/null 2>&1 || docker network create atlasmart-cassandra-lifedocker volume create atlasmart-life-1-datadocker volume create atlasmart-life-2-datadocker volume create atlasmart-life-3-datadocker run -d --name atlasmart-life-1 --hostname atlasmart-life-1 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -v atlasmart-life-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 is UN before starting peers.docker exec atlasmart-life-1 nodetool statusdocker run -d --name atlasmart-life-2 --hostname atlasmart-life-2 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -e CASSANDRA_SEEDS=atlasmart-life-1 -v atlasmart-life-2-data:/var/lib/cassandra cassandra:5.0.9docker run -d --name atlasmart-life-3 --hostname atlasmart-life-3 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -e CASSANDRA_SEEDS=atlasmart-life-1 -v atlasmart-life-3-data:/var/lib/cassandra cassandra:5.0.9# Continue only after all three appear UN from at least two observers.docker exec atlasmart-life-1 nodetool statusdocker exec atlasmart-life-2 nodetool status

CREATE KEYSPACE IF NOT EXISTS atlasmart_lifecycleWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_lifecycle.order_probe (  order_id text PRIMARY KEY,  customer_id text,  status text,  total decimal,  updated_at timestamp) WITH compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_lifecycle.order_probe (order_id,customer_id,status,total,updated_at)VALUES ('ord-1001','cust-42','PAID',129.90,'2026-09-08T06:00:00Z');INSERT INTO atlasmart_lifecycle.order_probe (order_id,customer_id,status,total,updated_at)VALUES ('ord-1002','cust-77','PACKING',89.50,'2026-09-08T06:01:00Z');INSERT INTO atlasmart_lifecycle.order_probe (order_id,customer_id,status,total,updated_at)VALUES ('ord-1003','cust-42','SHIPPED',42.00,'2026-09-08T06:02:00Z');SELECT * FROM atlasmart_lifecycle.order_probe WHERE order_id='ord-1001';

docker exec atlasmart-life-1 sh -lc 'du -sh /var/lib/cassandra/data/atlasmart_lifecycle* 2>/dev/null || true'docker exec atlasmart-life-2 sh -lc 'du -sh /var/lib/cassandra/data/atlasmart_lifecycle* 2>/dev/null || true'docker exec atlasmart-life-3 sh -lc 'du -sh /var/lib/cassandra/data/atlasmart_lifecycle* 2>/dev/null || true'docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-1 nodetool getendpoints atlasmart_lifecycle order_probe ord-1001

docker volume create atlasmart-life-4-datadocker run -d --name atlasmart-life-4 --hostname atlasmart-life-4 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -e CASSANDRA_SEEDS=atlasmart-life-1 -v atlasmart-life-4-data:/var/lib/cassandra cassandra:5.0.9# Poll from separate terminals while the node is joining; a tiny dataset may finish too quickly to catch UJ.docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-4 nodetool netstats -Hdocker exec atlasmart-life-4 nodetool bootstrap# Final acceptance requires node 4 to be UN and schema versions to agree.docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-1 nodetool describecluster

Wait for node 4 to be UN, verify representative reads and let immediate streaming/compaction settle before cleanup. On a tiny fixture disk reduction may be too small to notice. That is expected; the lab teaches evidence and sequencing, not a guaranteed megabyte saving.

3. Capture old-owner disk state, then cleanup deliberately

docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-2 nodetool status atlasmart_lifecycledocker exec atlasmart-life-1 nodetool checktokenmetadatadocker exec atlasmart-life-1 nodetool describeclusterdocker exec atlasmart-life-1 cqlsh -e "CONSISTENCY LOCAL_QUORUM; SELECT * FROM atlasmart_lifecycle.order_probe WHERE order_id='ord-1001';"docker exec atlasmart-life-4 cqlsh -e "CONSISTENCY LOCAL_QUORUM; SELECT * FROM atlasmart_lifecycle.order_probe WHERE order_id='ord-1001';"# Record resource state before cleanup.docker exec atlasmart-life-1 nodetool compactionstatsdocker stats --no-stream atlasmart-life-1 atlasmart-life-2 atlasmart-life-3 atlasmart-life-4

# Run one node at a time so disk/compaction headroom is observable.docker exec atlasmart-life-1 nodetool cleanup -j 1 atlasmart_lifecycle order_probedocker exec atlasmart-life-1 nodetool compactionstatsdocker exec atlasmart-life-2 nodetool cleanup -j 1 atlasmart_lifecycle order_probedocker exec atlasmart-life-2 nodetool compactionstatsdocker exec atlasmart-life-3 nodetool cleanup -j 1 atlasmart_lifecycle order_probedocker exec atlasmart-life-3 nodetool compactionstats

It is safe for cleanup to do little on a node that did not lose relevant ranges. Do not force a large job count simply to make the command “finish faster”; parallel SSTable rewrites can steal disk headroom and latency from foreground requests. Current cleanup supports -j/--jobs; choose concurrency from measured hardware/workload, not a universal value.

4. Verify disk, ownership and replica coverage after cleanup

docker exec atlasmart-life-1 sh -lc 'du -sh /var/lib/cassandra/data/atlasmart_lifecycle* 2>/dev/null || true'docker exec atlasmart-life-2 sh -lc 'du -sh /var/lib/cassandra/data/atlasmart_lifecycle* 2>/dev/null || true'docker exec atlasmart-life-3 sh -lc 'du -sh /var/lib/cassandra/data/atlasmart_lifecycle* 2>/dev/null || true'docker exec atlasmart-life-4 sh -lc 'du -sh /var/lib/cassandra/data/atlasmart_lifecycle* 2>/dev/null || true'docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-1 nodetool describering atlasmart_lifecycledocker exec atlasmart-life-1 nodetool getendpoints atlasmart_lifecycle order_probe ord-1001docker exec atlasmart-life-1 nodetool checktokenmetadatadocker exec atlasmart-life-1 cqlsh -e "CONSISTENCY LOCAL_QUORUM; SELECT * FROM atlasmart_lifecycle.order_probe WHERE order_id='ord-1001';"

5. End-to-end lifecycle acceptance contract

docker rm -f atlasmart-life-1 atlasmart-life-2 atlasmart-life-3 atlasmart-life-4 atlasmart-life-4r atlasmart-life-dc2-1 2>/dev/null || truedocker volume rm atlasmart-life-1-data atlasmart-life-2-data atlasmart-life-3-data atlasmart-life-4-data atlasmart-life-4r-data atlasmart-life-dc2-1-data 2>/dev/null || truedocker network rm atlasmart-cassandra-life 2>/dev/null || true

Production judgment

Topology work consumes the same resources that serve customer traffic. Before a bootstrap, decommission, replacement, removal or rebuild, record Cassandra/JDK/driver versions, DC/rack layout, RF and query CLs, vnode count, per-node used/free disk, compaction backlog, streaming throughput, NIC saturation, JVM/GC pressure, repair age, hint window, backup status, SAI/vector indexes, tenant/security constraints, driver timeouts/retries/idempotency and representative p50/p95/p99 latency. Confirm that losing one additional host/rack during the operation still leaves the required replicas and operational headroom. Avoid concurrent range movements in the same failure domain unless the current version and runbook explicitly prove safety.

Streaming moves bytes; it does not cure bad partition keys, oversized partitions, old replica divergence, poor compaction headroom or missing repair. Cleanup reclaims no-longer-owned ranges; it is not anti-entropy. Replacement preserves ownership identity but still requires post-replacement convergence when downtime/write-forwarding exceeds the mechanisms that could cover missed writes. Managed Cassandra services may hide or forbid these commands; use the provider's documented replacement/scale workflow but keep the same evidence and acceptance model. Chapter 24 now focuses on anti-entropy. After learning how membership and ownership move, you can reason precisely about why hints, read repair and topology streaming still leave a scheduled repair obligation.

Summary and next bridge

Cleanup is the final ownership-reclamation step after a range movement, not a substitute for repair. Chapter 23 has separated six lifecycle workflows—bootstrap, decommission, replace, remove, rebuild and cleanup—by membership state, streaming source and ownership effect. Chapter 24 now asks the complementary question: how do replicas that are supposed to own the same ranges prove and restore data convergence over time?

Authoritative references

Lifecycle commands are version-sensitive and state-sensitive. Re-check these sources, the release notes and your deployment's orchestration/managed-service rules before applying a topology procedure.

• Apache Cassandra downloads / current 5.0 patch
• Adding, replacing, moving and removing nodes
• nodetool command reference
• nodetool bootstrap
• nodetool decommission
• nodetool removenode
• nodetool rebuild
• nodetool cleanup
• nodetool netstats
• cassandra-env JVM/system properties including replacement
• Compaction overview / cleanup semantics
• Docker Official Cassandra 5.0 packaging