Remove Unreachable Nodes, Rebuild from Another Datacenter, and Understand Streaming Sources

Learning outcomes

AtlasMart has two different incidents: one dead node will never return and should be removed; separately, a surviving node needs to repopulate local replica data from a remote datacenter. Both involve streaming, but their ownership semantics are opposite. removenode changes membership; rebuild keeps membership and repopulates the current node from selected sources.

Terms and lifecycle mental model

A Cassandra node is one member of a peer-to-peer cluster. A datacenter (DC) and rack are logical topology labels used by replication placement to represent failure domains. A partition key is hashed by the partitioner to a token; with virtual nodes (vnodes), one physical node owns many token positions. A replica stores a copy of a token range according to the keyspace replication strategy. A coordinator is whichever node handles one request, not a permanent leader. A consistency level (CL) defines the replica acknowledgments/responses required for an operation.

A topology transition changes cluster membership or token ownership. Bootstrap is the join process in which a new node receives ranges by streaming from current replicas before becoming a normal owner. Pending ranges are ownership changes that are being prepared during a transition: Cassandra must keep writes safe while the future replica set is not yet fully ready. Streaming transfers SSTable data over the internode network. Decommission removes a healthy node and streams its ranges away. Replacement gives the logical ownership of a dead member to a fresh node identified by the dead endpoint. removenode removes an unreachable member from another live node and re-replicates from survivors. rebuild repopulates data on the current node by streaming from selected source nodes/DCs without changing that node's ownership. cleanup is a compaction-style rewrite that discards ranges a node no longer owns after a completed range movement. Repair is anti-entropy reconciliation between replicas; bootstrap/rebuild/cleanup do not make pre-existing replica inconsistencies disappear.

Node identity includes endpoint/broadcast address, host ID and token ownership recorded by cluster metadata. Headroom is spare disk, network, CPU, memory, compaction and replica availability capacity required to perform the transition without violating service objectives. A rollback/abort path is the documented action for a stalled or failed lifecycle sequence; it must be chosen from the current Cassandra version's supported topology state, not improvised by deleting system tables or reusing data directories.

1. removenode: the member is dead and will not stream

For an unreachable node, execute nodetool removenode <host-id> from another live member. Surviving replicas stream data needed to restore the configured replica placement. The host ID is the membership identity shown by nodetool status/system metadata; it is not interchangeable with an IP string. removenode status reports an in-progress removal and removenode force exists to force completion of a stuck removal, not as the normal first command. Cassandra lists assassinate as a last resort when removenode cannot be used because it removes membership without normal re-replication safety.

2. Controlled dead-node removal

# These names are dedicated to Chapter 23; the reset block never targets the shared course cluster.docker network inspect atlasmart-cassandra-life >/dev/null 2>&1 || docker network create atlasmart-cassandra-lifedocker volume create atlasmart-life-1-datadocker volume create atlasmart-life-2-datadocker volume create atlasmart-life-3-datadocker run -d --name atlasmart-life-1 --hostname atlasmart-life-1 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -v atlasmart-life-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 is UN before starting peers.docker exec atlasmart-life-1 nodetool statusdocker run -d --name atlasmart-life-2 --hostname atlasmart-life-2 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -e CASSANDRA_SEEDS=atlasmart-life-1 -v atlasmart-life-2-data:/var/lib/cassandra cassandra:5.0.9docker run -d --name atlasmart-life-3 --hostname atlasmart-life-3 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -e CASSANDRA_SEEDS=atlasmart-life-1 -v atlasmart-life-3-data:/var/lib/cassandra cassandra:5.0.9# Continue only after all three appear UN from at least two observers.docker exec atlasmart-life-1 nodetool statusdocker exec atlasmart-life-2 nodetool status

CREATE KEYSPACE IF NOT EXISTS atlasmart_lifecycleWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_lifecycle.order_probe (  order_id text PRIMARY KEY,  customer_id text,  status text,  total decimal,  updated_at timestamp) WITH compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_lifecycle.order_probe (order_id,customer_id,status,total,updated_at)VALUES ('ord-1001','cust-42','PAID',129.90,'2026-09-08T06:00:00Z');INSERT INTO atlasmart_lifecycle.order_probe (order_id,customer_id,status,total,updated_at)VALUES ('ord-1002','cust-77','PACKING',89.50,'2026-09-08T06:01:00Z');INSERT INTO atlasmart_lifecycle.order_probe (order_id,customer_id,status,total,updated_at)VALUES ('ord-1003','cust-42','SHIPPED',42.00,'2026-09-08T06:02:00Z');SELECT * FROM atlasmart_lifecycle.order_probe WHERE order_id='ord-1001';

docker volume create atlasmart-life-4-datadocker run -d --name atlasmart-life-4 --hostname atlasmart-life-4 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -e CASSANDRA_SEEDS=atlasmart-life-1 -v atlasmart-life-4-data:/var/lib/cassandra cassandra:5.0.9# Poll from separate terminals while the node is joining; a tiny dataset may finish too quickly to catch UJ.docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-4 nodetool netstats -Hdocker exec atlasmart-life-4 nodetool bootstrap# Final acceptance requires node 4 to be UN and schema versions to agree.docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-1 nodetool describecluster

docker exec atlasmart-life-1 nodetool status atlasmart_lifecycle# Copy the Host ID shown for atlasmart-life-4 / its endpoint before continuing.DEAD_HOST_ID=<paste-node-4-host-id-here>docker stop atlasmart-life-4docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-2 nodetool status atlasmart_lifecycle

DEAD_HOST_ID=<paste-node-4-host-id-here># Terminal A:docker exec atlasmart-life-1 nodetool removenode "$DEAD_HOST_ID"# Terminal B, while removal is active:docker exec atlasmart-life-1 nodetool removenode statusdocker exec atlasmart-life-1 nodetool netstats -Hdocker exec atlasmart-life-2 nodetool netstats -H# Acceptance:docker exec atlasmart-life-1 nodetool status atlasmart_lifecycledocker exec atlasmart-life-1 nodetool checktokenmetadatadocker exec atlasmart-life-1 cqlsh -e "CONSISTENCY LOCAL_QUORUM; SELECT * FROM atlasmart_lifecycle.order_probe WHERE order_id='ord-1001';"

3. rebuild: same node, same ownership, chosen streaming source

nodetool rebuild repopulates data on the node on which it runs by streaming from other nodes. It is commonly used when populating/restoring a datacenter or when a node needs data rebuilt from another DC. The command accepts a source DC and current 5.0 options for a keyspace, explicit sources/tokens, or excluding the local DC. Unlike bootstrap/decommission/removenode, rebuild does not itself transfer token ownership away from or to the current member.

The local extension below adds one dc2 source and changes only the disposable keyspace to RF dc1:3, dc2:1. A repair populates the new DC before the rebuild test; production multi-DC expansion requires a deliberate DC bootstrap/repair plan rather than this compressed lab sequence.

docker volume create atlasmart-life-dc2-1-datadocker run -d --name atlasmart-life-dc2-1 --hostname atlasmart-life-dc2-1 --network atlasmart-cassandra-life \  -e CASSANDRA_CLUSTER_NAME=atlasmart-lifecycle -e CASSANDRA_DC=dc2 -e CASSANDRA_RACK=rack1 \  -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 \  -e CASSANDRA_SEEDS=atlasmart-life-1 -v atlasmart-life-dc2-1-data:/var/lib/cassandra cassandra:5.0.9# Wait for the dc2 node to be UN, then change this disposable keyspace only.docker exec atlasmart-life-1 cqlsh -e "ALTER KEYSPACE atlasmart_lifecycle WITH replication = {'class':'NetworkTopologyStrategy','dc1':3,'dc2':1};"# Populate new replicas; exact repair flags/cadence are Chapter 24 material.docker exec atlasmart-life-1 nodetool repair --full atlasmart_lifecycle order_probedocker exec atlasmart-life-dc2-1 cqlsh -e "CONSISTENCY LOCAL_ONE; SELECT * FROM atlasmart_lifecycle.order_probe WHERE order_id='ord-1001';"

# Terminal A:docker exec atlasmart-life-1 nodetool rebuild -ks atlasmart_lifecycle dc2# Terminal B:docker exec atlasmart-life-1 nodetool netstats -Hdocker stats --no-stream atlasmart-life-1 atlasmart-life-dc2-1# Rebuild should not change node 1 token ownership/membership.docker exec atlasmart-life-1 nodetool status atlasmart_lifecycle

4. Source-selection and network judgment

A rebuild from another DC is not automatically a good idea because it is syntactically available. Check source DC health, WAN capacity/cost, encryption, throttles, locality and whether the chosen source has been repaired. On a managed service, direct rebuild/removenode may be unavailable; use provider workflows and retain the same evidence model.

5. Verification and reset

docker rm -f atlasmart-life-1 atlasmart-life-2 atlasmart-life-3 atlasmart-life-4 atlasmart-life-4r atlasmart-life-dc2-1 2>/dev/null || truedocker volume rm atlasmart-life-1-data atlasmart-life-2-data atlasmart-life-3-data atlasmart-life-4-data atlasmart-life-4r-data atlasmart-life-dc2-1-data 2>/dev/null || truedocker network rm atlasmart-cassandra-life 2>/dev/null || true

Production judgment

Topology work consumes the same resources that serve customer traffic. Before a bootstrap, decommission, replacement, removal or rebuild, record Cassandra/JDK/driver versions, DC/rack layout, RF and query CLs, vnode count, per-node used/free disk, compaction backlog, streaming throughput, NIC saturation, JVM/GC pressure, repair age, hint window, backup status, SAI/vector indexes, tenant/security constraints, driver timeouts/retries/idempotency and representative p50/p95/p99 latency. Confirm that losing one additional host/rack during the operation still leaves the required replicas and operational headroom. Avoid concurrent range movements in the same failure domain unless the current version and runbook explicitly prove safety.

Streaming moves bytes; it does not cure bad partition keys, oversized partitions, old replica divergence, poor compaction headroom or missing repair. Cleanup reclaims no-longer-owned ranges; it is not anti-entropy. Replacement preserves ownership identity but still requires post-replacement convergence when downtime/write-forwarding exceeds the mechanisms that could cover missed writes. Managed Cassandra services may hide or forbid these commands; use the provider's documented replacement/scale workflow but keep the same evidence and acceptance model. Lesson 5 closes the lifecycle by reclaiming ranges that old owners intentionally retained after movement and proving that cleanup changes disk state without being mistaken for repair.

Summary and next bridge

Removing a dead member and rebuilding a live member can both stream bytes but have different ownership semantics. removenode changes membership and relies on survivors; rebuild keeps membership while choosing source replicas. The final lifecycle lesson handles the data deliberately left behind after range movement.

Authoritative references

Lifecycle commands are version-sensitive and state-sensitive. Re-check these sources, the release notes and your deployment's orchestration/managed-service rules before applying a topology procedure.

• Apache Cassandra downloads / current 5.0 patch
• Adding, replacing, moving and removing nodes
• nodetool command reference
• nodetool bootstrap
• nodetool decommission
• nodetool removenode
• nodetool rebuild
• nodetool cleanup
• nodetool netstats
• cassandra-env JVM/system properties including replacement
• Compaction overview / cleanup semantics
• Docker Official Cassandra 5.0 packaging