Inspect Token Ownership and Distribution with nodetool and System Tables

Learning outcomes

The chapter ends by combining token metadata, keyspace replication, vnode ownership, request-key evidence, and streaming into one operator workflow. The goal is not to memorize nodetool columns; it is to know which command or system table answers each placement question and how to avoid false conclusions.

1. Build an evidence matrix before touching topology

docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 accepts CQL before starting peers.docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_SEEDS=atlasmart-cass-1 -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_SEEDS=atlasmart-cass-1 -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9# Wait for all nodes to become Up/Normal.docker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 nodetool version

docker exec atlasmart-cass-1 cqlsh -e "CREATE KEYSPACE IF NOT EXISTS atlasmart_tokens WITH replication = {'class':'NetworkTopologyStrategy','dc1':3};"docker exec atlasmart-cass-1 cqlsh -e "CREATE TABLE IF NOT EXISTS atlasmart_tokens.products_by_id (product_id text PRIMARY KEY, category text, name text, price_cents int);"docker exec atlasmart-cass-1 cqlsh -e "INSERT INTO atlasmart_tokens.products_by_id (product_id,category,name,price_cents) VALUES ('p-1001','laptop','AtlasBook 14',129900); INSERT INTO atlasmart_tokens.products_by_id (product_id,category,name,price_cents) VALUES ('p-1002','camera','AtlasCam X',89900); INSERT INTO atlasmart_tokens.products_by_id (product_id,category,name,price_cents) VALUES ('p-1003','audio','AtlasPods',14900); INSERT INTO atlasmart_tokens.products_by_id (product_id,category,name,price_cents) VALUES ('p-2001','home','AtlasLamp',7900);"

Each source answers a different question. Capture all of them before adding a node so the post-bootstrap comparison has a baseline.

docker exec atlasmart-cass-1 nodetool status atlasmart_tokensdocker exec atlasmart-cass-1 sh -lc "nodetool ring atlasmart_tokens > /tmp/ring-before.txt"docker exec atlasmart-cass-1 cqlsh -e "SELECT host_id, partitioner, tokens FROM system.local;"docker exec atlasmart-cass-1 cqlsh -e "SELECT peer,peer_port,host_id,data_center,rack,tokens FROM system.peers_v2;"docker exec atlasmart-cass-1 cqlsh -e "SELECT product_id,token(product_id) AS token_value FROM atlasmart_tokens.products_by_id;"docker exec atlasmart-cass-1 nodetool netstats -H

2. Build a deterministic partition-placement report

For each sample product, capture its token and replica set. This is stronger evidence than saying “RF=3 so it must be everywhere,” because it proves the actual metadata path on the running cluster.

# 1) Capture the business-key/token pairs.docker exec atlasmart-cass-1 cqlsh -e "SELECT product_id,token(product_id) AS token_value FROM atlasmart_tokens.products_by_id;"# 2) For each captured token, query replicas in an interactive cqlsh session.docker exec -it atlasmart-cass-1 cqlsh# SHOW REPLICAS <captured_token> atlasmart_tokens;# 3) Cross-check effective ownership and racks.docker exec atlasmart-cass-1 nodetool status atlasmart_tokens

Record host IDs/racks rather than relying only on ephemeral container IP addresses. In production, addresses can change while host identity and topology metadata are the more durable correlation keys.

3. Add node 4 and observe the ownership transition

Adding capacity is called bootstrap. The joining node receives token positions and streams the data needed for the ranges it will replicate. Define the blast radius first: this lab changes only disposable containers/volumes on atlasmart-cassandra; it does not remove production nodes, alter host networking, or modify an unrelated keyspace.

docker volume create atlasmart-cass-4-datadocker run -d --name atlasmart-cass-4 --hostname atlasmart-cass-4 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_SEEDS=atlasmart-cass-1 -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-4-data:/var/lib/cassandra cassandra:5.0.9# Sample repeatedly while bootstrap is active.docker exec atlasmart-cass-4 nodetool netstats -Hdocker exec atlasmart-cass-1 nodetool status atlasmart_tokensdocker exec atlasmart-cass-4 nodetool status atlasmart_tokens# After node 4 is Up/Normal, capture post-state.docker exec atlasmart-cass-1 cqlsh -e "SELECT peer,peer_port,host_id,data_center,rack,tokens FROM system.peers_v2;"docker exec atlasmart-cass-1 sh -lc "nodetool ring atlasmart_tokens > /tmp/ring-after.txt"

Do not assume the tiny dataset keeps the bootstrap state visible long enough for a single sample. On a fast laptop it may finish before netstats is run. Record that honestly and rely on before/after token metadata if the stream window is too short; for performance work, load a controlled synthetic dataset and measure the stream.

4. Compare before/after without confusing disk leftovers with ownership

After node 4 joins, some existing nodes lose token ranges. Cassandra's topology-change guidance notes that old range data is not automatically removed from nodes that lost ownership; operators run nodetool cleanup after they are satisfied the new node is healthy. This safety behavior means raw disk load can remain temporarily higher than the new ownership map implies.

# Compare current ownership to the saved ring files.docker exec atlasmart-cass-1 nodetool status atlasmart_tokensdocker exec atlasmart-cass-1 sh -lc "diff -u /tmp/ring-before.txt /tmp/ring-after.txt | head -120 || true"# Learn the cleanup scope first. Do not run blindly on unrelated clusters.docker exec atlasmart-cass-1 nodetool help cleanup# In this disposable lab, cleanup can be applied to the course keyspace after verification:docker exec atlasmart-cass-1 nodetool cleanup atlasmart_tokens

cleanup is not a rebalance command and does not “fix the ring.” It removes data that the node no longer owns after a range movement. Run it only after validating the topology change and with disk/compaction capacity awareness.

5. Chapter acceptance report and production judgment

A useful operator report names the exact Cassandra patch, partitioner, vnode policy, node/DC/rack map, keyspace RF, evidence time, sample partition-key tokens, replica sets, ownership/load before and after topology change, streaming evidence, and whether cleanup has run. It also states what was not measured: production latency, large-data stream throughput, repair duration, and real AZ failure tolerance.

Do not use manual token assignment or repeated scale-out as a substitute for schema review. If token ownership is balanced but toppartitions shows one key dominating traffic, Chapter 03's conclusion is a data-model action. If request distribution is healthy but ownership/load are uneven, investigate token allocation, hardware asymmetry, data-size distribution, compaction, and topology state separately.

Verification checklist

• Exactly 16 vnode positions per course node were explicitly configured and observed.
• You captured system/local peer token metadata and keyspace-aware nodetool output.
• You mapped sample product keys to tokens and replicas.
• You captured before/during/after evidence for the fourth-node bootstrap.
• You can explain why disk bytes can remain on former owners until cleanup.
• You did not present ownership percentages as request-distribution measurements.

Cleanup

docker rm -f atlasmart-cass-4 atlasmart-cass-1 atlasmart-cass-2 atlasmart-cass-3 2>/dev/null || truedocker volume rm atlasmart-cass-4-data atlasmart-cass-1-data atlasmart-cass-2-data atlasmart-cass-3-data 2>/dev/null || truedocker network rm atlasmart-cassandra 2>/dev/null || true

docker rm -f atlasmart-cass-4 atlasmart-cass-1 atlasmart-cass-2 atlasmart-cass-3 2>$nulldocker volume rm atlasmart-cass-4-data atlasmart-cass-1-data atlasmart-cass-2-data atlasmart-cass-3-data 2>$nulldocker network rm atlasmart-cassandra 2>$null

Summary and next step

This lesson’s concepts, evidence path, failure boundaries, and production judgment should now be explicit enough to verify rather than assume. Re-run the check-your-understanding prompts and preserve any lab evidence you need before changing or cleaning up the environment.

Next, continue to CREATE KEYSPACE and Replication Configuration: Durable Writes and Namespaces.

Authoritative references

• Apache Cassandra 5.0 documentation — Official documentation entry point for the current 5.0 line.
• Apache Cassandra downloads — Official release page used to verify Cassandra 5.0.9 as the current GA patch.
• Dynamo architecture: token ring, vnodes, replication — Official explanation of consistent hashing, token ranges, vnodes, natural replicas, and ring membership.
• cassandra.yaml configuration — Official partitioner, num_tokens, token-allocation, and related configuration reference.
• Production token recommendations — Current guidance for vnode counts and token-allocation tradeoffs.
• CQL token() function — Official semantics for mapping partition-key values to partitioner tokens.
• cqlsh SHOW REPLICAS — Official cqlsh command for resolving a token to replicas for a keyspace.
• nodetool ring — Official ring command and keyspace requirement for topology-aware ownership.
• Topology changes and bootstrap streaming — Official bootstrap/token-allocation/streaming and cleanup guidance.
• nodetool netstats — Official command for observing streaming/network activity.
• Java Driver 4.19 TokenMap — Driver-side token ranges, node tokens, and replica lookup API.
• nodetool status troubleshooting — Official interpretation of status, state/load/token/effective ownership output.