Inspect system_schema and Trace Schema Changes Across a Cluster

Learning outcomes

AtlasMart has a rolling deployment that adds one optional product attribute. The team needs evidence for when Cassandra nodes agree on the schema—and equally important, evidence that this milestone does not prove application rollout is complete. We will inspect system_schema locally on each node, pause one member, apply additive DDL, and watch it catch up after recovery.

1. system_schema is observable metadata, not an application table

Cassandra stores schema definitions in the system_schema keyspace. You can query keyspace, table and column definitions with ordinary SELECT syntax, but these tables are Cassandra-maintained metadata; application code should not mutate them directly. DDL is the supported mutation interface.

# Disposable Chapter 05 cluster: three nodes, three racks, one DC, 16 vnodes/node# If these objects already exist from Chapters 01–04, reuse them instead of recreating them.docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra   -e CASSANDRA_CLUSTER_NAME=atlasmart-course   -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1   -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch   -e CASSANDRA_NUM_TOKENS=16   -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait for node 1 to accept CQL, then start the two peers.docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra   -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_SEEDS=atlasmart-cass-1   -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2   -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch   -e CASSANDRA_NUM_TOKENS=16   -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra   -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_SEEDS=atlasmart-cass-1   -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3   -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch   -e CASSANDRA_NUM_TOKENS=16   -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9docker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 java -version

CREATE KEYSPACE IF NOT EXISTS atlasmart_cqlWITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3}AND durable_writes = true;CONSISTENCY QUORUM;CREATE TABLE IF NOT EXISTS atlasmart_cql.products_by_id (    product_id text PRIMARY KEY,    name text,    category text,    price_cents bigint,    status text,    description text,    updated_at timestamp);CREATE TABLE IF NOT EXISTS atlasmart_cql.products_by_category (    category text,    bucket tinyint,    product_id text,    name text,    status text,    price_cents bigint,    PRIMARY KEY ((category, bucket), product_id));CREATE TABLE IF NOT EXISTS atlasmart_cql.inventory_by_product (    product_id text,    location_id text,    quantity int,    status text,    promo_note text,    PRIMARY KEY ((product_id), location_id));

SELECT keyspace_name, durable_writes, replicationFROM system_schema.keyspacesWHERE keyspace_name='atlasmart_cql';SELECT table_name, idFROM system_schema.tablesWHERE keyspace_name='atlasmart_cql';SELECT table_name,column_name,kind,position,typeFROM system_schema.columnsWHERE keyspace_name='atlasmart_cql'  AND table_name='products_by_id';

Save this output as the before evidence. The table ID and exact metadata ordering are implementation/version evidence, not values to hard-code into application logic.

2. Schema agreement is a cluster state you can poll

nodetool describecluster prints the cluster name, snitch, partitioner and schema-version information. In a healthy settled cluster, participating reachable nodes should converge on the same schema version. Automation should poll this evidence with a bounded deadline rather than assume “DDL returned, therefore all nodes agree.”

docker exec atlasmart-cass-1 nodetool describeclusterdocker exec atlasmart-cass-2 nodetool describeclusterdocker exec atlasmart-cass-3 nodetool describecluster

Record the schema UUID(s) shown. A single UUID across the three-node lab is the expected settled state. Exact output sections differ across versions and reachability conditions, so the lesson does not fabricate a UUID.

3. Controlled failure: keep one node from hearing the DDL

Blast radius: only atlasmart-cass-3 in the disposable local Docker cluster is paused. No host firewall or clock changes are required. RF remains 3. We are testing metadata propagation, not claiming application availability or durability from this experiment.

docker pause atlasmart-cass-3docker exec atlasmart-cass-1 cqlsh -e "ALTER TABLE atlasmart_cql.products_by_id ADD IF NOT EXISTS rollout_note text;"docker exec atlasmart-cass-1 cqlsh -e "SELECT column_name,type FROM system_schema.columns WHERE keyspace_name='atlasmart_cql' AND table_name='products_by_id' AND column_name='rollout_note';"docker exec atlasmart-cass-2 cqlsh -e "SELECT column_name,type FROM system_schema.columns WHERE keyspace_name='atlasmart_cql' AND table_name='products_by_id' AND column_name='rollout_note';"docker exec atlasmart-cass-1 nodetool describecluster

While the container is paused, do not try to docker exec cqlsh inside it and treat the hang as a Cassandra query failure—the container process is intentionally frozen. The useful evidence is that reachable nodes can adopt the new schema while another member is unavailable, and cluster schema-version output may reflect reachability/disagreement depending on timing.

4. Recover, observe convergence, and verify locally on the returning node

docker unpause atlasmart-cass-3# Give the node a chance to resume gossip, then inspect—do not rely on sleep alone in automation.docker exec atlasmart-cass-3 nodetool statusdocker exec atlasmart-cass-3 nodetool describeclusterdocker exec atlasmart-cass-3 cqlsh -e "SELECT column_name,type FROM system_schema.columns WHERE keyspace_name='atlasmart_cql' AND table_name='products_by_id' AND column_name='rollout_note';"docker exec atlasmart-cass-1 nodetool describeclusterdocker exec atlasmart-cass-2 nodetool describecluster

Acceptance requires both metadata observations: the returning node can see rollout_note in its local system_schema view, and the cluster returns to a common schema version. If it does not converge within a reasonable lab deadline, collect logs and network/gossip state instead of repeatedly issuing DDL.

5. Trace a query after schema convergence—and keep the evidence categories separate

The prompt asks for tracing selected queries. After convergence, use cqlsh tracing on a normal read. This proves coordinator/replica request flow; it is different evidence from schema-version convergence.

TRACING ON;CONSISTENCY QUORUM;SELECT product_id,name,status,rollout_noteFROM atlasmart_cql.products_by_idWHERE product_id='p-501';TRACING OFF;

If the row does not exist because Lesson 4 was not run immediately before this lesson, insert one first. Record the trace session/events from your machine. Schema agreement does not imply rollout_note has a non-null value; historical data remains unchanged until writers/backfills populate it.

6. Wrong approach: use schema agreement as the deployment health check

Suppose all nodes agree after adding rollout_note, but half of AtlasMart still runs an old binary whose decoder assumes a fixed column list or whose prepared SELECT * metadata is stale in an older protocol/client combination. Cassandra can be healthy while the application is broken. Conversely, an application can tolerate a mixed rollout while one down node has not yet learned the schema.

The repair is a staged contract: additive DDL, explicit agreement gate, compatibility-tested old/new readers, new writers, optional backfill, observability, then destructive cleanup only after the rollback window. Prepared statements should list intended columns rather than relying casually on SELECT *, especially across schema evolution.

Verification checklist

• Before-state system_schema metadata is recorded.
• Node 3 alone is paused; no host networking is modified.
• The additive column appears on reachable nodes.
• After unpause, node 3 sees the same column and cluster schema versions converge.
• A traced post-change query is captured separately from schema-agreement evidence.
• You can name application/backfill checks still required after agreement.

Summary and next bridge

Chapter 05 closes with a precise separation between CQL syntax and distributed behavior: schema changes propagate, mutations reconcile by timestamps, reads are constrained by access paths, drivers prepare recurring query structure, and cqlsh remains an operator client. Chapter 06 uses those foundations to design tables from query patterns, bounded partitions and deliberate denormalization.

# Chapter-only reset or deliberate preservationdocker exec atlasmart-cass-1 cqlsh -e "DROP KEYSPACE IF EXISTS atlasmart_cql;"# Full reset (deletes only the dedicated course containers/volumes/network)docker rm -f atlasmart-cass-1 atlasmart-cass-2 atlasmart-cass-3docker volume rm atlasmart-cass-1-data atlasmart-cass-2-data atlasmart-cass-3-datadocker network rm atlasmart-cassandra

Authoritative references

• Apache Cassandra downloads — Current Cassandra and ASF Java-driver release baseline.
• CQL data definition — Official keyspace/table DDL and schema options.
• CQL data manipulation — Official INSERT/UPDATE/DELETE/SELECT, timestamps, TTL, WRITETIME and filtering semantics.
• cqlsh documentation — Official shell commands, tracing, paging, SOURCE/CAPTURE/COPY behavior and compatibility boundary.
• Native protocol — PREPARE/EXECUTE protocol semantics and version framing.
• Apache Cassandra Java Driver prepared statements — Prepared/bound statement behavior, caching and routing metadata.
• nodetool describecluster — Official schema-version/cluster description command.