Diagnose a Slow Read by Partition Size, SSTable Count, Tombstones, Cache, and Disk Evidence

Learning outcomes

AtlasMart receives a p99 alert for an order-read endpoint. Instead of immediately enlarging caches or blaming disk, the team will collect evidence from the distributed request, partition shape, SSTable amplification, tombstones, cache/Bloom state, and host resources.

Core terms before tracing a read

A coordinator is the Cassandra node handling one client request; it is not a permanent leader. A replica stores a copy of a partition according to replication placement. A partition is the rows sharing a partition key; the partitioner hashes that key to a token, which participates in replica ownership. A consistency level (CL) specifies how many appropriately scoped replica responses a coordinator needs for the operation. An SSTable (Sorted String Table) is immutable on-disk storage. A tombstone is a distributed deletion/expiration marker. A Bloom filter is a probabilistic membership structure that can say “definitely absent” or “possibly present”; for its intended membership check it can produce false positives but not false negatives. Reconciliation merges cell versions and tombstones to construct the newest visible result. Read repair may write that reconciled result back to stale replicas involved in a request; it does not replace scheduled anti-entropy repair.

1. Diagnose in layers

Each layer can explain the next. A large partition with tombstones can cause disk work despite excellent Bloom behavior. Many overlapping SSTables can make a small logical read expensive. A tiny warm working set can make local cache measurements look unrealistically good.

2. Build a bounded but amplification-prone fixture

docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 java -version# Recreate only if the shared course lab does not already exist.docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 is UN before starting peers.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9docker exec atlasmart-cass-1 nodetool statusdocker exec -it atlasmart-cass-1 cqlsh

CREATE KEYSPACE IF NOT EXISTS atlasmart_readpathWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_readpath.slow_read_probe (  bucket text, event_time timestamp, event_id uuid,  payload text, obsolete text,  PRIMARY KEY (bucket,event_time,event_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_readpath.slow_read_probe VALUES ('b-2026-09-07','2026-09-07T18:00:00Z',00000000-0000-0000-0000-000000000001,'p1','x1');INSERT INTO atlasmart_readpath.slow_read_probe VALUES ('b-2026-09-07','2026-09-07T18:01:00Z',00000000-0000-0000-0000-000000000002,'p2','x2');INSERT INTO atlasmart_readpath.slow_read_probe VALUES ('b-2026-09-07','2026-09-07T18:02:00Z',00000000-0000-0000-0000-000000000003,'p3','x3');

for n in 1 2 3; do docker exec atlasmart-cass-$n nodetool flush atlasmart_readpath slow_read_probe; done

UPDATE atlasmart_readpath.slow_read_probe SET payload='p2-new'WHERE bucket='b-2026-09-07' AND event_time='2026-09-07T18:01:00Z'AND event_id=00000000-0000-0000-0000-000000000002;DELETE obsolete FROM atlasmart_readpath.slow_read_probeWHERE bucket='b-2026-09-07' AND event_time='2026-09-07T18:00:00Z'AND event_id=00000000-0000-0000-0000-000000000001;

for n in 1 2 3; do docker exec atlasmart-cass-$n nodetool flush atlasmart_readpath slow_read_probe; donedocker exec atlasmart-cass-1 nodetool tablestats atlasmart_readpath.slow_read_probedocker exec atlasmart-cass-1 nodetool tablehistograms atlasmart_readpath slow_read_probe

3. Trace, then separate warmup from evidence

CONSISTENCY LOCAL_QUORUM;TRACING ON;SELECT event_time,event_id,payload FROM atlasmart_readpath.slow_read_probeWHERE bucket='b-2026-09-07'AND event_time >= '2026-09-07T18:00:00Z'AND event_time < '2026-09-07T18:03:00Z';TRACING OFF;

docker exec atlasmart-cass-1 nodetool info | grep -i -A4 -B1 cache || truedocker exec atlasmart-cass-1 nodetool tablestats atlasmart_readpath.slow_read_probedocker exec atlasmart-cass-1 nodetool tablehistograms atlasmart_readpath slow_read_probedocker stats --no-stream atlasmart-cass-1 atlasmart-cass-2 atlasmart-cass-3

Repeat untraced reads only to establish a distribution; report first/cold-ish and later/warm observations separately. OS page cache, Cassandra caches, background compaction, and container scheduling can all change between attempts.

4. Compare physical rewrite with model improvement

docker exec atlasmart-cass-1 nodetool compact atlasmart_readpath slow_read_probedocker exec atlasmart-cass-1 nodetool tablehistograms atlasmart_readpath slow_read_probedocker exec atlasmart-cass-1 nodetool tablestats atlasmart_readpath.slow_read_probe

Compaction changes the physical file set. A query-first redesign changes the unit of data a request addresses. For an ever-growing time partition, an hour/day bucket can bound partition growth if the application can route to the right bucket and accept bounded fan-out.

CREATE TABLE IF NOT EXISTS atlasmart_readpath.slow_read_probe_hourly (  bucket_day date, bucket_hour tinyint, event_time timestamp, event_id uuid, payload text,  PRIMARY KEY ((bucket_day,bucket_hour),event_time,event_id)) WITH CLUSTERING ORDER BY (event_time DESC,event_id ASC)  AND compaction = {'class':'UnifiedCompactionStrategy'};INSERT INTO atlasmart_readpath.slow_read_probe_hourly VALUES('2026-09-07',18,'2026-09-07T18:01:00Z',00000000-0000-0000-0000-000000000002,'p2-new');SELECT * FROM atlasmart_readpath.slow_read_probe_hourlyWHERE bucket_day='2026-09-07' AND bucket_hour=18;

5. Incident record and acceptance checks

Production judgment

Read latency is not a single disk or cache number. Evaluate partition rows/bytes and skew, clustering slices, RF and CL, replica locality/health, p95/p99 latency, SSTables touched per logical read, tombstones scanned, compaction state, Bloom false positives, key/chunk/page-cache state, disk throughput/queueing, JVM/GC, driver timeout/retry/speculation, repair state, and failure-domain health. Large partitions or overlapping SSTables can dominate an otherwise healthy cache. Conversely, aggressive cache allocation can steal memory from the OS page cache or other off-heap structures.

Do not generalize a warm laptop result. Record the Cassandra patch, SSTable format, topology, RF/CL, dataset and partition distribution, payload/result size, TTL/delete rate, compaction state, concurrent load, disk/network, warmup, and failure injection. Forced flush/compaction is a controlled lab/maintenance action, not routine performance tuning. Chapter 11 opens the SSTable itself and examines its Data, index/trie, filter, statistics, compression, TOC, and streaming details.

Summary and next bridge

Slow-read diagnosis should move from distributed request evidence to partition shape, SSTable/tombstone amplification, caches/Bloom filters, and host I/O. Chapter 11 now examines SSTable components and on-disk storage internals directly.

Authoritative references

Re-check these version-sensitive sources when regenerating the lesson.

• Apache Cassandra downloads
• Storage Engine / SSTable formats
• CQL DDL: Bloom, cache, read_repair options
• cassandra.yaml cache/SSTable settings
• nodetool tablestats
• nodetool tablehistograms
• Repair
• nodetool getsstables