Hinted Handoff: Best-Effort Delivery of Missed Writes and Retention Limits

Learning outcomes

AtlasMart deploys a pricing change while one Cassandra replica restarts. The checkout write succeeds at LOCAL_QUORUM, yet the restarted node initially lacks the mutation. The operating question is not simply “did Cassandra replicate it?” It is: which replica missed the write, which coordinator stored a hint, how long can new hints be generated, when are they replayed, and what mechanism guarantees convergence if hint delivery is incomplete?

Failure-handling vocabulary

A coordinator is the Cassandra node handling one client request; it is not a permanent leader. A replica stores one copy of a partition according to the keyspace replication strategy. A partition is the set of rows sharing a partition key; Cassandra hashes that key to a token, and token ownership plus topology determines the replica set. RF (replication factor) is the number of replicas; CL (consistency level) is the response requirement for one operation. A hint is a durable record held by another node for a mutation a replica could not receive. Reconciliation chooses the newest visible cell/tombstone versions among replica responses. Read repair may write the reconciled result back to stale replicas involved in a read. Anti-entropy repair is the operator-run process that compares token-range data and streams differences; it is the comprehensive convergence mechanism. Speculation means starting redundant work before the original attempt has definitively failed. Idempotent means that repeating an operation produces the same intended database state as doing it once. A timeout means the operation did not complete within a deadline; it does not by itself prove that no replica applied a write. Unavailable means the coordinator already knows there are too few live replicas for the requested CL. Overloaded is a server response indicating the coordinator cannot currently process the request because its resources/backlog are exhausted.

1. Why Cassandra records a hint

For a normal RF=3 write, the coordinator sends the mutation to all three natural replicas even if the requested CL needs fewer acknowledgments. With LOCAL_QUORUM in this one-DC lab, two local acknowledgments satisfy the client. If the third replica is unavailable, the coordinator can persist a hint on its own local filesystem that identifies the target endpoint and mutation. The client-visible write can therefore succeed while replica state is temporarily divergent.

Current Cassandra 5.0 defaults enable hinted handoff and generate new hints for a failed endpoint for up to the configured max_hint_window, documented as three hours by default. This is a creation window, not a promise that every outage shorter than three hours converges exclusively through hints. Hint storage can be lost, delivery can lag, an outage can exceed the window, and operational mistakes can remove pending hints. Cassandra's documentation therefore classifies hints and read repair as best effort and keeps operator-run repair as the mechanism that provides comprehensive replica synchronization.

2. Reproduce a missed write and hold delivery long enough to observe it

docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 reports UN before starting peers.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9# Continue only when all three replicas are UN.docker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 java -versiondocker exec atlasmart-cass-1 cqlsh -e "SHOW VERSION"

CREATE KEYSPACE IF NOT EXISTS atlasmart_failureWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_failure.order_state_by_id (    order_id text PRIMARY KEY,    status text,    version int,    note text,    updated_at timestamp) WITH compaction = {'class':'UnifiedCompactionStrategy'}  AND read_repair = 'BLOCKING'  AND speculative_retry = 'NONE';CONSISTENCY ALL;INSERT INTO atlasmart_failure.order_state_by_id(order_id,status,version,note,updated_at)VALUES ('order-1501','CREATED',0,'baseline','2026-09-08T03:30:00Z');SELECT * FROM atlasmart_failure.order_state_by_id WHERE order_id='order-1501';

docker exec atlasmart-cass-1 nodetool statushandoffdocker exec atlasmart-cass-1 nodetool getmaxhintwindowdocker exec atlasmart-cass-1 sh -lc "grep -n -E 'hinted_handoff_enabled|max_hint_window|hints_directory|hinted_handoff_throttle' /etc/cassandra/cassandra.yaml || true"docker exec atlasmart-cass-1 sh -lc "ls -lah /var/lib/cassandra/hints || true"

For evidence, pause delivery on node 1 while still allowing it to store hints. Then make node 3 unavailable and send the write through node 1. pausehandoff is different from disablehandoff: the former is useful here because it lets pending hints accumulate without immediately replaying them when the target returns.

docker exec atlasmart-cass-1 nodetool pausehandoffdocker pause atlasmart-cass-3docker exec atlasmart-cass-1 nodetool status

CONSISTENCY LOCAL_QUORUM;UPDATE atlasmart_failure.order_state_by_idSET status='PAID', version=1, note='missed-by-node3', updated_at='2026-09-08T03:40:00Z'WHERE order_id='order-1501';SELECT * FROM atlasmart_failure.order_state_by_id WHERE order_id='order-1501';

docker exec atlasmart-cass-1 nodetool listpendinghintsdocker exec atlasmart-cass-1 sh -lc "ls -lah /var/lib/cassandra/hints || true"# Exact file names, bytes, endpoint IDs, and counts are runtime-dependent.

3. Recover the replica, prove it is stale, then replay hints

Bring node 3 back while node 1's hint delivery remains paused. A direct LOCAL_ONE query through node 3 can expose the old value if node 3 is a natural replica for this partition—which it is in this RF=3/three-node fixture. This is controlled evidence of temporary divergence, not a production recipe for forcing reads to a particular copy.

docker unpause atlasmart-cass-3# Wait until node 3 is UN.docker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-3 cqlsh -e "CONSISTENCY LOCAL_ONE; SELECT * FROM atlasmart_failure.order_state_by_id WHERE order_id='order-1501';"

docker exec atlasmart-cass-1 nodetool resumehandoff# Poll rather than assuming replay is instantaneous.docker exec atlasmart-cass-1 nodetool listpendinghintsdocker exec atlasmart-cass-3 cqlsh -e "CONSISTENCY LOCAL_ONE; SELECT * FROM atlasmart_failure.order_state_by_id WHERE order_id='order-1501';"docker exec atlasmart-cass-1 nodetool status

4. Verification and reset

• All nodes return UN.
• statushandoff reports enabled future handoff and delivery is resumed.
• Pending hints for the exercise drain or are explained if still replaying.
• Node 3 returns status='PAID' and version=1 after replay.
• No hint files were edited/deleted manually.

Production judgment

Failure handling is part of the application contract, not a last-minute driver knob. Record the operation's business invariant, RF/CL, local/remote datacenter scope, partition size/cardinality, payload size, write type, TTL/delete rate, compaction/SSTable state, repair cadence, hint window and delivery backlog, driver timeout/retry/speculation policy, idempotency decision, concurrency, routing, p95/p99/p99.9 latency, server overload/failure counts, JVM/GC, disk/network saturation, and the exact failure injection. A successful retry can hide an incident; a failed timeout can hide a successful mutation. Neither result is enough without postcondition verification.

Hints consume disk and replay network/write capacity; read repair adds foreground read/write work; scheduled repair consumes disk/network/CPU; speculative execution duplicates requests; retries can amplify overload. SAI/vector queries can have different tail-latency and duplicate-work costs, managed services may hide or constrain nodetool/JMX/driver controls, and security/TLS/authentication failures must not be misclassified as ordinary replica availability. Do not lower consistency, enlarge timeouts, extend hint windows, raise overload limits, or enable aggressive speculation from folklore. Test rollback and recovery under representative load. Lesson 2 deliberately removes hints from the equation so you can see reconciliation/read repair itself and why it still does not replace scheduled repair.

Summary and next bridge

Hinted handoff shortens the time a recovered replica stays stale, but it is bounded, best effort, and capacity-sensitive. The next lesson creates replica divergence with hints disabled so that reconciliation and read repair can be isolated from hint replay.

Authoritative references

Version-sensitive claims in this lesson should be rechecked against these current Apache sources when the lesson is regenerated. Driver 4.19.3 is the pinned artifact; some manual pages remain published under the 4.19.0 documentation path but the relevant semantics are also represented in the current 4.19.x source/changelog.

• Apache Cassandra downloads and current 5.0 release
• Apache Cassandra hinted handoff
• Apache Cassandra replica synchronization / consistency architecture
• Apache Cassandra repair
• CQL CREATE TABLE: read_repair and speculative_retry options
• Apache Cassandra native protocol error codes
• Apache Cassandra Java Driver changelog
• Java Driver retries
• Java Driver idempotence
• Java Driver speculative execution
• nodetool listpendinghints
• nodetool statushandoff