Timestamps, Conflict Resolution, Tombstones, TTL, and Last-Write-Wins at the Cell Level

Learning outcomes

Two AtlasMart services race to update the same order status. One retry arrives late with an older timestamp; another writes a TTL; a delete follows. Cassandra must reconcile replicas and SSTables without in-place overwrites, so each live cell or deletion marker carries metadata that determines which version wins. The application therefore needs timestamp discipline, not “last HTTP request wins” intuition.

1. Cassandra resolves versions at cell granularity

For ordinary CQL writes, if no timestamp is supplied, the coordinator assigns a timestamp in microseconds near the start of statement execution. Cassandra stores mutation timestamps and reconciles competing versions primarily by timestamp. That means last write wins means “highest relevant mutation timestamp wins,” not “the request that physically arrived last at this node wins.”

Different non-primary-key cells in one row can have different timestamps and TTLs. A retry with an older explicit timestamp can still receive a successful CL acknowledgment because replicas accepted the mutation operation, yet the older value will lose during reconciliation to an already stored newer cell version.

2. Lab: prove an older acknowledged write can lose

# Verify the shared lab if it already exists.docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool status# Standalone recreation path. Skip resources that already exist.docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 answers before starting peers.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9# Wait until all three nodes show UN in dc1 before continuing.docker exec atlasmart-cass-1 nodetool status# Open cqlsh on node 1 for the SQL/CQL blocks that follow.docker exec -it atlasmart-cass-1 cqlsh

CREATE KEYSPACE IF NOT EXISTS atlasmart_writepathWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_writepath.order_write_events (    order_id text,    event_time timeuuid,    status text,    source text,    note text,    expires_at timestamp,    PRIMARY KEY ((order_id), event_time)) WITH CLUSTERING ORDER BY (event_time DESC)  AND compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;SELECT keyspace_name, replicationFROM system_schema.keyspacesWHERE keyspace_name = 'atlasmart_writepath';

CONSISTENCY LOCAL_QUORUM;-- Run these statements inside cqlsh on atlasmart-cass-1.INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-clock', 8d2c0f30-6d91-11f0-8000-000000000001, 'PAID', 'checkout-api', 'newer timestamp')USING TIMESTAMP 2000000;SELECT status, WRITETIME(status), source, WRITETIME(source)FROM atlasmart_writepath.order_write_eventsWHERE order_id='order-clock';UPDATE atlasmart_writepath.order_write_eventsUSING TIMESTAMP 1000000SET status='CREATED', note='older retry should lose for overlapping cells'WHERE order_id='order-clock'  AND event_time=8d2c0f30-6d91-11f0-8000-000000000001;SELECT status, WRITETIME(status), note, WRITETIME(note)FROM atlasmart_writepath.order_write_eventsWHERE order_id='order-clock';

The overlapping status cell should retain the version with timestamp 2,000,000. The note outcome depends on its own prior timestamp because reconciliation is cell-level. This is why “row timestamp” shorthand can hide important behavior.

3. TTL is future deletion metadata, not a scheduled physical erase

Time to live (TTL) attaches an expiration time to written values. TTL(column) reports remaining seconds while a value is live. Updating a cell can reset or replace its TTL according to the new mutation. After expiration, Cassandra must continue to suppress older replicas/SSTables that still contain the pre-expiry value, so expiration participates in tombstone/deletion semantics rather than simply removing bytes at the deadline.

INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-ttl', now(), 'TEMPORARY', 'lab', 'expires for the exercise')USING TTL 120;SELECT status, TTL(status), WRITETIME(status), note, TTL(note)FROM atlasmart_writepath.order_write_eventsWHERE order_id='order-ttl';# Re-run the SELECT after a short interval and record the lower TTL.# Do not wait for compaction or modify gc_grace_seconds merely to make tombstones disappear.

The TTL countdown is direct evidence of expiry metadata. It is not evidence that bytes will be removed from every SSTable at the exact expiration second. Physical reclamation depends on compaction and deletion-safety conditions.

4. Deletes write tombstones so replicas cannot resurrect old data

An immutable SSTable cannot be edited in place to remove a cell. Cassandra therefore writes a tombstone, a deletion marker with a timestamp, which wins over older live values during reconciliation. Tombstones remain relevant long enough for replicas to learn about the deletion. Repair is critical because a replica that misses a delete and later presents an older live value can otherwise participate in resurrection risks after deletion metadata is garbage-collected.

UPDATE atlasmart_writepath.order_write_eventsUSING TIMESTAMP 3000000SET status='PACKED'WHERE order_id='order-clock'  AND event_time=8d2c0f30-6d91-11f0-8000-000000000001;DELETE status FROM atlasmart_writepath.order_write_eventsUSING TIMESTAMP 4000000WHERE order_id='order-clock'  AND event_time=8d2c0f30-6d91-11f0-8000-000000000001;SELECT status, note, WRITETIME(note)FROM atlasmart_writepath.order_write_eventsWHERE order_id='order-clock';

# Force flush only to make a disposable lab SSTable visible.docker exec atlasmart-cass-1 nodetool flush atlasmart_writepath order_write_eventsdocker exec atlasmart-cass-1 nodetool tablestats atlasmart_writepath.order_write_events# Version-dependent diagnostic tools can inspect SSTable contents, but exact file paths# and tool output vary. First locate data files rather than hard-coding a table UUID.docker exec atlasmart-cass-1 nodetool datapaths atlasmart_writepath.order_write_eventsdocker exec atlasmart-cass-1 sh -lc "find /var/lib/cassandra/data/atlasmart_writepath -type f -name '*Data.db' -printf '%p\\n' 2>/dev/null | sort"

Do not lower gc_grace_seconds just to make a demo shorter. Tombstone safety depends on repair and failure windows. “I deleted it and SELECT no longer returns it” proves client-visible reconciliation, not immediate physical purge across all replicas and SSTables.

5. Production judgment

Timestamp ownership should be documented alongside retry/idempotency rules. Client-generated timestamps can be useful but turn clock quality and request replay ordering into data-correctness dependencies. Monitor tombstone scan warnings, read latency, partition size, TTL/delete rates, repair completion, clock synchronization, write timeouts, and application retry storms together.

Verification checklist

• WRITETIME(status) proves the higher-timestamp value won.
• The older timestamp mutation does not replace the newer overlapping cell.
• TTL() visibly decreases on the expiring fixture.
• A cell delete becomes invisible to SELECT without pretending the old SSTable bytes vanished immediately.
• gc_grace_seconds remains unchanged.

Summary and next bridge

Cassandra’s write path carries more than values: timestamps decide winners, TTL schedules expiry, and tombstones preserve deletion across immutable files and replicas. The final lesson combines these storage semantics with tracing and controlled replica failure so every acknowledgment is related to a concrete consistency requirement and recovery path.

Authoritative references

• Apache Cassandra 5.0 — Storage Engine / write path, commit log, memtables, flushes
• Apache Cassandra 5.0 — Hinted handoff
• Apache Cassandra CQL — DML, TIMESTAMP, TTL, WRITETIME
• nodetool tablestats
• nodetool listpendinghints
• Apache Cassandra native protocol — write timeout/failure metadata
• Apache Cassandra downloads and current releases
• Apache Cassandra Java Driver