Commit Log for Durability and Memtables for In-Memory Sorted State

Learning outcomes

AtlasMart has just proved that a coordinator waits for replica acknowledgments. Now the platform team asks the harder question: what must happen inside an acknowledging replica before it responds? Cassandra’s write path deliberately avoids in-place random updates. A replica records the mutation in a commit log for crash recovery and applies it to the table’s in-memory memtable; the immutable SSTable arrives later during flush.

1. Two writes happen locally for one logical mutation

On each replica that accepts a normal table mutation, Cassandra appends the mutation to the node’s commit log and updates the table’s memtable. The commit log is sequential, append-oriented recovery data shared by many tables; the memtable is an in-memory, table-specific sorted structure that serves as the mutable representation of recent partitions.

This pairing is why Cassandra can acknowledge a write without performing an in-place update to an SSTable. SSTables are immutable. If the process crashes before a dirty memtable is flushed, Cassandra can replay eligible commit-log mutations during startup and reconstruct the in-memory state.

2. “Committed” is configuration-sensitive: periodic versus batch sync

Cassandra 5.0 documents commitlog_sync modes with different acknowledgment semantics. In periodic mode—the documented default in current 5.0 storage-engine configuration—the server may acknowledge after the mutation reaches the commit-log buffer, with fsync occurring periodically. In batch mode Cassandra does not acknowledge until the commit log has been fsynced. Therefore a production durability statement must name the mode, filesystem/device assumptions, RF, CL, and correlated-failure model.

3. Lab: observe commit-log configuration and memtable state

# Verify the shared lab if it already exists.docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool status# Standalone recreation path. Skip resources that already exist.docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 answers before starting peers.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9# Wait until all three nodes show UN in dc1 before continuing.docker exec atlasmart-cass-1 nodetool status# Open cqlsh on node 1 for the SQL/CQL blocks that follow.docker exec -it atlasmart-cass-1 cqlsh

CREATE KEYSPACE IF NOT EXISTS atlasmart_writepathWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_writepath.order_write_events (    order_id text,    event_time timeuuid,    status text,    source text,    note text,    expires_at timestamp,    PRIMARY KEY ((order_id), event_time)) WITH CLUSTERING ORDER BY (event_time DESC)  AND compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;SELECT keyspace_name, replicationFROM system_schema.keyspacesWHERE keyspace_name = 'atlasmart_writepath';

# Configuration file location is from the official Docker image layout.docker exec atlasmart-cass-1 sh -lc "grep -E '^[[:space:]]*commitlog_sync:|^[[:space:]]*commitlog_sync_period:' /etc/cassandra/cassandra.yaml || true"# Inspect current commit-log files and disk footprint. Exact names/sizes vary.docker exec atlasmart-cass-1 sh -lc 'du -sh /var/lib/cassandra/commitlog; ls -lh /var/lib/cassandra/commitlog | tail -n +1'# Baseline table statistics before adding this lesson's rows.docker exec atlasmart-cass-1 nodetool tablestats atlasmart_writepath.order_write_events

CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-2001',now(),'CREATED','checkout-api','memtable evidence A');INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-2002',now(),'CREATED','checkout-api','memtable evidence B');INSERT INTO atlasmart_writepath.order_write_events(order_id,event_time,status,source,note)VALUES ('order-2003',now(),'CREATED','checkout-api','memtable evidence C');

docker exec atlasmart-cass-1 nodetool tablestats atlasmart_writepath.order_write_eventsdocker exec atlasmart-cass-1 sh -lc 'du -sh /var/lib/cassandra/commitlog'# In tablestats, record memtable_cell_count, memtable_data_size,# memtable_switch_count, pending_flushes, local_write_count/latency, and sstable_count.# Do not copy sample numbers: capture your own values.

Because RF=3, a specific row may have been coordinated by one node and replicated to all three. Local tablestats is node-local evidence; compare nodes if you need replica-level visibility. A small fixture can be affected by background flushes, so treat “memtable count is exactly N” as a hypothesis, not a guaranteed constant.

4. Flush converts mutable memory into immutable SSTables

A manual nodetool flush is appropriate here only because the cluster and keyspace are disposable and the lesson needs deterministic evidence. It is not routine performance tuning. Flush writes the current memtable to an SSTable, creates associated index/metadata components, and allows commit-log segments to become recyclable only when all relevant dirty data in those shared segments is covered.

# Record the data directories first.docker exec atlasmart-cass-1 nodetool datapaths atlasmart_writepath.order_write_events# Disposable-lab-only deterministic transition.docker exec atlasmart-cass-1 nodetool flush atlasmart_writepath order_write_events# Observe the new state.docker exec atlasmart-cass-1 nodetool tablestats atlasmart_writepath.order_write_eventsdocker exec atlasmart-cass-1 sh -lc "find /var/lib/cassandra/data/atlasmart_writepath -type f -name '*Data.db' -printf '%p %s bytes\\n' 2>/dev/null | sort"docker exec atlasmart-cass-1 sh -lc 'du -sh /var/lib/cassandra/commitlog' 

Expect the active memtable for that table to become smaller or reset and the SSTable count/files to change. Do not expect every commit-log file to disappear: commit-log segments are shared across tables and their reclamation depends on all mutations covered by each segment, not only this table.

5. Why disabling the commit log is not a casual latency trick

Cassandra exposes durable_writes at the keyspace level. Turning durability off changes the recovery contract and is not equivalent to “use a faster disk.” A memtable exists only in process memory until flush; without the recovery log, process or host failure before flush can discard acknowledged data. This chapter does not disable durability because the failure mechanism is obvious without risking confusing recovery state.

In production, pair commit-log mode with dedicated storage design, filesystem/device behavior, write SLOs, RF/CL, node-failure correlation, and recovery tests. Watch local write latency, commit-log device latency, pending flushes, memtable memory, dropped mutations, JVM/GC pressure, and commit-log space together.

Verification checklist

• You recorded the actual commitlog_sync setting from the running image.
• You observed nonzero write activity in tablestats.
• You captured memtable statistics before the manual flush.
• You observed SSTable/data-directory evidence after the flush.
• You can explain why commit-log disk usage need not drop immediately after flushing one table.

Summary and next bridge

Replica acknowledgment is backed by a local commit-log/memtable transition, but fsync guarantees depend on commit-log sync mode and SSTables arrive later. Next we examine the triggers that switch memtables, the disk transition itself, and the backpressure signals that appear when flushing cannot keep up with incoming writes.

Authoritative references

• Apache Cassandra 5.0 — Storage Engine / write path, commit log, memtables, flushes
• Apache Cassandra 5.0 — Hinted handoff
• Apache Cassandra CQL — DML, TIMESTAMP, TTL, WRITETIME
• nodetool tablestats
• nodetool listpendinghints
• Apache Cassandra native protocol — write timeout/failure metadata
• Apache Cassandra downloads and current releases
• Apache Cassandra Java Driver