SERIAL / LOCAL_SERIAL Consistency, Replica Scope, Latency, and Failure Tradeoffs

Learning outcomes

AtlasMart expands from one region to two datacenters. Engineers copy the single-DC setting LOCAL_SERIAL everywhere and assume it means the same thing as SERIAL. The decision actually changes which replicas participate in the Paxos phase and therefore changes WAN latency/failure scope.

Core terms for this chapter

A lightweight transaction (LWT) is Cassandra's conditional mutation mechanism. It uses the Paxos consensus protocol so competing operations on the same logical Paxos scope can agree on one ordered outcome. Compare-and-set (CAS) means “apply this mutation only if the current value satisfies the condition.” A conditional mutation is CQL such as INSERT ... IF NOT EXISTS or UPDATE ... IF column = value. Linearizable means successful operations appear to occur in a single real-time-compatible order within the guarantee's documented scope; it is stronger than ordinary eventual consistency.

A coordinator is the Cassandra node handling one request. A replica is a node that stores the partition according to the keyspace replication strategy. A partition groups rows by partition key and maps through the partitioner to a token; the token determines natural replicas. RF is replication factor. A regular CL controls the data/learn phase of the operation. SERIAL and LOCAL_SERIAL are serial consistency levels that control the Paxos phase. A ballot is the proposal identity/order used by Paxos rounds. Contention occurs when concurrent conditional operations compete for the same Paxos state. A hot partition is a partition receiving disproportionately high request volume. A retry repeats an operation after an error or timeout; for LWT, an ambiguous outcome must be reconciled before blind retry. SSTables are immutable on-disk table files, while compaction rewrites SSTables. repair is Cassandra's anti-entropy process and is separate from Paxos agreement. Storage-Attached Indexing (SAI) and vector search can help locate rows but do not create uniqueness or cross-row transactional invariants.

1. Two consistency decisions in one LWT

For a conditional mutation, Cassandra needs a serial consistency level for Paxos and a regular consistency level for the learned/data mutation. In cqlsh those are configured independently with SERIAL CONSISTENCY ... and CONSISTENCY .... In the Java Driver they are separate statement attributes. If the native-protocol serial consistency is omitted for a conditional query, the protocol default is SERIAL; production applications should make the intended scope explicit instead of relying on an unseen default.

2. Make the separation explicit in cqlsh

docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 java -versiondocker exec atlasmart-cass-1 cqlsh -e "SELECT cluster_name, data_center, rack, release_version FROM system.local;"docker exec atlasmart-cass-1 cqlsh -e "SELECT peer, data_center, rack, release_version FROM system.peers_v2;"# Continue only when all three nodes are UN in dc1.# If the course cluster is absent, recreate it with the same Chapter 01 conventions# and pinned cassandra:5.0.9 image before running this chapter.

CREATE KEYSPACE IF NOT EXISTS atlasmart_lwtWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_lwt.inventory_guard (    sku text PRIMARY KEY,    available int,    reservation_owner text,    revision int) WITH compaction = {'class':'UnifiedCompactionStrategy'};CREATE TABLE IF NOT EXISTS atlasmart_lwt.unique_claim (    claim_type text,    claim_value text,    owner_id text,    created_at timestamp,    PRIMARY KEY ((claim_type, claim_value))) WITH compaction = {'class':'UnifiedCompactionStrategy'};CREATE TABLE IF NOT EXISTS atlasmart_lwt.order_projection (    order_id text PRIMARY KEY,    customer_id text,    state text,    updated_at timestamp) WITH compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;SERIAL CONSISTENCY LOCAL_SERIAL;INSERT INTO atlasmart_lwt.inventory_guard(sku, available, reservation_owner, revision)VALUES ('sku-42', 10, 'NONE', 0);

CONSISTENCY LOCAL_QUORUM;UPDATE atlasmart_lwt.inventory_guard SET available=9, reservation_owner='seed', revision=1 WHERE sku='sku-42';SERIAL CONSISTENCY LOCAL_SERIAL;UPDATE atlasmart_lwt.inventory_guardSET available=8, reservation_owner='order-local', revision=2WHERE sku='sku-42'IF revision=1;SERIAL CONSISTENCY SERIAL;UPDATE atlasmart_lwt.inventory_guardSET available=7, reservation_owner='order-global', revision=3WHERE sku='sku-42'IF revision=2;

On a single-DC cluster, these commands primarily prove syntax and phase separation; they do not prove a WAN latency difference. The replica scope becomes materially different only when the keyspace has replicas in multiple DCs.

3. Java Driver: keep normal and serial CL visible in code/profile

import static com.datastax.oss.driver.api.core.DefaultConsistencyLevel.*;import com.datastax.oss.driver.api.core.cql.SimpleStatement;SimpleStatement claim = SimpleStatement.builder(    "INSERT INTO atlasmart_lwt.unique_claim " +    "(claim_type, claim_value, owner_id, created_at) " +    "VALUES (?, ?, ?, toTimestamp(now())) IF NOT EXISTS")    .addPositionalValues("email", "a@example.test", "customer-42")    .setConsistencyLevel(LOCAL_QUORUM)      // learn/data phase    .setSerialConsistencyLevel(LOCAL_SERIAL) // Paxos phase    .setIdempotence(false) // do not assume a timed-out CAS is safe to repeat blindly    .build();var row = session.execute(claim).one();boolean applied = row != null && row.getBoolean("[applied]");

The driver statement is immutable: setter methods return a new statement. For high-value invariants, place these attributes in a named execution profile or a well-reviewed statement builder so the regular and serial CL do not disappear into application defaults.

4. Optional local two-DC experiment

To reproduce the scope difference without a cloud account, extend the Docker cluster with three additional Cassandra nodes labeled dc2/rack1..rack3, then alter only the disposable keyspace to dc1=3, dc2=3 and run repair so replicas exist in both DCs. This requires materially more RAM/disk; if your machine cannot host six Cassandra JVMs, use the arithmetic/failure table as a deterministic simulation and do not pretend you measured WAN behavior.

CREATE KEYSPACE IF NOT EXISTS atlasmart_lwt_multidcWITH replication = {'class':'NetworkTopologyStrategy','dc1':3,'dc2':3};CREATE TABLE IF NOT EXISTS atlasmart_lwt_multidc.scope_probe (    k text PRIMARY KEY, v text) WITH compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY EACH_QUORUM;INSERT INTO atlasmart_lwt_multidc.scope_probe (k,v) VALUES ('scope-1','baseline');DESCRIBE KEYSPACE atlasmart_lwt_multidc;

# 1) Verify all six nodes UN and create atlasmart_lwt_multidc after both DCs are healthy.# 2) Pause all three dc2 containers (names depend on your Chapter 14 optional topology).# 3) With a dc1 coordinator, test:#    SERIAL CONSISTENCY LOCAL_SERIAL + CONSISTENCY LOCAL_QUORUM#    versus SERIAL CONSISTENCY SERIAL + CONSISTENCY QUORUM.# 4) Capture unavailable/timeout behavior and trace/latency.# 5) Unpause dc2, wait for UN, verify the scope_probe row, then drop atlasmart_lwt_multidc.# 6) The dedicated keyspace keeps the mandatory atlasmart_lwt fixture unchanged.# Never use host firewall changes or a real WAN outage for this course lab.

SELECT * FROM atlasmart_lwt_multidc.scope_probe WHERE k='scope-1';DROP KEYSPACE IF EXISTS atlasmart_lwt_multidc;

5. Verification checklist

• Record normal CL and serial CL independently for every conditional operation.
• Explain the single-DC limitation of the mandatory lab.
• If using the optional topology, document dc1/dc2 node/rack names, RF per DC, repair completion, and resource limits.
• Verify all paused nodes are restored and UN after the experiment.
• Do not claim LOCAL_SERIAL is “stronger” or “weaker” in the abstract; explain scope and invariant requirements.

Production judgment

LWT is an invariant tool, not a “strong consistency” checkbox for every write. Before using it, define the exact invariant, its partition/key scope, contender cardinality, expected concurrency, RF and datacenter placement, regular and serial CL, failure behavior, acceptable p95/p99 latency, retry/reconciliation rules, and how a timed-out outcome will be discovered. Measure CASRead/CASWrite latency, timeouts, failures, unavailables, condition-not-met counts, and contention histograms alongside ordinary request latency, CPU, JVM garbage collection, network, disk, compaction, repair state, and hot-key distribution.

Do not infer a universal LWT throughput number from this laptop lab. SAI/vector indexes do not make a search result safe as a uniqueness lock; security/tenant boundaries require authorization and data-model controls, not LOCAL_SERIAL. Managed Cassandra services can constrain JMX, Paxos variants, or topology settings, so translate the same invariant and evidence model to the service's supported telemetry. Migration and rollback must account for application semantics: replacing a normal write with LWT can change latency and availability, while removing LWT can silently weaken an invariant. Lesson 4 deliberately creates contention on a hot CAS key and then redesigns the invariant so fewer operations need Paxos at all.

Summary and next bridge

LWT has two consistency choices: serial consistency for Paxos and regular consistency for the learned mutation. In multi-DC deployments the difference between local and global scope can dominate latency and availability. Next, make contention visible and redesign hot invariants to reduce CAS frequency.

Authoritative references

Use these current official sources as the version-sensitive source of truth. Re-check them when regenerating this chapter because Paxos variants, driver behavior, metrics, and operational recommendations can evolve.

• Apache Cassandra downloads — 5.0.9 and Java Driver 4.19.3
• Cassandra guarantees — LWT and linearizable consistency
• cqlsh consistency and serial consistency
• cassandra.yaml — paxos_variant
• Cassandra monitoring metrics — CASRead/CASWrite
• nodetool proxyhistograms — CAS latency distributions
• Java Driver statement attributes — regular and serial consistency
• Java Driver retries and idempotence
• Java Driver query timestamps — LWT restriction