Define Per-Query Consistency from Business Invariants Instead of One Cluster-Wide Habit

Learning outcomes

AtlasMart has product browsing, order-state reads, payment/idempotency records, inventory reservations, analytics counters, and operational control data. Giving every request the same consistency level either wastes availability/latency or fails important invariants. This lesson builds a per-query contract and encodes it in the application driver.

docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Wait until node 1 reports UN before starting peers.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9# Continue only after all three nodes are UN.docker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 java -versiondocker exec atlasmart-cass-1 cqlsh -e "SHOW VERSION"

CREATE KEYSPACE IF NOT EXISTS atlasmart_consistencyWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_consistency.order_status_by_id (    order_id text PRIMARY KEY,    status text,    version int,    updated_at timestamp) WITH compaction = {'class':'UnifiedCompactionStrategy'};CREATE TABLE IF NOT EXISTS atlasmart_consistency.inventory_reservation_by_product (    product_id text,    reservation_id text,    customer_id text,    state text,    created_at timestamp,    PRIMARY KEY (product_id, reservation_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_consistency.order_status_by_id(order_id,status,version,updated_at)VALUES ('order-42','PAID',1,toTimestamp(now()));DESCRIBE KEYSPACE atlasmart_consistency;

Core terms used throughout this chapter

Replication factor (RF) is the number of replicas Cassandra is configured to keep for each partition in a datacenter. A replica stores a copy of a partition; the coordinator is the node handling one client request and is not a permanent leader. A consistency level (CL) is the per-request rule that says how many and which replicas must acknowledge a write or satisfy a read before the coordinator can return success. A datacenter (DC) is Cassandra's logical locality/failure-domain grouping; a rack is a smaller placement grouping within a DC. A quorum is a majority, calculated as floor(RF/2)+1 for the relevant replica scope. CQL means Cassandra Query Language; cqlsh is its shell. A Lightweight Transaction (LWT) is a conditional CQL operation using Paxos consensus; it has a serial phase and a regular learn/data phase. Availability here means whether enough replicas are alive/responding for the requested CL—not whether the cluster has any node alive. Staleness means a read can return an older replica version because the requested read CL did not necessarily intersect the replicas that acknowledged a prior write.

1. Start with the invariant, not the consistency label

These are candidate designs, not universal prescriptions. A different topology, RF, region failover model, regulator requirement, or user-visible freshness promise changes the answer.

2. Encode per-query CL in the Java driver

Apache Cassandra Java Driver 4.19.3 exposes normal and serial consistency on each statement. Statements are immutable: the setter returns a new statement instance. In a production application, define named execution profiles for common request classes so consistency, timeout, retry, speculative execution, and idempotency are reviewed together.

import com.datastax.oss.driver.api.core.CqlSession;import com.datastax.oss.driver.api.core.DefaultConsistencyLevel;import com.datastax.oss.driver.api.core.cql.SimpleStatement;// Catalog: availability-first, bounded staleness accepted.SimpleStatement catalogRead = SimpleStatement.builder(    "SELECT * FROM atlasmart_consistency.order_status_by_id WHERE order_id=?")    .addPositionalValue("order-42")    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_ONE)    .setIdempotent(true)    .build();// Order workflow: local quorum intersection for the stated freshness contract.SimpleStatement orderRead = catalogRead    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM);// Conditional reservation: ordinary learn phase + Paxos serial phase.SimpleStatement reserve = SimpleStatement.builder(    "INSERT INTO atlasmart_consistency.inventory_reservation_by_product " +    "(product_id,reservation_id,customer_id,state,created_at) " +    "VALUES (?,?,?,?,toTimestamp(now())) IF NOT EXISTS")    .addPositionalValues("sku-9","slot-2","customer-C","HELD")    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM)    .setSerialConsistencyLevel(DefaultConsistencyLevel.LOCAL_SERIAL)    .setIdempotent(false) // review semantics before enabling retries/speculation    .build();

3. Test invariants under failure, not just happy-path correctness

For every query class, write an acceptance test with the failure conditions the API promises to tolerate. At minimum: one replica down, stale replica after a missed write, local quorum unavailable, remote DC unavailable (if multi-DC), timeout/ambiguous write outcome, and concurrent LWT contention. Record the actual exception class, received/applied status, trace/metrics, and post-repair state.

CASE catalog_read:  RF: dc1=3  CL: LOCAL_ONE  promise: may be stale; should survive up to two replica losses if one local replica remains  test: pause two replicas -> read; restore -> repair -> verify equalityCASE order_status_read_after_write:  RF: dc1=3  write CL: LOCAL_QUORUM  read CL: LOCAL_QUORUM  promise: local quorum intersection under one replica loss  test: pause one replica -> write -> read -> restore -> repairCASE inventory_reservation:  RF: dc1=3  regular CL: LOCAL_QUORUM  serial CL: LOCAL_SERIAL  promise: only one concurrent IF NOT EXISTS wins  test: race two writers; assert exactly one applied=trueCASE remote_dc_outage (if dc2 exists):  compare LOCAL_QUORUM vs QUORUM/EACH_QUORUM  promise: only the explicitly local query class should remain available

4. Decision worksheet

Production judgment

Choose consistency from a business invariant, topology, RF, and failure budget—not from a cluster-wide slogan. Record which reads must observe which writes, whether the invariant is local to one DC or global, whether conditional uniqueness/compare-and-set is required, and what latency/availability degradation is acceptable when replicas or a whole DC are unavailable. Then test that contract with the real driver, routing policy, request timeout, retry/speculative-execution policy, idempotency classification, and representative network latency.

Consistency level does not replace durable commit-log/storage design, repair, backup/restore, security isolation, schema/partition design, compaction/tombstone management, or application-level idempotency. ALL is not “permanent durability”; LOCAL_ONE is not tenant isolation; a timeout is not proof that a write failed; and a successful weak write does not promise a subsequent weak read will be fresh. Storage-Attached Indexing (SAI) or vector search can add read work but do not redefine RF/CL arithmetic. Managed Cassandra services may restrict topology visibility or CL choices; verify provider semantics rather than assuming Apache Cassandra behavior is exposed unchanged. Chapter 15 now focuses on what happens when requests do not complete cleanly: hints, read repair, speculative retry, timeouts, unavailable errors, overload, and safe client retry behavior.

Summary and next bridge

Tunable consistency becomes engineering only when each request has an explicit invariant, RF/DC arithmetic, latency/availability target, retry/idempotency rule, and failure test. Chapter 15 continues with failure handling, where timeout, unavailable, hints, reconciliation, repair, and speculative execution must be kept distinct.

Authoritative references

Re-check these version-sensitive sources when regenerating this course.

• Apache Cassandra 5.0 downloads and Java Driver releases
• Dynamo architecture and tunable consistency semantics
• cqlsh CONSISTENCY and SERIAL CONSISTENCY commands
• Native protocol consistency-level codes
• Cassandra guarantees and LWT linearizable consistency
• Hints and eventual convergence
• Apache Cassandra Java Driver 4.19 consistency API
• Java Driver statement consistency attributes