Tunable Consistency: Per-Operation Choices and Their Failure/Latency Consequences

Learning outcomes

AtlasMart does not have one universal read/write path. Product browse traffic may prefer low latency; checkout inventory needs a much stronger freshness/correctness contract. Tunable consistency means the client or operation can request different acknowledgement/response thresholds. That flexibility is useful only when the team understands the failure and latency consequences of each choice.

1. A consistency level is an operation contract

For an operation over replicated state, the selected level decides how much evidence the coordinator must collect before completing. On writes, that may be acknowledgements from one, a local majority, a global majority, or all eligible replicas. On reads, it may be one local copy, multiple digests/values, or another product-specific protocol. The selected level therefore changes both the success condition and the latency path.

A low threshold generally gives the coordinator more ways to succeed when replicas are slow or unavailable. But if replicas diverge, a low-threshold read is more likely to land on a stale copy. A high threshold asks for more corroboration, but can turn one slow/unreachable replica into a tail-latency spike or an unavailable operation.

2. Latency follows the slowest required responder

Suppose A responds in 3 ms, B in 8 ms, and a remote C in 72 ms. A ONE-like operation can complete after A. A two-response quorum-like operation completes after B if A and B are eligible. An ALL-like operation cannot complete before C. These are controlled numbers for reasoning, not benchmark predictions.

The same pattern appears in healthy steady state as PACELC reasoning from Chapter 03: even without a network partition, stronger coordination can add network and queueing delay. Tail latency matters more than averages because a quorum waits for enough individual tails to arrive.

3. Read and write levels form a policy pair

Choosing a read level without the write level is incomplete. If writes acknowledge locally on one replica and reads also accept one arbitrary replica, stale reads are unsurprising. If both operations use intersecting strict quorums with compatible version semantics, freshness improves. If a business rule cannot tolerate an oversell, the team should not “tune” away the serialization/ownership mechanism protecting inventory merely to lower read latency.

Also separate availability from success rate during a particular outage. An ALL-like operation can be perfectly reliable in healthy conditions yet intentionally fail during one-replica loss. That can be the correct choice for a strong invariant if the business prefers an explicit failure to an ambiguous result.

4. Deliberately wrong approach — one global level for every AtlasMart request

Setting every operation to ALL-like makes browse and profile reads inherit the slowest replica and fail during routine maintenance. Setting everything to ONE-like makes checkout depend on whichever replica happens to answer first. Both choices ignore the application’s different invariants.

The repair is an operation matrix. For each path, record authoritative owner, tolerated staleness, write conflict semantics, failure mode, required response set, latency SLO, and fallback behavior. Review that matrix when topology, replication factor, or region placement changes.

5. AtlasMart lab — latency, stale state, and unavailability by threshold

from dataclasses import dataclass

@dataclass(frozen=True)
class Replica:
    name: str
    region: str
    latency_ms: int
    version: int
    reachable: bool = True

replicas = [
    Replica("A", "east", 3, 12),
    Replica("B", "east", 8, 12),
    Replica("C", "west", 72, 11),
]

levels = {
    "ONE-like": 1,
    "QUORUM-like": 2,
    "ALL-like": 3,
}

for name, needed in levels.items():
    reachable = sorted([r for r in replicas if r.reachable], key=lambda r: r.latency_ms)
    if len(reachable) < needed:
        print(name, "UNAVAILABLE")
        continue
    chosen = reachable[:needed]
    latency = max(r.latency_ms for r in chosen)
    observed = max(chosen, key=lambda r: r.version).version
    print(name, "acks/responses=", [r.name for r in chosen],
          "latency_ms=", latency, "observed_version=", observed)

print("\nFAILURE: B becomes unreachable")
replicas2 = [Replica("A", "east", 3, 12), Replica("B", "east", 8, 12, False), Replica("C", "west", 72, 11)]
for name, needed in levels.items():
    reachable = sorted([r for r in replicas2 if r.reachable], key=lambda r: r.latency_ms)
    status = "SUCCESS" if len(reachable) >= needed else "UNAVAILABLE"
    print(name, status, "reachable=", [r.name for r in reachable])

print("\nLOCAL-ONE-LIKE READ")
local = min([r for r in replicas if r.region == "west"], key=lambda r: r.latency_ms)
print("nearest local response", local.name, "version", local.version, "latency_ms", local.latency_ms)
print("fast and available does not mean freshest")
print("latencies are deterministic teaching inputs, not database benchmark results")

Verification checklist

• ONE-like completes using the fastest eligible responder.
• QUORUM-like waits for two responders and returns the highest version among those sampled by the toy reconciler.
• ALL-like includes the remote 72 ms path and becomes unavailable when one replica is unreachable.
• A west-local read can be fast yet return version 11 while east holds version 12.
• The displayed latency values are labeled model inputs, not product benchmark results.

6. Production judgment

Observe per-level latency histograms, timeouts, unavailable errors, replica lag/divergence, cross-region traffic, and retry volume. Capacity plans must include failure mode: a local quorum that is comfortable with three healthy replicas can saturate the remaining two during maintenance. Security and tenant authorization must remain identical regardless of consistency level; “eventual” is never permission to expose unauthorized state.

The next lesson explains how replicas that missed writes are discovered and repaired rather than merely tolerated.

Authoritative references

• Apache Cassandra 5.0 — cqlsh consistency levels — current named-level implementation example
• Apache Cassandra — Guarantees — current architecture/consistency discussion
• Dynamo: Amazon’s Highly Available Key-value Store — primary N/R/W tunability reference