Why CAP Does Not Mean Pick Two During Normal Operation

Learning outcomes

After Lesson 1, AtlasMart’s architecture review contains a dangerous sentence: “inventory is CP, catalog is AP, so we already picked two.” That sentence compresses too much. Healthy networks allow many systems to answer requests both consistently and successfully; partitioned networks trigger the theorem’s dilemma; and one system can expose different semantics for different operations. This lesson replaces classification slogans with a state-and-operation view.

1. Why the triangle picture causes trouble

The famous triangle invites a static product-classification question: “Which two does this database choose?” The formal result is conditional. When the network prevents replicas from communicating, a service cannot guarantee both linearizable consistency and availability for all requests. When there is no partition, that impossibility does not force a tradeoff between those two properties. A leader can answer quickly, replicas can be synchronized, and requests can succeed.

2. Normal operation and partitioned operation are different states

This table does not imply that catalog reads are “AP forever” or that inventory is “CP forever.” It says AtlasMart has two different business contracts. A healthy system may meet both contracts simultaneously. Under failure, each contract specifies its own degraded behavior.

3. CAP C and ACID C are different ideas

Atomicity, Consistency, Isolation, Durability (ACID) describes transaction properties. Its “consistency” means a transaction moves the database from one state satisfying declared integrity rules to another such state, assuming those rules and transactions are correctly designed. CAP’s “consistency” is about the visibility/order of operations across replicas in the theorem’s model. A system can have ACID transactions on one node yet return stale replicated reads elsewhere; conversely, a linearizable register can expose a single ordered value without implementing general multi-row ACID transactions.

4. Per-operation semantics beat product labels

AtlasMart’s same data platform might expose a local stale-tolerant product lookup, a quorum/leader-coordinated inventory mutation, and a write-any telemetry append. The relevant design surface includes request routing, read/write consistency settings, session guarantees, region ownership, and conflict policy. Saying “the database is AP” can hide that one endpoint is deliberately unavailable on an isolated side while another remains useful.

GET /catalog/sku-42
  partitioned region: serve local version, include freshness age

POST /inventory/sku-42/reservations
  partitioned non-owner region: 503 cannot establish reservation authority

POST /telemetry/click
  partitioned region: accept event with idempotency key; reconcile later

5. Deliberately wrong approach — configure from the label

Suppose an engineer writes “we chose AP because checkout must stay online” and therefore permits every region to reserve the last unit independently. The availability goal is real, but the chosen mechanism ignores the inventory invariant. Another engineer writes “we chose CP” and therefore sends every catalog browse across an ocean to the leader, adding latency for a path whose users would tolerate a few seconds of staleness. Both decisions are cargo-cult CAP: they optimize a label rather than the operation.

The repair is to write failure contracts per operation: what can be stale, what may be rejected, what may be queued, what can be compensated, and what must remain single-owner. Those contracts later become tests and SLOs.

6. AtlasMart lab — same system, different operations

from dataclasses import dataclass

@dataclass(frozen=True)
class Operation:
    name: str
    invariant: str
    partition_policy: str

ops = [
    Operation("browse catalog", "stale price tolerated briefly", "serve local replica"),
    Operation("reserve last unit", "stock must not go below zero", "refuse side without authority"),
    Operation("write click metric", "eventual count is acceptable", "accept locally and reconcile"),
]

for partitioned in (False, True):
    print("PARTITION" if partitioned else "HEALTHY")
    for op in ops:
        if not partitioned:
            result = "serve normally; network can coordinate"
        else:
            result = op.partition_policy
        print(f"- {op.name:18} | {result}")
    print()

print("CAP C is a single-copy/linearizability-style visibility property in the theorem.")
print("ACID C means a transaction preserves declared database/application invariants.")

Expected output

HEALTHY
- browse catalog     | serve normally; network can coordinate
- reserve last unit  | serve normally; network can coordinate
- write click metric | serve normally; network can coordinate

PARTITION
- browse catalog     | serve local replica
- reserve last unit  | refuse side without authority
- write click metric | accept locally and reconcile

CAP C is a single-copy/linearizability-style visibility property in the theorem.
ACID C means a transaction preserves declared database/application invariants.

The healthy branch is the important corrective: no theorem requires the service to throw away either consistency or availability when required communication is working. The partition branch shows why the choice belongs to the operation and invariant.

7. Production judgment and bridge

Use CAP as a fault-condition theorem, not a procurement quadrant. Architecture reviews should capture the exact object/operation, fault model, authority assumptions, client contract, and recovery behavior. Tests should include healthy operation, minority isolation, asymmetric reachability, and healing. Metrics should distinguish successful-but-stale responses from rejected-for-safety responses rather than flattening both into one availability number.

The next lesson extends the reasoning beyond faults. Even when every node can communicate, coordinating across distance costs time. PACELC gives a compact lens for that healthy-network latency-versus-consistency tension.

Authoritative references

• Gilbert & Lynch — CAP theorem formalization — the conditional impossibility result under partition
• Herlihy & Wing — Linearizability — clarifies the consistency notion commonly used when explaining CAP
• Daniel Abadi — Consistency Tradeoffs in Modern Distributed Database System Design — explains why CAP alone is not enough to characterize normal-operation tradeoffs