Distinguish Consensus for Cluster Metadata from Replication of Application Data

1. The misleading architecture diagram

An AtlasMart architecture slide says “Raft-based database.” Engineers then assume a read from any replica after a successful write must return the new value. That conclusion is not justified by the word Raft. The first question is: what state does the consensus group actually replicate?

In this lesson's example, consensus commits only the mapping catalog-17 → leader A, epoch 7. Application data under that shard uses leader/follower replication with a configurable acknowledgement boundary. Leader A can acknowledge version 10 while followers B and C still hold version 9. A follower-local read can therefore be stale even though every control-plane node agrees perfectly on leader A.

2. Trace guarantees by operation path

For any database claim, draw the concrete path: client → router → metadata lookup → data owner → replication acknowledgements → client response → later read target. Ask which steps use consensus, which use quorum overlap, which are asynchronous, and which state is durable before acknowledgement. A single product can expose multiple consistency modes.

3. AtlasMart lab: metadata is current while data is stale

The control-plane mapping is already committed at metadata revision 104. The data plane then contains version 10 only on leader A while B and C remain version 9. Reading B demonstrates the exact false inference this lesson is designed to prevent.

metadata = {
    "epoch": 7,
    "shard": "catalog-17",
    "leader": "A",
    "replicas": ["A", "B", "C"],
    "metadata_commit": 104,
}
data = {
    "A": {"version": 10, "price": 90},
    "B": {"version": 9, "price": 100},
    "C": {"version": 9, "price": 100},
}

print("CONSENSUS-COMMITTED CONTROL PLANE")
print(metadata)
print("all clients can agree that A owns catalog-17 at epoch 7")

print("\nAPPLICATION DATA USES DIFFERENT REPLICATION SEMANTICS")
print("A accepted version 10 and acknowledged after its configured local durability boundary")
print("replication to B/C is still pending")
print("data replicas:", data)

print("\nCLIENT READS FOLLOWER B")
read = data["B"]
print("B returned:", read)
print("stale relative to leader A:", read["version"] < data["A"]["version"])
print("metadata consensus did NOT make this follower read linearizable")

print("\nREPLICATION CATCH-UP")
for replica in ["B", "C"]:
    data[replica] = dict(data["A"])
print("data replicas:", data)
print("converged:", len({(v['version'], v['price']) for v in data.values()}) == 1)

print("\nARCHITECTURE NOTE")
print("a different system may run Raft/Paxos per data shard and consensus-commit user writes")
print("therefore infer guarantees from the user-operation path, not from the mere presence of Raft/Paxos somewhere")

4. Deliberately wrong approach: market the strongest subsystem guarantee as the whole-system guarantee

A system may have linearizable metadata, strongly consistent schema changes, or consensus-based leader election while offering eventual/tunable consistency for ordinary records. Documentation and architecture reviews must state guarantees per operation, not per brand. Likewise, a “quorum write” is not automatically a consensus decision; Chapters 5–6 showed that replica quorums and consensus quorums solve related but different problems.

The safe correction is to publish a guarantee matrix: operation, routing path, acknowledgement set, durability boundary, read modes, failover behavior, stale-read allowance, maximum tolerated lag if bounded, and recovery/reconciliation path.

5. What if data shards also use consensus?

Many production systems do use Raft/Paxos-family protocols for each replicated data range/shard. Then a successful write can mean the shard's consensus log committed the command before acknowledgement, and linearizable reads may be available through leader/read-index/lease mechanisms depending on the implementation. But the performance envelope changes: every write normally pays quorum/leader coordination within that shard, hot shards can bottleneck, and cross-shard transactions require additional coordination.

Therefore the right statement is conditional: this user operation is consensus-committed by group G under policy P, not the database uses Raft.

6. Observability proves which path happened

Expose metadata revision/epoch, shard ID, data leader, write acknowledgement policy, commit/applied index where applicable, replica version/lag, read consistency mode, and client request ID in traces. During incident review, these fields answer whether the client hit stale routing, a lagging follower, a lost leadership term, or a genuinely committed data-log entry. “Consensus healthy” is not enough if the data plane is 30 seconds behind.

For backups and disaster recovery, separately test restoration of consensus metadata and application data. A perfectly restored control plane pointing at incomplete data is not a successful restore.

7. Security and operational boundaries

Consensus metadata is often extremely sensitive: membership, credentials/leases, encryption configuration, network endpoints, and tenant routing can create broad blast radius. Authenticate members, encrypt peer/client traffic, restrict administrative APIs, patch security issues, and audit changes. etcd's July 2026 patch release is a useful reminder that authorization bugs can exist even in a mature consensus-backed store; consensus safety does not imply access-control correctness.

8. Chapter synthesis and bridge to conflict resolution

Consensus gives a group a safe authoritative decision sequence under stated fault/timing assumptions. Raft expresses that through leaders, terms, and a replicated log; Paxos through ballots, promises, acceptors, and chosen values. Linearizability turns that ordered state into a real-time client contract when the operation path implements it. Fencing makes ownership robust against paused stale actors. Finally, architecture determines whether consensus protects only control-plane metadata or every application-data write.

Chapter 18 now assumes that distinction and studies what happens when application writes are allowed to occur concurrently without one total consensus order: lost updates, duplicate events, last-write-wins, application merges, conflict-free replicated data types (CRDTs), and idempotency.

References

Foundational claims use primary research where practical. Product documentation is used only as a current implementation example and is not required for the mandatory labs.

• Ongaro & Ousterhout — Raft — Primary source for replicated-log consensus and the distinction between committed and merely replicated entries.
• Lamport — Paxos Made Simple — Primary source for Paxos chosen-value safety and quorum intersection.
• etcd v3.7 — API guarantees — Current product example showing that read consistency mode changes client-visible guarantees even within one Raft-backed store.
• etcd v3.7.1 patch release — Current version/security context; reinforces that authorization and consensus correctness are separate concerns.
• Herlihy & Wing — Linearizability — Primary definition used to evaluate user-operation histories rather than product labels.