Linearizability, Leader Leases, Fencing Tokens, and Metadata/Control-Plane Consensus

1. The problem: the control plane elected B, but A wakes up late

AtlasMart's control plane has consensus-committed worker-A as shard owner with fencing token 42. A pauses for longer than its lease. The majority later assigns ownership to worker-B with token 43. B starts writing. Then A resumes with stale memory that says “my lease was valid.” If the downstream database accepts A's write, consensus correctly chose B but the application still suffers stale-owner corruption.

This is the difference between deciding authority and enforcing authority. A fencing token is a monotonically increasing generation attached to operations. The protected resource remembers the highest generation it has accepted and rejects lower ones. Fencing converts stale ownership into an observable rejected write.

2. Linearizability gives a real-time contract

Linearizability requires each operation to appear to take effect atomically at some instant between invocation and response, while respecting real-time precedence: if operation X completes before Y begins, the linearized history places X before Y. Consensus is a common mechanism for implementing a linearizable replicated state machine, but “uses consensus” and “every API read is linearizable” are not synonymous; the read path must satisfy the required protocol too.

For metadata such as shard assignments, lock ownership, schema generations, or feature/config epochs, linearizability is often valuable because clients need one current answer rather than divergent eventually consistent ownership.

3. Leases make time part of the safety argument

A lease grants authority until an expiry condition. Lease safety depends on timing assumptions: which clock measures expiry, how clock drift/suspension is bounded, whether the holder can act after losing contact, and whether the grantor prevents overlapping leases. Monotonic clocks help with local elapsed time but do not magically establish global time agreement.

Fencing is complementary. Even if an old process resumes after a pause and mistakenly believes its lease is valid, a downstream service that rejects token 42 after seeing 43 prevents stale mutation. A lock API that returns only “acquired=true” without a generation the resource can verify may be insufficient for unsafe external side effects.

4. AtlasMart lab: lock without fencing vs fenced resource

The lab first demonstrates corruption when the downstream resource trusts whichever worker writes last. The corrected path advances the control-plane generation to 43 and makes the storage layer reject A's old token 42.

control_plane = {
    "term": 12,
    "owner": "worker-A",
    "fence": 42,
    "lease_until_ms": 1000,
}
storage = {"highest_fence": 42, "value": "price=100"}

print("INITIAL LINEARIZABLE METADATA DECISION")
print("control plane:", control_plane)
print("storage:", storage)

print("\nA PAUSES; MAJORITY REASSIGNS AFTER LEASE/FAILURE POLICY")
control_plane.update(term=13, owner="worker-B", fence=43, lease_until_ms=2000)
print("new control-plane state:", control_plane)

print("\nBROKEN DOWNSTREAM: LEASE/LOCK WITHOUT FENCING")
naive_storage = {"value": "price=100"}
# B writes first under the new ownership.
naive_storage["value"] = "price=90 by B"
# A resumes with stale local belief and overwrites it.
naive_storage["value"] = "price=80 by stale A"
print("naive final value:", naive_storage["value"])
print("stale owner corrupted newer state:", "stale A" in naive_storage["value"])

print("\nFENCED DOWNSTREAM")
def fenced_write(token, value):
    if token < storage["highest_fence"]:
        return False
    storage["highest_fence"] = token
    storage["value"] = value
    return True

print("B token 43 accepted:", fenced_write(43, "price=90 by B"), storage)
print("stale A token 42 rejected:", not fenced_write(42, "price=80 by A"), storage)
print("final value:", storage["value"])
print("note: the fencing token must be checked by the resource being protected")

5. Control plane and data plane can have different semantics

The control plane holds relatively small but correctness-critical metadata: membership, leader/shard assignment, schemas, routing generations, or configuration. The data plane carries high-volume user records. A system may use Raft/Paxos for control-plane decisions while using asynchronous replication, leader/follower replication, tunable quorums, or another mechanism for user data. This split keeps consensus load focused on the decisions that require a single authoritative order.

6. Current implementation example: etcd 3.7.1

As of August 2026, etcd's current 3.7 branch has patch release 3.7.1 and is Apache-2.0 licensed. Its v3.7 API documentation distinguishes default linearizable operations from lower-latency serializable reads that may be stale relative to quorum. That is exactly the lesson's point: even inside one consensus-backed product, the client-visible guarantee depends on the operation/read mode, not on the product label alone.

The July 2026 patch release also fixed authorization-boundary issues in watch responses. If a control plane stores tenant/security-sensitive metadata, correctness includes authentication, TLS, RBAC boundaries, and patch management—not just consensus safety.

7. Production judgment and bridge

Use linearizable control-plane decisions when stale metadata can create dual ownership, invalid configuration, or unsafe coordination. Keep the control plane small enough that consensus latency and storage are predictable. Track lease renewal margin, observed clock anomalies, term/revision changes, rejected fencing generations, stale-client error rate, leader-election latency, and authorization failures. Test process pauses—not just clean crashes—because pauses are where unfenced locks often fail.

The final lesson makes the architectural boundary explicit: a database may use consensus for membership/leader metadata while user writes follow a different replication path. You must trace the path of the specific operation before claiming it is consensus-committed or linearizable.

References

Foundational claims use primary research where practical. Product documentation is used only as a current implementation example and is not required for the mandatory labs.

• Herlihy & Wing — Linearizability — Primary definition of linearizability and real-time operation ordering.
• etcd v3.7 — API guarantees — Current official example distinguishing default linearizable operations from serializable reads that may be stale.
• etcd — July 23, 2026 patch releases — Official v3.7.1 release/security context for the optional implementation example.
• etcd GitHub repository — Official source and Apache License 2.0 licensing information.
• Ongaro dissertation — Raft — Detailed Raft treatment including reads, timing, leadership, and practical implementation considerations.