Multi-Leader Replication: Geographic Writes, Conflict Risk, and Topology Complexity

Learning outcomes

AtlasMart now operates in North America and Europe. Sending every profile edit across the Atlantic to one leader adds avoidable latency, so the team considers one writable leader per region. The price of local writes is that the same logical object can be changed concurrently in disconnected regions. Multi-leader replication therefore needs conflict semantics, loop prevention, convergence, and ownership discipline.

1. Multi-leader moves the write-latency problem into conflict space

With multiple writable leaders, a US client can commit to a US leader while a European client commits to an EU leader. If the leaders are connected, replication propagates both changes. If the link is unavailable, each side continues building local history. When communication returns, histories must be reconciled.

This design can improve geographic write latency and tolerate some forms of regional isolation, but it does not remove coordination cost. It relocates cost to conflict detection/resolution, topology management, loop prevention, schema compatibility, and operational repair.

2. Concurrent does not mean “later timestamp loses”

Chapter 04 showed why physical timestamps do not reliably encode causality. In the profile example, both leaders edit version 7 during a partition: US updates email, EU updates phone. These writes are concurrent relative to the shared base. A last-write-wins (LWW) rule can select one version and silently discard the other field update even though both were valid.

A domain-aware merge can preserve independent profile fields, but that does not generalize to every AtlasMart invariant. Two concurrent decrements of scarce inventory or two exclusive order-state transitions are not made correct by “merge the JSON objects.”

3. Replication loops require operation identity and origin metadata

In a mesh of leaders, update A can reach B, then be forwarded from B to C, then return toward A. Without origin/version/event identity, systems can reapply or endlessly circulate logically identical changes. Practical multi-leader systems track revision ancestry, origin identifiers, checkpoints, replication slots/positions, or equivalent metadata.

The exact mechanism varies. The architectural requirement is to know whether an incoming change is new, already represented, ancestral, concurrent, or conflicting.

4. CouchDB 3.5 is a concrete conflict-preserving example

Apache CouchDB 3.5 documentation describes decentralized replication where conflicting document revisions can coexist. Replicas deterministically choose a winning revision for normal reads while retaining conflicting leaves for application-level reconciliation. That is useful evidence that convergence of replica storage and business conflict resolution are different tasks.

This course is not a CouchDB command course. The transferable idea is that a system may safely retain concurrent branches, but AtlasMart still must define how those branches become a business-valid state.

5. Deliberately wrong approach — use arrival-order LWW for every replicated object

During a partition, both region leaders accept valid profile updates. Replication later arrives in EU after the local update, so an arrival-order LWW rule keeps only the US event and reverts the EU phone change. Every replica may eventually show the same winner, yet user intent has been lost.

The repair is to classify the object/fields first. Merge independent profile preferences where semantics permit; require explicit conflict review for ambiguous fields; and keep non-mergeable invariants behind a single owner, conditional write, or stronger coordination mechanism.

6. AtlasMart lab — concurrent region-local writes and a safe merge boundary

from dataclasses import dataclass

@dataclass(frozen=True)
class Update:
    event_id: str
    origin: str
    base_version: int
    fields: dict

base = {"email": "old@example.test", "phone": "+1-000", "version": 7}
us = Update("evt-us-91", "us-east", 7, {"email": "new@example.test"})
eu = Update("evt-eu-44", "eu-west", 7, {"phone": "+44-777"})

print("BASE", base)
print("partition: both leaders accept a region-local write from version 7")
print("US", us)
print("EU", eu)
print("concurrent?", us.base_version == eu.base_version and us.event_id != eu.event_id)

print("\nWRONG: LWW by arrival order")
arrivals = [us, eu]
winning = arrivals[-1]
lww = dict(base)
lww.update(winning.fields)
lww["version"] += 1
print("winner", winning.event_id, "state", lww)
print("lost field update?", lww["email"] == base["email"])

print("\nREPAIR: domain-aware field merge")
merged = dict(base)
seen = set()
for update in arrivals:
    if update.event_id in seen:
        continue  # loop/duplicate prevention
    seen.add(update.event_id)
    merged.update(update.fields)
merged["version"] = base["version"] + 1
print("merged", merged)
print("seen event ids", sorted(seen))

print("\nNON-MERGEABLE EXAMPLE")
print("two leaders independently decrementing the same scarce inventory is not made safe by field merge")

Verification checklist

• Both regional writes share base version 7 and are therefore concurrent in the simulator.
• Naive LWW loses one independent field edit.
• Field-aware merge preserves both independent updates.
• Event IDs prevent duplicate/loop reapplication in the simplified model.
• The lab explicitly refuses to claim that inventory decrements are safely mergeable.

7. Production judgment and next bridge

Use multi-leader only when local write availability/latency is valuable enough to justify explicit conflict semantics and operator complexity. Measure conflicts by object/field, replication delay, reconciliation backlog, loop/duplicate suppression, schema-version mismatches, and regional isolation duration. Exercise conflict resolution with realistic data rather than assuming “rare conflicts.”

Lesson 4 removes permanent write leaders altogether for the data group. A coordinator sends an operation to a replica set and can use fallback replicas when preferred owners are unavailable. Dynamo-style leaderless replication improves some availability paths, but it introduces quorum, stale-replica, sibling, hint, and repair reasoning that Chapter 06 will develop further.

Authoritative references

• Apache CouchDB 3.5 — Replication and conflict model — official decentralized replication, conflicting revisions, deterministic winner, and application merge responsibilities
• Apache CouchDB 3.5 — Eventual Consistency — official conflict detection and convergence framing
• Lamport — Time, Clocks, and the Ordering of Events — primary foundation for causal ordering rather than wall-clock winner assumptions
• Dynamo: Amazon’s Highly Available Key-value Store — primary paper contrasting decentralized replication/versioning with single-owner models