Failure Domains, Zones/Regions, Network Partitions, Quorum Loss, and Split-Brain Avoidance

AtlasMart expands into multiple availability zones. A diagram with three server icons in different boxes looks resilient, but appearance is not enough. You need to ask which failures are correlated, where the primary majority remains after each failure, what cross-zone round trips cost every write, and what a network partition does when both sides are still running.

Learning outcomes

Lab baseline and scope

AtlasMart remains the same customer/order/product graph used throughout earlier chapters. This chapter changes deployment topology, not business identity. The optional licensed topology uses a separate database name atlasmart on disposable test infrastructure so cluster administration cannot damage the single-server continuity database.

1. A failure domain is a shared cause, not a drawing box

A failure domain is anything likely to fail together: a host, hypervisor, rack switch, power feed, availability zone, data center or regional network. Putting three primaries on three virtual machines does not provide three independent failure domains if all VMs share one physical host or one power/network dependency. The topology review must map causes to allocations.

2. Majority consensus is the split-brain safety rule

Consider a network partition. Both sides may have live Neo4j processes, but only a side that can communicate with a majority of the database’s primaries can continue fault-tolerant commits. In a 3-primary 2|1 partition, the side with two can retain quorum; the single-primary side cannot independently become a safe writer. That sacrifices availability on the minority side to preserve one committed history.

from math import floor

# Each tuple: (layout name, primaries total, primaries reachable by side A, side B)
partitions = [
    ("3P split 2|1", 3, 2, 1),
    ("5P split 3|2", 5, 3, 2),
    ("3P split 1|1 with one down", 3, 1, 1),
]

for name, total, a, b in partitions:
    majority = floor(total/2)+1
    print(name, {
        "majority": majority,
        "side_A_can_commit": a >= majority,
        "side_B_can_commit": b >= majority,
    })

3. Three primaries: tolerate one arbitrary primary loss

With three primaries, majority is two. This is compact and often appropriate when the required failure tolerance is one primary. But maintenance consumes the same headroom as a failure: if one primary is deliberately offline, a second failure removes quorum. Therefore a maintenance runbook must state the temporary reduction in fault tolerance.

4. Five primaries: more fault headroom, more coordination

Five primaries need a majority of three and can tolerate two arbitrary primary failures. That can be useful for combinations of maintenance and failure, or carefully designed multi-data-center placement. It also means more copies, more catch-up work and potentially more expensive quorum paths. “Five is safer” is incomplete unless the placement and network latency actually satisfy the intended failure model.

5. Multi-data-center placement: availability versus latency

Neo4j’s current guidance emphasizes that primary placement across distant data centers increases write latency because the writer must obtain quorum confirmation. Geo-distributing primaries can tolerate site loss when a majority remains, but every transactional write pays for the topology’s network path. A design must therefore define both fault-tolerance objective and latency SLO.

6. Secondaries help reads, not partition quorum

If a remote site contains only secondaries, local clients may continue to serve stale-tolerant reads when the primary site is unreachable, subject to routing and the local database state. They cannot form a new write majority. Promoting or recreating databases after disaster is an explicit recovery operation with possible data-loss considerations; it is not the normal meaning of a secondary.

7. Optional licensed partition acceptance matrix

Use safe, isolated failure injection. Prefer stopping an Enterprise test process/container or disconnecting an isolated test network segment. Do not change the host firewall or production routes for a course exercise. Before each injection capture topology, current writer, baseline request success, backup status and reset procedure.

8. Failure-domain verification checklist

• Every primary is mapped to an independent host/zone/site cause.
• For each planned failure, surviving primaries are counted and compared with majority.
• Write latency is measured across the real quorum path.
• Secondaries are not counted as write voters.
• Maintenance states are included in failure combinations.
• Backups remain independent from the cluster and restore-tested.

Production judgment

Summary and next step

Quorum defines safety under failure, but day-two operations can accidentally consume that safety margin. Lesson 4 turns to seeding, membership changes, cordoning, deallocation, reallocation and maintenance headroom.

Authoritative references

• Current Neo4j versions — Current Neo4j 2026.07.1, 5.26.30 LTS, Cypher Shell and GDS release snapshot.
• Clustering architecture — Current server/database decoupling, primaries, secondaries, majority writes, and causal consistency.
• Deploy a basic cluster — Current discovery/bootstrap configuration and TOPOLOGY examples.
• Cluster server discovery — LIST/DNS/Kubernetes discovery and the current discovery-service model.
• Leadership, routing, and load balancing — Per-database Raft leadership, routing tables, readers/writers/routers, and routing policy.
• Managing databases in a cluster — Primary/secondary allocation and topology changes.
• Managing servers in a cluster — Server enablement, constraints, deallocation, removal, and operational lifecycle.
• Server management command syntax — SHOW/ENABLE/ALTER/DEALLOCATE/REALLOCATE/DROP server commands.
• Show databases — Role, writer, allocation counts, replication lag, and database-state evidence.
• Monitor databases in a cluster — SHOW DATABASES-based per-allocation monitoring and status interpretation.
• Resilient multi-data-center cluster — Failure-domain placement, latency/fault-tolerance tradeoffs, and recommended/anti-pattern layouts.
• Cluster disaster recovery — Recovery when allocations or whole failure domains are lost.
• Built-in procedures — Current routing, cordon, deallocation and related procedure names/deprecations.
• Python driver transactions/routing — Read/write routing and explicit database selection in the official driver.
• Python driver bookmarks — Bookmark propagation and causal-consistency coordination across sessions.