Inject Node/Network Failures and Verify Driver Recovery, Transaction Outcomes, and Operational Alerts

The final exercise is not “kill a node and celebrate that the web page still loads.” AtlasMart must prove correct committed state, bounded client recovery, alerting, and operator understanding under several distinct failures. A driver timeout can occur even when the server committed; a secondary can lag after recovery; quorum can be lost while some processes remain alive. The drill must expose those nuances.

Learning outcomes

Lab baseline and scope

AtlasMart remains the same customer/order/product graph used throughout earlier chapters. This chapter changes deployment topology, not business identity. The optional licensed topology uses a separate database name atlasmart on disposable test infrastructure so cluster administration cannot damage the single-server continuity database.

1. Start with a failure hypothesis

Each injection needs one mechanism to test. “Node failure” is too broad. The secondary-loss experiment tests read-capacity reduction/catch-up without changing write quorum. Writer-loss tests election/routing/retry while majority survives. Two-primary loss tests safety when quorum disappears. Combining all faults at once makes diagnosis impossible.

2. Free deterministic trace first

Use a trace before a licensed chaos test. It forces you to state the expected system and application outcomes in advance. The none_temporarily writer state represents an election interval in the model; its duration is deliberately unspecified because the free trace cannot measure it.

# FREE deterministic trace — save as ha_trace.csv and reason about every row.
time,event,primaries_alive,writer,secondary_state,request,expected_business_outcome
00:00,healthy,3,P1,caught_up,write O-9100,commit once
00:01,stop secondary,3,P1,down,read catalog,served by eligible reader
00:02,restart secondary,3,P1,catching_up,read with bookmark,may wait until state available
00:03,stop writer P1,2,none_temporarily,cancelled client write,ambiguous until reconciled
00:04,elect P2,2,P2,caught_up,retry idempotent O-9101,commit once
00:05,stop P3,1,P2,caught_up,new write O-9102,write unavailable because no majority
00:06,start P1,2,P2,catching_up,new write O-9103,available after quorum restored

3. Stable business keys make ambiguous outcomes reconcilable

Suppose the writer commits an order but the connection breaks before the client receives success. The client sees a timeout, yet “retry the same business operation” and “create a second order” are not equivalent. Reuse the Chapter 6 idempotency discipline: stable orderId, uniqueness constraint, MERGE pattern scoped to identity, and post-error reconciliation.

from neo4j import GraphDatabase
from neo4j.exceptions import Neo4jError
import uuid

URI = "neo4j://server01.example.test:7687"
AUTH = ("atlasmart_app", "<synthetic-test-secret>")
ORDER_ID = "O-HA-" + uuid.uuid4().hex[:10]

QUERY = """
MERGE (o:Order {orderId:$order_id})
ON CREATE SET o.status='PENDING', o.createdAt=datetime()
RETURN o.orderId AS orderId, o.status AS status
"""

with GraphDatabase.driver(URI, auth=AUTH, max_transaction_retry_time=15) as driver:
    try:
        rows, summary, keys = driver.execute_query(
            QUERY,
            order_id=ORDER_ID,
            database_="atlasmart",
        )
        print("result", rows)
    except Neo4jError as exc:
        print("neo4j_code", exc.code)
        print("retryable", exc.is_retryable())
        # Do not blindly create a replacement business operation.
        # Reconcile ORDER_ID against the graph before deciding what the client should do.

4. Optional licensed failure sequence

This sequence assumes a disposable 3-primary + 1-secondary Enterprise test database. Do not use host firewall edits or real production networks. A process/container stop is sufficient for node-loss tests; a network-partition test should use an isolated test network with a documented reset command.

# OPTIONAL LICENSED LAB — execute only in an isolated test cluster.
# Before every step capture SHOW SERVERS, SHOW DATABASE atlasmart YIELD *, routing table,
# application success/error rate, retry count, and backup state.

1. Stop/restart only the secondary allocation host. Verify writes continue; measure catch-up/lag.
2. Stop the current writer primary. Verify another primary becomes writer while 2/3 majority remains.
3. During/around leader loss, issue an idempotent marker/order request with a stable business key.
   Record whether the client saw success, a retryable error, timeout, or ambiguous outcome.
4. Stop a second primary. Verify new writes do NOT commit with only 1/3 primaries alive.
5. Restore one primary. Verify quorum and writer availability return; reconcile all marker/order IDs.
6. Restore all members. Wait for healthy/online allocations and acceptable replication lag.
7. Compare final business invariants and alerts with the pre-injection baseline.

Reset rule: if any unexpected database becomes degraded, stop the exercise and restore topology
before injecting another fault.

5. Capture four evidence planes

6. Alerts must reflect topology semantics

“Host down” is useful but not enough. AtlasMart needs alerts that encode the database risk: no writer, current primaries below requested, replication lag beyond the workload’s freshness objective, store-copy/deallocating stuck, routing/retry error spikes, and capacity saturation during degraded operation. The same host failure can be low severity for a secondary and urgent for the second lost primary in a 3P database.

7. Measure failover without hiding overload

Record request throughput and p50/p95/p99 latency before, during and after failure; transaction error/retry counts; time without an available writer; pool queueing; CPU/disk/network on survivors; replication lag/catch-up; and final business invariants. Do not report one average or only the requests that succeeded. A cluster can “fail over” while the remaining nodes are saturated and p99/error rate violate the SLO.

8. Deliberately unsafe response: “increase retries”

When failover produces errors, a common reaction is to increase retry count/time globally. That can amplify overload, extend tail latency and repeat non-idempotent side effects. Repair the mechanism instead: current official driver, correct routing URI/access mode, bounded retry budget, small idempotent transaction functions, stable operation IDs, reconciliation, pool/queue limits and capacity headroom.

9. Exit criteria and cleanup

• All requested AtlasMart allocations are online and healthy.
• Exactly one writer exists for the database.
• Current primary/secondary counts match requested topology.
• Replication lag is inside the declared objective.
• Driver retry/error rates and pool queueing return to baseline.
• Every injected business operation is reconciled by stable ID with no unintended duplicates.
• Backup/restore status remains valid and independent of cluster copies.
• Every alert fired at the expected severity and cleared after recovery.

Production judgment

Summary and next step

Chapter 17 has turned “clustering” into measurable per-database topology, quorum, routing, failure-domain and application-recovery behavior. Chapter 18 now focuses on the metrics, logs, query/transaction visibility, memory, store and capacity evidence needed to operate that architecture continuously.

Authoritative references

• Current Neo4j versions — Current Neo4j 2026.07.1, 5.26.30 LTS, Cypher Shell and GDS release snapshot.
• Clustering architecture — Current server/database decoupling, primaries, secondaries, majority writes, and causal consistency.
• Deploy a basic cluster — Current discovery/bootstrap configuration and TOPOLOGY examples.
• Cluster server discovery — LIST/DNS/Kubernetes discovery and the current discovery-service model.
• Leadership, routing, and load balancing — Per-database Raft leadership, routing tables, readers/writers/routers, and routing policy.
• Managing databases in a cluster — Primary/secondary allocation and topology changes.
• Managing servers in a cluster — Server enablement, constraints, deallocation, removal, and operational lifecycle.
• Server management command syntax — SHOW/ENABLE/ALTER/DEALLOCATE/REALLOCATE/DROP server commands.
• Show databases — Role, writer, allocation counts, replication lag, and database-state evidence.
• Monitor databases in a cluster — SHOW DATABASES-based per-allocation monitoring and status interpretation.
• Resilient multi-data-center cluster — Failure-domain placement, latency/fault-tolerance tradeoffs, and recommended/anti-pattern layouts.
• Cluster disaster recovery — Recovery when allocations or whole failure domains are lost.
• Built-in procedures — Current routing, cordon, deallocation and related procedure names/deprecations.
• Python driver transactions/routing — Read/write routing and explicit database selection in the official driver.
• Python driver bookmarks — Bookmark propagation and causal-consistency coordination across sessions.