Run a Disaster-Recovery Drill and Measure RPO, RTO, Data Integrity, Application Recovery, and Operator Steps

Learning outcomes

Reproducible AtlasMart setup

The drill adds one tiny course-owned marker so recovery correctness has a deterministic invariant even if your earlier AtlasMart graph has grown. The marker is not a substitute for business reconciliation; it is a canary.

CYPHER 25
CREATE CONSTRAINT recovery_marker_id IF NOT EXISTS
FOR (m:RecoveryMarker) REQUIRE m.markerId IS UNIQUE;
MERGE (c:Customer {customerId:'C-1001'})
ON CREATE SET c.name = 'Mina Rahimi'
MERGE (m:RecoveryMarker {markerId:'DR-CH16-001'})
SET m.createdAt = datetime('2026-09-09T17:00:00Z'),
    m.expectedState = 'before-dump',
    m.exercise = 'chapter-16'
MERGE (c)-[:HAS_RECOVERY_MARKER]->(m);
MATCH (m:RecoveryMarker {markerId:'DR-CH16-001'})
RETURN m.markerId, m.expectedState;

1. Disaster recovery is a timed system test

The drill scenario is intentionally narrow: assume the AtlasMart primary data volume is unavailable at incident time. You may use only the previously created neo4j.dump, pinned container image, documented configuration and synthetic credentials. The original volume remains untouched so the exercise is reversible.

Observed RPO = T0 − Tbackup. Database recovery duration = Tdb − T0. End-to-end RTO = Tapp − T0. Use your real timestamps; no course-provided number is a benchmark.

2. Preflight: freeze evidence, do not mutate the source

set -euo pipefail
mkdir -p ./neo4j-dr/run-001
 date -u +'%Y-%m-%dT%H:%M:%SZ' | tee ./neo4j-dr/run-001/T0.txt
sha256sum -c ./neo4j-dr/evidence/neo4j.dump.sha256
cp ./neo4j-dr/evidence/manifest.txt ./neo4j-dr/run-001/
docker image inspect neo4j:2026.07.1 --format '{{.Id}}' | tee ./neo4j-dr/run-001/image-id.txt

3. Restore into the isolated target and time it

docker rm -f atlasmart-neo4j-restore 2>/dev/null || true
docker volume rm atlasmart-neo4j-restore-data 2>/dev/null || true
docker volume create atlasmart-neo4j-restore-data

date -u +'%Y-%m-%dT%H:%M:%SZ' | tee ./neo4j-dr/run-001/restore-start.txt
docker run --rm \
  -v atlasmart-neo4j-restore-data:/data \
  -v "$PWD/neo4j-dr/backups:/backups" \
  neo4j/neo4j-admin:2026.07.1 \
  neo4j-admin database load neo4j --from-path=/backups --overwrite-destination=true

docker run -d --name atlasmart-neo4j-restore \
  -p 127.0.0.1:7475:7474 -p 127.0.0.1:7688:7687 \
  -v atlasmart-neo4j-restore-data:/data \
  -e NEO4J_AUTH=neo4j/atlasmart-course-2026 \
  neo4j:2026.07.1

4. Gate 1: database readiness and graph integrity

Do not declare success when the container status becomes “running.” The database must authenticate and return the expected recovery point.

until docker exec atlasmart-neo4j-restore cypher-shell \
  -u neo4j -p atlasmart-course-2026 \
  "RETURN 1 AS ready" >/dev/null 2>&1; do sleep 2; done

docker exec atlasmart-neo4j-restore cypher-shell \
  -u neo4j -p atlasmart-course-2026 \
  "CYPHER 25 MATCH (m:RecoveryMarker {markerId:'DR-CH16-001'}) RETURN m.expectedState;"
date -u +'%Y-%m-%dT%H:%M:%SZ' | tee ./neo4j-dr/run-001/Tdb.txt

CYPHER 25
MATCH (n) RETURN count(n) AS nodes;
MATCH ()-[r]->() RETURN count(r) AS relationships;
MATCH (c:Customer {customerId:'C-1001'})-[:HAS_RECOVERY_MARKER]->(m:RecoveryMarker {markerId:'DR-CH16-001'})
RETURN c.customerId, m.expectedState;
SHOW CONSTRAINTS;
SHOW INDEXES;

5. Gate 2: consistency and application recovery

For a direct store consistency check, stop the isolated restore DBMS, run database check, then restart it. Application recovery then verifies the actual repository/service layer from Chapter 13/14 against port 7688: authentication, a parameterized read, a safe idempotent write, result consumption, timeout/retry behavior and any external dependency needed by the chosen SLO.

docker stop atlasmart-neo4j-restore
docker run --rm -v atlasmart-neo4j-restore-data:/data \
  neo4j/neo4j-admin:2026.07.1 \
  neo4j-admin database check neo4j
docker start atlasmart-neo4j-restore

6. Failure injection: remove a dependency, not data

Safe drills should expose runbook weaknesses without corrupting the source. One reversible test is to start the restored service with a wrong synthetic password, confirm authentication failure, then correct the secret reference. Another is to rename the dump file before preflight and prove the runbook stops on missing/hash evidence.

7. Measure and interpret—do not celebrate one stopwatch

8. Runbook result and improvement backlog

A useful drill ends with decisions. Examples: increase backup frequency to meet RPO; move artifacts to a separate account/site; shorten restore time by adjusting full/differential cadence in Enterprise; automate hash/metadata checks; pre-stage images; reduce manual secret/DNS steps; or add application-level reconciliation for external payments/orders.

9. Cleanup/reset

docker rm -f atlasmart-neo4j-restore 2>/dev/null || true
docker volume rm atlasmart-neo4j-restore-data 2>/dev/null || true
# Keep backup/evidence if you want to continue the recovery exercises.
docker start atlasmart-neo4j 2>/dev/null || true

Production judgment

Summary and next step

Chapter 16 has turned backup from a file-creation task into measured recoverability and upgrade rollback evidence. Chapter 17 moves from recovery of a failed deployment to keeping service available during member/network failures through Neo4j clustering, consensus, routing and failure-domain design.

Authoritative references

• Current Neo4j versions — Current server and 5.26 LTS release snapshot.
• Backup and restore — Edition-aware entry point for dump/load, online backup, restore and planning.
• Backup and restore planning — RPO/RTO, backup mode, storage location, cadence and retention planning.
• Back up an offline database — neo4j-admin database dump semantics and Community offline boundary.
• Restore a database dump — neo4j-admin database load semantics, overwrite rules and edition differences.
• Back up an online database — Enterprise full/differential backup artifacts and chain semantics.
• Restore a database backup — Enterprise recovery of backup chains and restore-until predicates.
• Check database consistency — neo4j-admin database check for stores, dumps and recovered full backups.
• Transaction logging — Transaction-log retention, checkpointing and pruning behavior.
• Store formats — Current aligned/block formats, limits and legacy-format deprecation.
• Migrate a database — neo4j-admin database migrate and store-format migration boundaries.
• System requirements — Supported Java/runtime and platform requirements for current Neo4j.
• APOC installation — APOC/server release compatibility and restart/deployment coupling.
• GDS compatibility — Graph Data Science and Neo4j version compatibility matrix.
• Python driver installation — Current 6.x driver/server compatibility baseline.