Build a Recoverable CDC Pipeline and Prove Restart, Duplicate, Gap, and Backfill Behavior

AtlasMart is ready to call the synchronization pipeline production-capable only if it can prove recovery, not merely process a happy-path event. The acceptance drill deliberately creates the failures operators will eventually see: duplicate delivery, crash after durable side effect, missing retained history, incompatible source epoch after restore, schema evolution, target outage and a potential feedback loop. Every failure must end in a known state with an observable checkpoint and a tested remediation.

Learning outcomes

Lab contract and assumptions

1. Build the mandatory free recovery harness

Use the exact cdc_sim.py from Lessons 1–2. Optionally create the Community target constraints and replace the simulator’s JSON durable_apply with the Python driver transaction pattern. The simulator remains the source because Community does not expose Neo4j CDC.

CYPHER 25
CREATE CONSTRAINT ch20_product_replica_id IF NOT EXISTS
FOR (p:ProductReplica) REQUIRE p.productId IS UNIQUE;
CREATE CONSTRAINT ch20_customer_replica_id IF NOT EXISTS
FOR (c:CustomerReplica) REQUIRE c.customerId IS UNIQUE;
CREATE CONSTRAINT ch20_order_replica_id IF NOT EXISTS
FOR (o:OrderReplica) REQUIRE o.orderId IS UNIQUE;
CREATE CONSTRAINT ch20_receipt_id IF NOT EXISTS
FOR (r:SyncReceipt) REQUIRE r.eventId IS UNIQUE;

python cdc_sim.py reset
rm -f .atlasmart_ch20_state.json .atlasmart_ch20_checkpoint.json
# Then run the scenario-specific command below.

2. Acceptance matrix: prove each failure mode

3. Run the duplicate scenario

python cdc_sim.py reset
python cdc_sim.py consume --duplicate sim-2004
# Expected around sim-2004:
# APPLIED id=sim-2004 ...
# DUPLICATE id=sim-2004 ...
# CHECKPOINT ordinal=5 id=sim-2005

4. Run crash → replay → recovery

$ python cdc_sim.py reset
$ python cdc_sim.py consume --crash-after sim-2004
...
APPLIED id=sim-2004 txId=505 seq=0 key=P-2001
CRASH after durable target apply, before checkpoint advance

$ python cdc_sim.py consume
DUPLICATE id=sim-2004 txId=505 seq=0 key=P-2001
APPLIED id=sim-2005 txId=506 seq=0 key=P-2002
CHECKPOINT ordinal=5 id=sim-2005

# The replay is expected. Receipt-based idempotency turns the duplicate into a no-op.

In a real Kafka path, the analog is target commit succeeded but the consumer/Kafka Connect offset did not commit. In a direct db.cdc.query consumer, the analog is target commit succeeded but the application checkpoint did not persist. Both require safe replay.

5. Run gap → stop → backfill

$ python cdc_sim.py reset
$ python cdc_sim.py consume --drop sim-2003
APPLIED id=sim-2000 txId=500 seq=0 key=P-2001
APPLIED id=sim-2001 txId=500 seq=1 key=P-2002
APPLIED id=sim-2002 txId=502 seq=0 key=C-2001
GAP expectedOrdinal=3 got=4 -> BACKFILL_REQUIRED

$ python cdc_sim.py backfill
BACKFILL entities=3 products=1 customers=1 orders=1
CHECKPOINT ordinal=5 id=sim-2005

Real Neo4j CDC does not expose the simulator ordinal. The equivalent operational trigger is an invalid/pruned cursor or a restore/pause-resume history reset. On self-managed systems, transaction-log retention determines how far back changes remain queryable. On Aura, fixed log-space rotation means the available time window varies with workload. Never promise a time window from one quiet-period observation.

6. Real CDC operator runbook (optional licensed path)

CYPHER 25
CALL db.cdc.earliest() YIELD id RETURN id AS earliest;
CALL db.cdc.current() YIELD id, txCommitTime RETURN id AS current, txCommitTime;
SHOW PROCEDURES YIELD name
WHERE name STARTS WITH 'db.cdc.' OR name STARTS WITH 'cdc.'
RETURN name ORDER BY name;
// Upgrade gate: application uses db.cdc.*; legacy cdc.* must not be a dependency.

7. Reconciliation is the final truth test

8. Upgrade and migration gate

9. Choose the simplest mechanism that meets the requirement

10. Production judgment

11. Chapter 20 completion checklist

12. Cleanup

CYPHER 25
MATCH (n) WHERE n.labTag = 'ch20' DETACH DELETE n;
DROP CONSTRAINT ch20_product_replica_id IF EXISTS;
DROP CONSTRAINT ch20_customer_replica_id IF EXISTS;
DROP CONSTRAINT ch20_order_replica_id IF EXISTS;
DROP CONSTRAINT ch20_receipt_id IF EXISTS;

rm -f .atlasmart_ch20_state.json .atlasmart_ch20_checkpoint.json cdc_sim.py

Summary and next step

Chapter 20 treats synchronization as a recovery protocol: opaque source cursors, stable business identity, at-least-once replay, atomic target receipts, versioned schemas, measurable lag, explicit gap/backfill transitions and no feedback loops. Chapter 21 can now add full-text retrieval knowing how near-real-time source changes become durable, testable graph/search state.

Authoritative references

• Current Neo4j versions — Current database release 2026.07.1 and current 5.26 LTS patch 5.26.30.
• CDC introduction — Current CDC availability and purpose; CDC is a change feed, not an exact database-copy mechanism.
• CDC on self-managed Neo4j — Enterprise-only enablement, OFF/DIFF/FULL txLogEnrichment modes, security, retention, disk and unrecorded-change boundaries.
• CDC on Aura — Current AuraDB Business Critical / Virtual Dedicated Cloud enablement, pause/resume and snapshot-reset behavior.
• CDC procedures — db.cdc.earliest/current/query semantics, exclusive cursors, txId/seq, metadata and change identifiers.
• CDC event schema — Current node/relationship event envelope including operation, keys, before/after state and metadata.
• CDC selectors — Server-side entity/operation/label/type/key/property and metadata filtering.
• CDC examples — Official cursor-management examples and empty-result/current-cursor retention guidance.
• CDC backup/restore behavior — Why restores invalidate old change identifiers and require consumer state/backfill/reinitialization planning.
• CDC troubleshooting — Disabled, scan-failure and invalid-identifier error mechanisms including retention and wrong-database cursors.
• CDC known issues — ElementId and change-identifier instability across restore/import/copy/pause-resume operations.
• CDC changelog — Current db.cdc.* namespace and deprecation history of cdc.* procedures.
• Neo4j Connector for Kafka — Current source/sink connector architecture and CDC/query source strategies.
• Kafka source CDC quickstart — Current CDC source configuration, txLogEnrichment prerequisite, event fields, offsets, topics and failure modes.
• Kafka connector installation — Current connector release 5.5.3 and Kafka Connect plugin deployment model.
• Kafka source query quickstart — Community-compatible polling source alternative and its delete/soft-delete limitation.
• Kafka sink CDC quickstart — Neo4j-to-Neo4j CDC sink pattern, source header requirement, key constraints and loop warning.
• Python driver 6.3 API — Official driver 6.3 and Bolt compatibility used for optional application examples.