Chapter 08 · Importing Data: LOAD CSV, Data Importer, Bulk Import, Transformation, and Validation
Reconcile Counts, Constraints, Orphans, Duplicates, and Business Invariants After a Large Import
Close every import with a reproducible source-to-graph reconciliation gate.
Learning outcomes
An AtlasMart import is not finished when a command exits successfully. Cutover requires a reconciliation suite that proves source lineage, identity, relationship completeness and business invariants.
Build source-to-target control totals for nodes and relationships.
Detect duplicates, orphans, null/type defects and invalid cardinalities.
Validate constraints separately from control-total reconciliation.
Distinguish technical completion from business acceptance.
Design recovery for partially committed online batches and bulk-import cutover.
The mandatory lab continues Neo4j Community
2026.07.1, database neo4j, explicit
CYPHER 25 for version-sensitive examples,
authentication enabled, no mandatory APOC/GDS plugin, and
stable AtlasMart business identifiers from Chapters 01–07.
Neo4j 5.26.30 remains the LTS comparison line.
Local file:/// sources are a self-managed
server-filesystem feature and are not available on Aura.
This generation environment does not run Neo4j or Docker.
Commands were checked against current official documentation
but were not executed here. Expected outputs are fixture
invariants, not fabricated captures. Every write is scoped to
labTag='ch08', dedicated CSV files, or a
disposable database. Never use broad cleanup, store overwrite,
or offline import commands against unrelated or production
data.
Deterministic Chapter 08 source fixture
The source intentionally contains a duplicate business key, an empty optional field, a malformed numeric value, and orphan foreign keys. Those defects are evidence for preflight and reconciliation, not details to ignore.
customerId,name,email,tierC-8001,Ada Lovelace,ada@example.test,goldC-8002,Grace Hopper,grace@example.test,silverC-8002,Grace Hopper Duplicate,grace.dup@example.test,silverC-8003,Linus Torvalds,,bronzeproductId,name,categoryId,priceP-8001,Trail Camera,CAT-8001,129.90P-8002,Weather Case,CAT-8001,39.50P-8003,Sensor Hub,CAT-8002,not-a-numberP-8004,Field Battery,CAT-MISSING,49.00orderId,customerId,orderedAt,statusO-8001,C-8001,2026-09-01T10:00:00Z,PAIDO-8002,C-8002,2026-09-02T11:30:00Z,SHIPPEDO-8003,C-MISSING,2026-09-03T12:00:00Z,PAIDorderId,productId,quantity,unitPriceO-8001,P-8001,1,129.90O-8001,P-8002,2,39.50O-8002,P-8004,1,49.00O-9999,P-8001,1,129.90The corrected files remove/quarantine duplicate and orphan rows, repair the malformed price, and apply a documented empty-email policy. Keep the rejection ledger so accepted + rejected rows reconcile to the source.
1. Control totals must balance
| Control | Acceptance rule |
|---|---|
| Customers | accepted customer rows = distinct ch08 customer IDs |
| Orders | accepted order rows = distinct ch08 order IDs |
| Order items | accepted item rows = expected CONTAINS edges |
| Rejected rows | source = accepted + quarantined/rejected |
| Required references | zero unresolved endpoints after policy |
Do not assume target node count must equal raw source rows; a documented transformation can legitimately reject or collapse rows. The ledger explains why.
2. Identity and constraint audit
CYPHER 25SHOW CONSTRAINTS;MATCH (c:Customer) WITH c.customerId AS id,count(*) AS n WHERE n>1 RETURN 'Customer' AS entity,id,nUNION ALLMATCH (p:Product) WITH p.productId AS id,count(*) AS n WHERE n>1 RETURN 'Product' AS entity,id,nUNION ALLMATCH (o:Order) WITH o.orderId AS id,count(*) AS n WHERE n>1 RETURN 'Order' AS entity,id,n;3. Relationship reconciliation
CYPHER 25MATCH (o:Order) WHERE o.labTag='ch08'WITH o,count { (:Customer)-[:PLACED]->(o) } AS customerLinksWHERE customerLinks <> 1RETURN o.orderId,customerLinks;MATCH (o:Order)-[r:CONTAINS]->(p:Product)WHERE o.labTag='ch08' OR p.labTag='ch08'RETURN count(r) AS containsCount, count(DISTINCT [o.orderId,p.productId]) AS distinctPairs;The pair uniqueness check is only valid if AtlasMart allows at most one line per product per order. If line items have independent identity, reconcile against that model instead. Acceptance queries must encode the actual business contract.
4. Business invariants catch structurally valid mistakes
CYPHER 25MATCH (p:Product) WHERE p.labTag='ch08' AND (p.price IS NULL OR p.price < 0)RETURN p.productId,p.price;MATCH (o:Order) WHERE o.labTag='ch08' AND NOT EXISTS { MATCH (o)-[:CONTAINS]->(:Product) }RETURN o.orderId;MATCH (p:Product)-[:IN_CATEGORY]->(c:Category) WHERE p.labTag='ch08'RETURN p.productId,count(c) AS categoryLinks ORDER BY p.productId;5. Deliberately wrong: trust the exit code
A successful command plus one Browser screenshot cannot reveal missed tail batches, wrong relationship direction, duplicate collapse or orphan policy errors. Acceptance must be machine-readable and repeatable.
6. Recovery differs by import mechanism
For CALL { … } IN TRANSACTIONS, earlier batches may
already be committed when a later one fails; fix the defect and
rerun an idempotent pipeline, then reconcile. For bulk import,
prefer validating a disposable destination before
promotion/cutover instead of repairing an unaccepted store in
place.
Chapter 08 acceptance lab
CYPHER 25MATCH (c:Customer) WHERE c.labTag='ch08' WITH count(c) AS customersMATCH (p:Product) WHERE p.labTag='ch08' WITH customers,count(p) AS productsMATCH (o:Order) WHERE o.labTag='ch08' WITH customers,products,count(o) AS ordersOPTIONAL MATCH (o2:Order)-[r:CONTAINS]->(:Product) WHERE o2.labTag='ch08'RETURN customers,products,orders,count(r) AS itemRelationships;CYPHER 25MATCH (n) WHERE n.labTag='ch08' DETACH DELETE n;Check your understanding
- Why may raw source rows differ from target nodes?
- Does SHOW CONSTRAINTS replace duplicate checks?
- Why validate relationship cardinality?
- How does a partial LOAD CSV import recover?
- When is the migration complete?
Review the answers
1. Rejected rows or intentional MERGE collapse can change counts; the control-total ledger must explain the transformation.
2. No. Constraint state is one contract; observed target/source reconciliation is separate evidence.
3. Correct node counts can still coexist with missing, reversed or multiplied edges.
4. Correct the cause, rerun the idempotent import and execute the full acceptance suite.
5. When technical and business acceptance checks pass and source/rejection lineage is retained for cutover/audit.
Summary and next step
Chapter 08 establishes a trustworthy import lifecycle: preflight, bounded or bulk ingestion, and reconciliation. Chapter 09 can now add constraints and indexes to a graph whose contents are trusted.
Authoritative references
- Current Neo4j versions — Release/LTS snapshot used for the chapter baseline.
- Cypher LOAD CSV — Current LOAD CSV parsing, typing and local/remote source semantics.
- CALL subqueries in transactions — Bounded inner transactions, default batch size and partial-commit behavior.
- Neo4j Admin import — Current full/incremental import, dry run, headers, memory and report behavior.
- Default file locations — Self-managed import-directory and store/log boundaries.
- Constraints — Integrity-contract semantics used in acceptance.
- Indexes — Index lifecycle/state concepts after import.
- Admin import — Importer fault/report behavior and current full/incremental boundaries.