Chapter 12 · Source System Profiling, Data Contracts, Lineage, and Ingestion Readiness
Build a Source Readiness Report and Reject Ambiguous Semantics Before Pipeline Development
Combine inventory, profiling, contracts, mappings, lineage, security, and change checks into an evidence-based readiness gate that can stop pipeline development when source semantics remain ambiguous.
Learning outcomes
AtlasMart has a source inventory, profile evidence, source-to-target mappings, lineage, and a versioned ERP contract. The final question is operational: should the team build/activate ingestion now? A readiness report needs explicit gates and evidence, not a subjective “looks good” score.
Assemble source trust, ownership, profiling, semantic completeness, mappings, lineage, security, and change-management evidence into a readiness decision.
Use blocking conditions rather than arbitrary maturity scores when an unresolved ambiguity can corrupt history or metrics.
Produce a baseline READY decision and an injected-change BLOCKED decision from the same deterministic fixture.
Trace each blocking issue to the owner, contract/mapping, warehouse object, metric, and consumer at risk.
Define acceptance, reset, rollback, and the bridge from source readiness to Chapter 13 data-quality engineering.
Chapter 12 preserves the accepted AtlasMart warehouse contracts from Chapters 01–11. The current published sales state after Chapter 11 contains eight current paid order-line facts, five paid orders, ten units, 690 paid GMV, 425 cost-at-sale, and 265 gross profit. Customer history still uses half-open business-effective intervals, source-qualified durable identity, governed unknown members, and auditable corrections. This chapter moves upstream: it defines what evidence a source must provide before new ingestion or transformation code is trusted. The synthetic source extracts therefore reflect the accepted Chapter 11 corrections rather than silently reverting to the original 625-GMV fixture.
The mandatory exercises are free, local, synthetic, and deterministic. They were validated with Python 3.13.5 using only standard-library data structures/JSON/hash functions. The examples prove the stated fixture, contract, and gate behavior; they do not prove production-source correctness, network extraction behavior, organizational ownership, legal compliance, or the behavior of a managed ingestion service. Treat every source statement as a contract that must be verified against the actual producer.
Readiness is scoped to a source version and intended consumer. A READY report is permission to proceed to implementation/testing under the stated assumptions—not a guarantee that production data will remain correct forever.
1. A readiness report is an evidence gate, not a score
A numeric “87% ready” can hide a fatal ambiguity. AtlasMart instead uses blocking predicates: ownership assigned; extraction/change semantics documented; freshness stated; blocking profile invariants pass; grain/keys/units/time zones resolved; mappings complete; lineage declared; and security classification present. If a measure's currency unit is unresolved, the decision is BLOCKED regardless of how many other boxes are green.
| Gate | Baseline result |
|---|---|
| owners_assigned | PASS |
| extraction_and_change_mechanisms_documented | PASS |
| freshness_contracts_documented | PASS |
| source_profile_passes_blocking_invariants | PASS |
| grain_keys_units_timezones_resolved | PASS |
| mappings_complete | PASS |
| lineage_declared | PASS |
| security_classification_present | PASS |
Baseline decision: READY.
2. Evidence bundle for the certified v1 source
The readiness artifact should link—not merely summarize—the source inventory, current profile, contract version/hash, mapping version, lineage edges, expected controls, and security classification. That bundle makes a later incident answerable: which assumptions were certified when the load was approved?
| Source | Owner | Extract/change mechanism | Freshness contract | Primary risks |
|---|---|---|---|---|
| ERP orders/order lines | Order Platform | 15-minute incremental; updated_at high-water mark + overlap/reconciliation | 95% certified ≤30 min after commit | missed/duplicate/partial parent-child changes |
| CRM customers | Customer Platform | daily snapshot + updated_at evidence | certified by 03:00 UTC | late snapshot, code drift, deletion state loss |
| Inventory snapshot | Supply Chain | 30-minute full product snapshot; snapshot_ts | certified ≤45 min from snapshot | partial snapshot, duplicate product, unit change |
| Dataset | Rows | Duplicate business keys | Key observations |
|---|---|---|---|
| ERP orders | 6 | 0 | 5 paid / 1 pending; channels web/mobile/sales |
| ERP order lines | 9 | 0 | quantity 1–2; unit_price 25–200; all-line value 1090 |
| CRM customers | 4 | 0 | 4 unique IDs; one erased synthetic identity marker |
| Inventory snapshot | 4 | 0 | on_hand min 12 / max 60 / sum 137 |
| Control | Accepted value | Meaning |
|---|---|---|
| Current paid order-line rows | 8 | Chapter 11 current-revision facts |
| Paid orders | 5 | distinct paid order_id |
| Paid units | 10 | sum quantity on paid lines |
| Paid GMV | 690 | sum quantity × unit_price on paid lines, USD |
| All order-line value | 1090 | includes pending O1004; not paid GMV |
| Inventory on-hand units | 137 | single 2026-09-20 snapshot; not additive over time |
3. Failure injection — v2 is structurally parseable but not ready
The proposed ERP v2 adds a nullable field, changes
unit_price from USD to USD cents, and changes
updated_at from UTC to local time with no offset.
Connectivity remains fine. A schema-only parser can still ingest
rows. But the semantic contract diff is breaking, so the source
profile for the intended v1 mapping is marked failed until the
producer/consumer migration is resolved.
| Severity | Change | Evidence |
|---|---|---|
| compatible | field_added | discount_code |
| breaking | unit_changed | unit_price: USD -> USD_cents |
| breaking | timezone_changed | updated_at: UTC -> local_no_offset |
Injected-change decision: BLOCKED. Failed gate(s): source_profile_passes_blocking_invariants.
4. Impact analysis from source ambiguity to business consumer
The unit change reaches fact_sales.extended_amount,
then paid_gmv, then the executive sales product.
The time-zone change can alter incremental ordering and the
derived order date; it can also affect event-time SCD key
resolution near a boundary. This is why lineage is part of
readiness rather than an afterthought.
| From | Relationship | To |
|---|---|---|
| src.erp.orders | declares | contract.erp.orders.v1 |
| src.erp.order_lines | declares | contract.erp.order_lines.v1 |
| contract.erp.order_lines.v1 | governs | map.sales.order_line.v1 |
| map.sales.order_line.v1 | loads | warehouse.fact_sales |
| src.crm.customers | identity/history input | warehouse.dim_customer |
| src.inventory.snapshot | loads | warehouse.fact_inventory_snapshot |
| warehouse.fact_sales | atomic input | metric.paid_gmv |
| warehouse.fact_inventory_snapshot | snapshot input | metric.on_hand_units |
| metric.paid_gmv | serves | product.executive_sales |
5. Local acceptance suite
The following condensed script models the final decision. It does not contact any real source; it demonstrates the gate logic and the distinction between baseline certification and a changed source version.
def decide(owner_ok, extract_ok, profile_ok, semantics_ok, mapping_ok, lineage_ok, security_ok): checks={"owner":owner_ok,"extract/change":extract_ok,"profile":profile_ok,"semantics":semantics_ok, "mapping":mapping_ok,"lineage":lineage_ok,"security":security_ok} failed=[k for k,v in checks.items() if not v] return ("READY" if not failed else "BLOCKED", failed)print("v1", decide(True,True,True,True,True,True,True))# v2 parses, but USD->cents and UTC->local-no-offset are unresolved for this consumer.print("v2", decide(True,True,False,False,True,True,True))paid_lines=8; paid_orders=5; paid_units=10; paid_gmv=690; inventory=137assert (paid_lines,paid_orders,paid_units,paid_gmv,inventory)==(8,5,10,690,137)print("controls",paid_lines,paid_orders,paid_units,paid_gmv,inventory)v1 ('READY', [])v2 ('BLOCKED', ['profile', 'semantics'])controls 8 5 10 690 137The acceptance suite proves the local decision logic and Chapter 11 continuity controls. It does not claim that a real source is available, that an SLA was measured, or that a producer accepted the contract.
6. Readiness report template
{ "source_version": "atlasmart.erp.order_lines@1.0.0", "decision": "READY", "controls": { "current_paid_lines": 8, "paid_orders": 5, "paid_units": 10, "paid_gmv": 690.0, "all_line_value": 1090.0, "inventory_units": 137 }, "contract_sha256": "e7d454c43fa5082b50b59a88e499603e4937a59ef823378a18f0413e2ae8ad85", "mapping_sha256": "69c73870e34a7d2b92af2e7ac65765a3b2ffa49e3a73ec906bfc8c7d4feb6427", "profile_sha256": "895acec573344bac8f5f63dd50b412f4977999cbbdb2d508fbc65a64cf9e5454", "lineage_sha256": "5fb6cb2f966af22d08c53e038644e13350ab1f256eeb569390359c23a04da208", "gates": { "owners_assigned": true, "extraction_and_change_mechanisms_documented": true, "freshness_contracts_documented": true, "source_profile_passes_blocking_invariants": true, "grain_keys_units_timezones_resolved": true, "mappings_complete": true, "lineage_declared": true, "security_classification_present": true }, "known_limits": [ "synthetic local fixture", "no production SLA observation", "no network/connector behavior exercised" ], "rollback": "remain pinned to contract/mapping v1 until a changed source version passes semantic diff and reconciliation"}Hashes identify exact serialized evidence in this lab; they do not prove that the evidence is correct. In production, retain versioned artifacts in an access-controlled repository with review/audit metadata.
7. Production judgment: when ambiguity should block development
Block when an unresolved question can change identity, grain, history, measure value, unit, time interpretation, delete behavior, or source completeness. Examples include “Does price mean dollars or cents?”, “Is updated_at event time or load time?”, “Can customer IDs be reused?”, and “Is this inventory export full or partial?” Starting transformation development before these are answered often embeds guesses that become expensive historical corrections later.
Non-blocking uncertainties can be recorded as limitations when they do not affect the current consumer contract, but they still need owners and follow-up dates. The decision should always be scoped and reviewable.
8. Security, observability, retries, and failure recovery
Readiness includes minimum source privileges and classification, but Chapter 22 later expands warehouse security. It records freshness expectations but does not substitute for Chapter 25 observability. It records extraction/change semantics, but Chapter 15 later proves retries, duplicates, late data, deletes, and watermark behavior. This separation prevents a source-readiness document from making guarantees it has not tested.
9. Cleanup/reset and chapter acceptance checklist
- Delete any local scripts/JSON artifacts; all fixtures are synthetic and disposable.
- Re-run baseline gates: v1 must be READY.
- Re-run injected v2: it must be BLOCKED due unresolved semantics/profile contract.
- Verify source controls remain 8 paid lines, 5 paid orders, 10 units, 690 USD GMV, 137 inventory units.
- Verify lineage includes source → contract/mapping → warehouse → metric → data product edges.
- Do not begin production ingestion until producer ownership and real source observations confirm the documented assumptions.
Knowledge check
Check your understanding
- Why avoid a readiness percentage?
- What is the baseline decision for the synthetic v1 fixture?
- Why is the proposed v2 BLOCKED even though it parses?
- What do hashes in the report prove?
- What chapter follows source readiness, and why?
Review the answers
1. A high score can hide one fatal ambiguity; blocking predicates make critical semantic gaps explicit.
2. READY under the stated local contract and profiling evidence.
3. Its unit and time-zone semantics break the current consumer contract and can corrupt metrics/incremental interpretation.
4. They fingerprint exact evidence artifacts; they do not prove semantic truth or production behavior.
5. Chapter 13: Data Quality Engineering, where validated source contracts become concrete standardization, validation, quarantine, matching, and reconciliation controls.
Summary and next step
Chapter 12 establishes a hard rule: connectivity is not readiness. A source becomes ingestible only when ownership, change semantics, profiles, mappings, units/time zones/codes, contracts, lineage, and security boundaries are explicit enough to test. Chapter 13 now converts those source contracts into data-quality engineering: validation, standardization, matching, reconciliation, and quarantine.
Authoritative references
- W3C — PROV Overview — Official W3C overview of provenance concepts used to reason about entities, activities, agents, and lineage relationships.
- OpenLineage — Specification — Open specification for dataset/job/run lineage events; referenced as a later operationalization option, not a prerequisite for this local lab.
- JSON Schema — Specification — Official JSON Schema specification; useful when a source contract is represented as JSON, while the business semantics still require explicit agreement beyond structure.
- RFC 3339 — Date and Time on the Internet — Authoritative timestamp format reference used for the UTC timestamp contract in the synthetic fixture.
- Python documentation — csv — Standard-library CSV support suitable for free/local deterministic source fixtures.
- Python documentation — json — Standard-library JSON support used for contract and readiness artifacts.
- Python documentation — hashlib — Standard-library hashing API used only for reproducibility/evidence fingerprints, not as a semantic validation substitute.