Chapter 12 · Source System Profiling, Data Contracts, Lineage, and Ingestion Readiness
Source-to-Target Mapping Specifications, Transform Rules, Data Types, Units, Time Zones, and Codes
Write source-to-target mappings that make field semantics executable: types, units, time zones, code systems, transforms, target grain, reject behavior, lineage, and ownership are explicit rather than hidden in transformation code.
Learning outcomes
AtlasMart now knows the ERP extract is internally consistent, but a clean source row is still not a warehouse fact. The team must specify how each source field becomes a target key or measure, which unit and time zone apply, how codes are interpreted, and what happens when conversion is impossible.
Define a source-to-target mapping as a semantic specification, not merely a pair of column names.
Document transform rules, target types, units, time zones, code sets, target grain, reject behavior, and owners.
Preserve Chapter 11 event-time customer-key resolution rather than mapping a late fact to the current dimension row.
Distinguish a deterministic conversion from an ambiguous coercion.
Connect mappings to source contracts, warehouse objects, metrics, and consumers through lineage edges.
Chapter 12 preserves the accepted AtlasMart warehouse contracts from Chapters 01–11. The current published sales state after Chapter 11 contains eight current paid order-line facts, five paid orders, ten units, 690 paid GMV, 425 cost-at-sale, and 265 gross profit. Customer history still uses half-open business-effective intervals, source-qualified durable identity, governed unknown members, and auditable corrections. This chapter moves upstream: it defines what evidence a source must provide before new ingestion or transformation code is trusted. The synthetic source extracts therefore reflect the accepted Chapter 11 corrections rather than silently reverting to the original 625-GMV fixture.
The mandatory exercises are free, local, synthetic, and deterministic. They were validated with Python 3.13.5 using only standard-library data structures/JSON/hash functions. The examples prove the stated fixture, contract, and gate behavior; they do not prove production-source correctness, network extraction behavior, organizational ownership, legal compliance, or the behavior of a managed ingestion service. Treat every source statement as a contract that must be verified against the actual producer.
A mapping can be executable and still be semantically wrong. Correctness depends on producer meaning and consumer requirements, not only whether code can parse the source value.
1. A mapping explains meaning through the transformation boundary
A source-to-target mapping states where a target attribute comes from and the exact rule that produces it. It should make review possible without reading an opaque ETL job. For a fact measure, that includes source grain, arithmetic, unit, filters, and error policy. For a dimension key, it includes identity namespace and history/as-of semantics.
| Source → target | Transform/semantics | Unit/time zone | Failure behavior |
|---|---|---|---|
| erp.orders.order_id → fact_sales.order_id | copy as degenerate transaction identifier | n/a / n/a | reject parent/child batch if relationship cannot be resolved |
| erp.order_lines.quantity → fact_sales.quantity | parse integer; require > 0 | item / n/a | quarantine row; never coerce malformed quantity to zero |
| erp.order_lines.unit_price → fact_sales.extended_amount | quantity * unit_price after unit contract validation | USD / n/a | block if currency/unit semantics unresolved |
| erp.orders.order_ts → fact_sales.order_date_key | parse RFC3339 UTC; derive governed calendar date | n/a / UTC | reject ambiguous/local timestamp |
| erp.orders.customer_id → fact_sales.customer_sk | source-qualified durable identity then event-time SCD as-of lookup | n/a / business effective time | governed unknown/inferred member policy; do not bind blindly to current row |
| inventory.on_hand_units → fact_inventory_snapshot.on_hand_units | copy integer state observation | item / snapshot_ts UTC | block incomplete/duplicate snapshot |
2. Type is not unit; timestamp syntax is not time-zone semantics
unit_price=190 is a number, but without the unit
contract it could mean 190 USD, 190 cents, or 190 in a source
currency. Likewise 2026-09-18T11:30:00 is
timestamp-shaped text but has no offset; treating it as UTC is
an assumption. AtlasMart requires RFC 3339/UTC at this boundary
so the order date and SCD as-of lookup are deterministic.
Codes need the same discipline. status='paid' must
have a producer-owned meaning. If a producer introduces
captured, the consumer cannot simply guess that it
is equivalent to paid because that changes KPI semantics.
3. Mapping a customer key preserves durable identity and business-effective history
Chapter 11 already established that a fact does not join to
“whatever customer row is current today.” The mapping for
customer_id therefore has two steps:
source-qualified durable identity resolution, then a Type 2
as-of lookup using the order event time. This requirement
belongs in the mapping because it materially changes historical
attribution.
ERP customer_id + source namespace → durable_customer_id →
customer_sk where effective_from ≤ order_ts <
effective_to. If no eligible version exists, apply the governed
unknown/inferred-member policy; never substitute the current
version silently.
4. Deliberately wrong approach: “cast until it fits”
A transformation receives quantity='two',
unit_price='19.99USD', or a timestamp without
offset. It strips non-digits, inserts zero on failure, and
assumes UTC. The load completes and dashboards render, but the
warehouse has fabricated business values.
The repair is an explicit conversion contract with a reject path. If the producer guarantees integer quantity and numeric USD unit price, non-conforming values are evidence of a contract violation. Preserve the original value, record the reason, and block/quarantine according to severity.
5. Local lab — validate the mapping specification
mappings=[ {"source":"erp.order_lines.quantity","target":"fact_sales.quantity","rule":"parse integer; require > 0","unit":"item","timezone":"n/a","on_error":"quarantine"}, {"source":"erp.order_lines.unit_price","target":"fact_sales.extended_amount","rule":"quantity * unit_price after unit validation","unit":"USD","timezone":"n/a","on_error":"block unresolved unit"}, {"source":"erp.orders.order_ts","target":"fact_sales.order_date_key","rule":"parse RFC3339 UTC; derive date","unit":"n/a","timezone":"UTC","on_error":"reject ambiguous timestamp"}, {"source":"erp.orders.customer_id","target":"fact_sales.customer_sk","rule":"durable identity + event-time SCD lookup","unit":"n/a","timezone":"business effective time","on_error":"governed unknown/inferred member"},]required_sources={"erp.order_lines.quantity","erp.order_lines.unit_price","erp.orders.order_ts","erp.orders.customer_id"}seen={m["source"] for m in mappings}ambiguous=[m["source"] for m in mappings if not m.get("rule") or not m.get("on_error")]print("missing",sorted(required_sources-seen))print("ambiguous",ambiguous)print("decision","READY" if required_sources<=seen and not ambiguous else "BLOCKED")missing []ambiguous []decision READYThis proves the local specification has the required entries and failure policies. It does not prove the producer actually emits USD or UTC; that evidence comes from the source contract and owner.
6. Lineage makes impact review possible before code exists
Lineage records meaningful dependency edges.
Even before ingestion is implemented, AtlasMart can document
intended lineage from source datasets through contracts/mappings
to warehouse objects and metrics. This is useful because a
change to unit_price should identify
fact_sales, paid_gmv, and the
executive sales product as affected surfaces.
| From | Relationship | To |
|---|---|---|
| src.erp.orders | declares | contract.erp.orders.v1 |
| src.erp.order_lines | declares | contract.erp.order_lines.v1 |
| contract.erp.order_lines.v1 | governs | map.sales.order_line.v1 |
| map.sales.order_line.v1 | loads | warehouse.fact_sales |
| src.crm.customers | identity/history input | warehouse.dim_customer |
| src.inventory.snapshot | loads | warehouse.fact_inventory_snapshot |
| warehouse.fact_sales | atomic input | metric.paid_gmv |
| warehouse.fact_inventory_snapshot | snapshot input | metric.on_hand_units |
| metric.paid_gmv | serves | product.executive_sales |
These edges are vendor-neutral conceptual identifiers. A later metadata platform or OpenLineage-compatible tool could operationalize them, but Chapter 12 does not require that tooling.
7. Verification, migration, and cleanup
- Every mapped measure has a unit and grain context.
- Every timestamp-sensitive transform declares a time-zone policy.
- Customer key resolution retains the Chapter 11 event-time rule.
- Every mapping has an explicit failure/reject behavior.
-
Cleanup: delete
mapping_gate.py; no persistent state was created.
If a mapping changes, version it alongside the source contract, identify affected lineage edges, reconcile old/new results, and keep a rollback path to the prior certified mapping until acceptance is complete.
Knowledge check
Check your understanding
- Why is a source/target column pair insufficient?
- Why can a numeric unit_price still be ambiguous?
- How should a late fact resolve customer_sk?
- When is coercion unsafe?
- What does lineage add to a mapping change?
Review the answers
1. It omits transformation semantics, grain, units, time zones, code meanings, error behavior, and ownership.
2. Numeric type does not specify currency or scale such as dollars versus cents.
3. Through source-qualified durable identity and the customer version effective at the fact event time.
4. When it invents meaning not guaranteed by the producer contract, such as assuming a missing time-zone offset.
5. It identifies downstream warehouse objects, metrics, and products that may need review/reconciliation.
Summary and next step
The mapping now makes the source-to-warehouse meaning explicit. Lesson 4 versions that agreement as a data contract and deliberately changes units/time zones without changing primitive field types, demonstrating why a schema-only compatibility check can approve a semantically breaking release.
Authoritative references
- W3C — PROV Overview — Official W3C overview of provenance concepts used to reason about entities, activities, agents, and lineage relationships.
- OpenLineage — Specification — Open specification for dataset/job/run lineage events; referenced as a later operationalization option, not a prerequisite for this local lab.
- JSON Schema — Specification — Official JSON Schema specification; useful when a source contract is represented as JSON, while the business semantics still require explicit agreement beyond structure.
- RFC 3339 — Date and Time on the Internet — Authoritative timestamp format reference used for the UTC timestamp contract in the synthetic fixture.
- Python documentation — csv — Standard-library CSV support suitable for free/local deterministic source fixtures.
- Python documentation — json — Standard-library JSON support used for contract and readiness artifacts.
- Python documentation — hashlib — Standard-library hashing API used only for reproducibility/evidence fingerprints, not as a semantic validation substitute.