Chapter 12 · Source System Profiling, Data Contracts, Lineage, and Ingestion Readiness
Data Contracts and Schema Change Detection: Compatible vs Breaking Changes
Version producer/consumer data contracts and prove why schema compatibility is not semantic compatibility by detecting structural, unit, time-zone, and code-set changes before ingestion.
Learning outcomes
The ERP producer proposes contract version 2. It keeps the same
primitive field types and adds one nullable field, so a
schema-only checker reports “compatible.” However,
unit_price changes from dollars to cents and
updated_at loses its UTC contract. If AtlasMart
accepts the release based only on field names/types, paid GMV
can inflate from 690 to 69,000 and incremental ordering can
become ambiguous.
Define a versioned data contract and separate structural compatibility from semantic compatibility.
Classify nullable additive fields, removals/renames, type changes, key/grain changes, unit changes, and time-zone changes.
Run both a schema-only diff and a semantic contract diff against the same proposed producer change.
Block a release whose structure is parseable but whose metric/time semantics are incompatible.
Design producer/consumer migration and rollback steps without silently accepting drift.
Chapter 12 preserves the accepted AtlasMart warehouse contracts from Chapters 01–11. The current published sales state after Chapter 11 contains eight current paid order-line facts, five paid orders, ten units, 690 paid GMV, 425 cost-at-sale, and 265 gross profit. Customer history still uses half-open business-effective intervals, source-qualified durable identity, governed unknown members, and auditable corrections. This chapter moves upstream: it defines what evidence a source must provide before new ingestion or transformation code is trusted. The synthetic source extracts therefore reflect the accepted Chapter 11 corrections rather than silently reverting to the original 625-GMV fixture.
The mandatory exercises are free, local, synthetic, and deterministic. They were validated with Python 3.13.5 using only standard-library data structures/JSON/hash functions. The examples prove the stated fixture, contract, and gate behavior; they do not prove production-source correctness, network extraction behavior, organizational ownership, legal compliance, or the behavior of a managed ingestion service. Treat every source statement as a contract that must be verified against the actual producer.
“Compatible” is always relative to a consumer contract. Even adding a nullable field can be operationally expensive or conflict with naming policies; conversely, some breaking structural changes can be safely migrated with coordinated versions. The lab models a strict AtlasMart consumer, not a universal compatibility standard.
1. Data contract = structure + semantics + ownership + change policy
A data contract is the versioned producer/consumer agreement that makes a dataset safe to depend on. A structural schema is part of it, but not all of it. AtlasMart records grain, keys, field types/nullability, units, timestamp/time-zone rules, code meanings, change evidence, owners, and a compatibility policy.
{ "contract_id": "atlasmart.erp.order_lines", "version": "1.0.0", "owner": "Order Platform", "consumer_owner": "Analytics Engineering", "grain": "one row per ERP order_id + line_no in an extract revision", "keys": [ "order_id", "line_no" ], "fields": { "order_id": { "type": "string", "nullable": false, "semantic": "ERP order identifier" }, "line_no": { "type": "integer", "nullable": false, "semantic": "line ordinal within order" }, "product_id": { "type": "string", "nullable": false, "semantic": "product business key" }, "quantity": { "type": "integer", "nullable": false, "unit": "item", "minimum": 1 }, "unit_price": { "type": "number", "nullable": false, "unit": "USD", "semantic": "price per one item before tax/shipping" }, "updated_at": { "type": "timestamp", "nullable": false, "timezone": "UTC", "format": "RFC3339" } }, "allowed_parent_statuses": [ "paid", "pending" ], "change_key": "updated_at", "compatibility_policy": "nullable additive field may be compatible; rename/type/unit/time-zone/key/grain changes require consumer review"}2. Compatible vs breaking depends on consumer semantics
In this course fixture, adding nullable
discount_code is structurally compatible because an
existing consumer can ignore it. Removing or renaming a required
field, changing its type, changing the business key/grain,
changing dollars to cents, or changing UTC timestamps to
unspecified local time is breaking for the current mappings. A
semantic change can be breaking even when every JSON/CSV token
still parses successfully.
3. Controlled failure — schema-only diff approves the dangerous release
The proposed v2 keeps field names and primitive types, adds
nullable discount_code, but changes two semantic
properties:
| Severity | Change | Evidence |
|---|---|---|
| compatible | field_added | discount_code |
| breaking | unit_changed | unit_price: USD -> USD_cents |
| breaking | timezone_changed | updated_at: UTC -> local_no_offset |
The deliberately weak schema-only checker sees only:
| Schema-only severity | Change | Field |
|---|---|---|
| compatible | added | discount_code |
If a producer emits the corrected paid values in cents while the warehouse still interprets them as dollars, the same business sales produce 69,000 “USD” instead of the accepted 690 USD. The pipeline can be syntactically successful and semantically catastrophic.
4. Local lab — compare structural and semantic diffs
from copy import deepcopyv1={"grain":"one row per order+line","keys":["order_id","line_no"],"fields":{ "unit_price":{"type":"number","nullable":False,"unit":"USD"}, "updated_at":{"type":"timestamp","nullable":False,"timezone":"UTC"}}}v2=deepcopy(v1)v2["fields"]["discount_code"]={"type":"string","nullable":True}v2["fields"]["unit_price"]["unit"]="USD_cents"v2["fields"]["updated_at"]["timezone"]="local_no_offset"def schema_only(a,b): out=[] for f in b["fields"].keys()-a["fields"].keys(): out.append(("compatible" if b["fields"][f].get("nullable") else "breaking","added",f)) for f in a["fields"].keys() & b["fields"].keys(): if a["fields"][f]["type"]!=b["fields"][f]["type"]: out.append(("breaking","type",f)) return sorted(out)def semantic(a,b): out=schema_only(a,b) for f in a["fields"].keys() & b["fields"].keys(): for prop in ("unit","timezone"): if a["fields"][f].get(prop)!=b["fields"][f].get(prop): out.append(("breaking",prop,f,a["fields"][f].get(prop),b["fields"][f].get(prop))) return outprint("schema_only",schema_only(v1,v2))print("semantic",semantic(v1,v2))print("decision","BLOCKED" if any(x[0]=="breaking" for x in semantic(v1,v2)) else "READY")schema_only [('compatible', 'added', 'discount_code')]semantic [('compatible', 'added', 'discount_code'), ('breaking', 'unit', 'unit_price', 'USD', 'USD_cents'), ('breaking', 'timezone', 'updated_at', 'UTC', 'local_no_offset')]decision BLOCKED5. Migration and rollback are part of compatibility
A safe unit migration could introduce a new explicitly named field or contract version, dual-publish long enough to reconcile, update mappings/tests/metrics, compare 690-USD controls across versions, and cut over only after acceptance. A timestamp migration likewise needs a declared conversion rule and effective boundary. If validation fails, the consumer remains pinned to v1 rather than guessing how to parse v2.
Contract versioning should be tied to source releases and lineage impact. Deprecation needs an owner and date; silently mutating a contract file in place destroys reproducibility.
6. Idempotency, retries, and schema evolution boundaries
A contract does not by itself make ingestion idempotent.
Duplicate delivery, retry, ordering, deletes, and watermark
advancement are separate runtime mechanics handled later. But
the contract must expose the fields/semantics those mechanisms
depend on. If updated_at loses a deterministic
time-zone/ordering meaning, a previously safe incremental policy
may no longer be safe.
7. Verification and cleanup
- Schema-only diff sees only the nullable added field.
- Semantic diff detects both unit and time-zone changes as breaking.
- The readiness decision is BLOCKED.
- The 690 → 69,000 example is an explicit unit calculation, not an invented benchmark.
- Cleanup: delete the local contract-diff script/files; no external service was used.
Knowledge check
Check your understanding
- Can two contracts have the same primitive schema but different business meaning?
- Why is nullable discount_code compatible in this fixture?
- What makes USD→USD_cents breaking?
- Why does updated_at timezone affect ingestion correctness?
- What should the consumer do while v2 is blocked?
Review the answers
1. Yes. Units, time zones, code definitions, grain, and key semantics can change without primitive type changes.
2. Existing consumers can ignore it under the stated consumer policy; that is not a universal rule.
3. The same numeric value is interpreted at a different scale, corrupting monetary measures unless mappings migrate.
4. Incremental ordering/watermarks and event interpretation depend on a deterministic time basis.
5. Stay pinned to the certified v1 contract/mapping and coordinate an explicit migration rather than guess.
Summary and next step
A source contract must protect semantics, not just parser compatibility. Lesson 5 assembles the chapter evidence into a readiness report and demonstrates that unresolved unit/time-zone semantics are sufficient to stop pipeline development even when connectivity and schema parsing succeed.
Authoritative references
- W3C — PROV Overview — Official W3C overview of provenance concepts used to reason about entities, activities, agents, and lineage relationships.
- OpenLineage — Specification — Open specification for dataset/job/run lineage events; referenced as a later operationalization option, not a prerequisite for this local lab.
- JSON Schema — Specification — Official JSON Schema specification; useful when a source contract is represented as JSON, while the business semantics still require explicit agreement beyond structure.
- RFC 3339 — Date and Time on the Internet — Authoritative timestamp format reference used for the UTC timestamp contract in the synthetic fixture.
- Python documentation — csv — Standard-library CSV support suitable for free/local deterministic source fixtures.
- Python documentation — json — Standard-library JSON support used for contract and readiness artifacts.
- Python documentation — hashlib — Standard-library hashing API used only for reproducibility/evidence fingerprints, not as a semantic validation substitute.