Trace AtlasMart dependencies from a source field through raw data, transformation jobs, warehouse columns, governed metrics, marts, reports, and named consumers so a source change has an observable blast radius.
Column/Table/Job Lineage from Source to Dashboard and Impact Analysis for Changes
Build an identity and access model for AtlasMart that separates humans from services, eliminates shared credentials, enforces environment boundaries, and proves least privilege with allow/deny evidence.
Learning outcomes
Distinguish table-, column-, job-, report-, and consumer-level lineage.
Traverse a dependency graph in the downstream direction for impact analysis.
Explain why lineage that stops at a warehouse table misses business blast radius.
Recognize the precision tradeoff between coarse job/table lineage and field-level lineage.
Connect lineage to change approval and rollback.
Chapter 23 begins from the governed and secured AtlasMart state produced by earlier chapters: 10 current paid lines, 8 orders, 12 units, 820 USD gross revenue, 495 USD cost, and 325 USD gross profit. The fact grain remains one current paid order line; the Chapter 20 metric contracts remain authoritative; Chapter 21 certified marts remain dependent on those contracts; and Chapter 22 security/privacy controls remain in force. Metadata and lineage describe, govern, and make change safer. They do not silently create a second business definition.
Runtime: Python 3.13.5 + SQLite 3.46.1. Mode: local, in-memory SQLite plus deterministic Python, no cloud account and no paid feature. Time: UTC. Currency: USD. Security: synthetic identities and non-secret metadata only; metadata can itself reveal sensitive topology, so production catalog access still requires policy. History: catalog entries and deprecation notices are versioned/effective-dated rather than overwritten without evidence. Non-guarantee: the lab proves graph/catalog logic for this fixture; it does not prove any vendor's automatic lineage completeness.
1. The realistic problem: ERP announces a rename—who must care?
The ERP team plans to rename
order_lines.line_amount_usd to
gross_line_amount_usd. The type and value semantics
remain the same, but the old name will disappear after a
compatibility window. A warehouse engineer can find the raw
mapping quickly. The harder question is: which integration code,
semantic measures, metrics, marts, reports, and named consumers
are exposed?
Lineage is the dependency/provenance relationship that explains what an asset derives from or influences. Impact analysis traverses those relationships from a proposed change to downstream assets. The correct unit is not always a table: field-level lineage is required when one column changes and other columns in the same job/table are unaffected.
2. Model the dependency graph explicitly
| Upstream | Edge | Downstream |
|---|---|---|
src.erp.order_lines.line_amount_usd |
IDENTITY | raw.erp_order_lines.line_amount_usd |
raw...line_amount_usd |
INPUT | job.integrate_sales |
raw...line_amount_usd |
TRANSFORMATION | fact_sales.line_amount_usd |
fact_sales.line_amount_usd |
AGGREGATION | measure.paid_line_amount_usd |
measure.paid_line_amount_usd |
METRIC_RULE | gross_revenue_usd.v1 |
gross_revenue_usd.v1 |
DEPENDENCY | finance/marketing marts + executive report |
reports |
CONSUMED_BY | named teams |
Job lineage and column lineage answer different questions. The
job must change because its mapping references the old field.
But the unrelated cost_amount_usd column does not
become impacted simply because the same job also writes it.
Coarse lineage may overstate blast radius; lineage that stops
too early understates it.
3. Executed recursive impact query
WITH RECURSIVE impacted(asset_id, depth, path) AS ( SELECT :changed_asset, 0, :changed_asset UNION ALL SELECT e.downstream_id, i.depth + 1, i.path || ' -> ' || e.downstream_id FROM impacted AS i JOIN lineage_edge AS e ON e.upstream_id = i.asset_id WHERE instr(i.path, e.downstream_id) = 0)SELECT DISTINCT asset_id, asset_type, MIN(depth) AS depthFROM impacted JOIN catalog_asset USING(asset_id)GROUP BY asset_id, asset_typeORDER BY depth, asset_type, asset_id;The path check prevents a cycle from recursing forever in this teaching fixture. Production graphs need stronger identity/version rules, cycle handling, event-time lineage, and potentially graph-optimized storage; none of those change the dependency semantics.
4. Observable blast radius: 15 assets
IMPACT_COUNT 15source_column 1raw_column 1job 1model_column 1measure 1metric 2mart 2report 3consumer 3| Depth | Type | Representative impacted asset |
|---|---|---|
| 0 | source_column | src.erp.order_lines.line_amount_usd |
| 1 | raw_column | raw.erp_order_lines.line_amount_usd |
| 2 | job/model |
job.integrate_sales,
fact_sales.line_amount_usd
|
| 3–4 | measure/metric |
paid_line_amount_usd, revenue/profit metrics
|
| 4–6 | mart/report | finance + marketing marts; three reports |
| 5–7 | consumer | finance, executive, and marketing teams |
gross_profit_usd.v1 is impacted even though
cost_amount_usd itself is not: gross profit
combines revenue and cost, so changing the revenue input path
changes the metric dependency. That is why impact analysis
should traverse semantic logic, not only physical columns.
5. Controlled failure: lineage stops at the warehouse table
SHALLOW_LINEAGE_SEES 4 OF 15MISSES: measure.paid_line_amount_usd metric.gross_profit_usd.v1 metric.gross_revenue_usd.v1 mart.finance.daily_sales mart.marketing.daily_sales report.exec_sales_summary report.finance_margin_daily report.marketing_revenue consumer.exec_team consumer.finance_team consumer.marketing_opsA technically correct warehouse-only lineage graph would tell an engineer which model code to edit but fail to tell product owners which certified interfaces and human decision processes are at risk. The repair is to model report/semantic/consumer edges as first-class dependencies and keep a consumer inventory.
6. Named consumer evidence
exec_team -> report.exec_sales_summary -> Weekly executive revenue reviewfinance_team -> mart.finance.daily_sales -> Ad-hoc finance reconciliationfinance_team -> report.finance_margin_daily -> Daily margin closemarketing_ops -> report.marketing_revenue -> Campaign revenue reportingThis proves which registered consumers are downstream. It does not prove there are no unregistered CSV exports, copied SQL snippets, or shadow dashboards. Consumer inventory completeness is a governance and observability problem.
7. Direct, indirect, and semantic influence
Column lineage should record whether a value is copied,
transformed, aggregated, or merely used to filter/join/sort a
result. A column can affect a metric without appearing in the
output value—for example, order_status may be used
only to filter to paid orders. If such indirect dependencies are
omitted, a status-code change can alter revenue while the
lineage graph claims revenue is unaffected.
8. Production judgment
Completeness: impact analysis is only as complete as emitted/curated edges and consumer inventory. Freshness: evaluate lineage version/effective time when code or schema changes. Ownership: route affected nodes to owners/stewards. Replay: lineage ingestion should be idempotent per run/version. Security: hide sensitive metadata from unauthorized users while retaining enough information for operators. Cost: field-level lineage costs more to collect/store than table lineage; choose precision from change-risk evidence. Rollback: preserve old paths during compatibility windows so a migration can be reversed.
9. Bridge to Lesson 3
Lineage tells us what is affected. Governance must tell us who decides, what SLA/certification is promised, and how a change is deprecated. Lesson 3 turns those labels into gates.
Knowledge check
Acceptance questions
- Why can table-level lineage overstate or understate impact?
- Why is a consumer node useful if a report node already exists?
- What is an indirect lineage dependency?
- Does 15 impacted assets mean exactly 15 production risks?
Review the answers
1. A table/job may contain unrelated columns, while downstream semantic/report dependencies can exist beyond the table.
2. It identifies accountable users/use cases that require communication and migration.
3. An input used in filter/join/group/sort/conditional logic that influences results without becoming the output value directly.
4. No; it is the registered fixture blast radius, bounded by metadata completeness.
Authoritative references
- OpenLineage — Lineage Dataset FacetCurrent specification for expressing dataset/job/field dependencies; useful for portable lineage concepts without making OpenLineage a prerequisite.
- OpenLineage — Column Level Lineage Dataset FacetShows fine-grained field dependencies and distinguishes identity, transformation, aggregation, join, filter, sort, window, and conditional influence.
- W3C — Data Catalog Vocabulary (DCAT) Version 3A standard vocabulary for describing cataloged datasets/data services and their metadata; referenced as an interoperability model, not as a required implementation.
- W3C — PROV-OStable provenance vocabulary for entities, activities, and agents; useful for reasoning about lineage/provenance boundaries.
- SQLite — WITH / recursive common-table expressionsThe local lab uses a recursive CTE to traverse downstream lineage deterministically.
- Python — hashlibUsed to fingerprint the deterministic metadata state after catalog repair and change registration.
10. Lab cleanup/reset
Close the in-memory database and rerun the fixture. Repeated impact queries are read-only; repeated metadata ingestion should preserve stable asset IDs rather than duplicate graph nodes.