Observe AtlasMart data across freshness, volume, distribution, schema, lineage, and quality changes, using baselines and contracts to distinguish incidents from harmless variance.
Data Observability: Freshness, Volume, Distribution, Schema, Lineage, and Quality Changes
Build an identity and access model for AtlasMart that separates humans from services, eliminates shared credentials, enforces environment boundaries, and proves least privilege with allow/deny evidence.
Learning outcomes
Observe data along freshness, volume, distribution, schema, lineage, and quality dimensions.
Use baselines as context rather than treating every variance as an incident.
Recognize distinct signal fingerprints for lateness, schema change, and semantic corruption.
Use lineage to convert a field/metric defect into an explicit consumer blast radius.
Design alerts that carry enough evidence for action.
Chapter 25 begins from the accepted AtlasMart state established through Chapters 01–24: 10 current paid lines, 8 orders, 12 units, 820 USD gross revenue, 495 USD cost, and 325 USD gross profit. The fact grain remains one current accepted paid order line, source progress remains committed through sequence 208, Chapter 20 metric contracts remain authoritative, Chapter 21 certified marts remain dependent on conformed assets, Chapter 22 security/privacy controls remain in force, Chapter 23 lineage/ownership metadata supplies blast-radius context, and Chapter 24 tests remain the correctness gates. Observability adds continuous evidence and incident handling; it must not silently reinterpret business rules merely to make a dashboard look healthy.
Runtime: Python 3.13.5 + SQLite 3.46.1. Storage: local in-memory structures plus SQLite semantics where transactions matter. Clock: UTC with explicit timestamps. Security: synthetic identifiers only. Cost: free/local. Healthy run: 10 records in/out, 4,820 bytes, 7.0 minutes, zero rejects/retries, 860 ms fixture CPU, 34 MB fixture peak memory, and certification 12 minutes after source readiness. Important limitation: these CPU/memory numbers are fixture measurements used to teach signal relationships; they are not performance recommendations for any warehouse engine or cloud service.
1. The realistic problem: three incidents, three different signatures
AtlasMart experiences three isolated failures against the same
canonical dataset: the source manifest arrives late; a source
column is unexpectedly renamed; and a dashboard query silently
excludes the sales channel. A single “row count
changed” alert cannot distinguish them. Data observability means
collecting and relating evidence across multiple properties of
the data product so the operator can identify the mechanism,
affected consumers, and safe response.
2. Six observability dimensions—and their limits
| Dimension | Question | AtlasMart evidence |
|---|---|---|
| Freshness | Is the product as current as its contract requires? | source-ready and certified timestamps |
| Volume | Did expected rows/bytes arrive? | manifest count, records in/out, rejects |
| Distribution | Did value shape shift? | channel/category shares, min/max/quantiles where meaningful |
| Schema | Did structure/type/nullability change? | expected vs observed field contract |
| Lineage | What downstream assets depend on the changed object? | field→job→model→metric→mart→report→consumer graph |
| Quality | Do business/relationship/reconciliation rules still hold? | Chapter 24 blockers and control totals |
No single dimension is a proxy for accuracy. A distribution can look normal while values are systematically wrong; a schema can be unchanged while metric semantics drift.
3. Baselines describe normal variation; they do not define business truth
recent duration minutes: 6.8, 7.1, 7.0, 7.3, 6.9, 7.2, 7.0median duration: 7.0max / p95-proxy*: 7.3recent row counts: 9, 10, 10, 9, 10, 10, 10median rows: 10observed range: 9..10*Tiny fixture: max is shown only as an explicit teaching proxy, not a statistical p95 estimator.A production baseline needs enough history, seasonality/context, and a response policy. Never call every departure from yesterday an incident. Promotions, month-end loads, new products, or valid backfills can change distributions.
4. Incident fingerprint A: source lateness
source-ready objective: <= 08:10 UTCactual source-ready: 09:05 UTCsource-ready breach: 55 minutescertified objective: <= 08:30 UTCactual certified: 09:18 UTCcertified breach: 48 minutespipeline status: SUCCESSdata controls after load: 10 / 8 / 12 / 820 / 495 / 325The data is correct but late. The signature is freshness/SLO failure with normal schema, volume, quality, and semantic reconciliation once the source finally arrives.
5. Incident fingerprint B: breaking schema drift
missing expected field: line_amount_usdnew observed field: gross_line_amount_usdrows published: 0contract action: BLOCK BEFORE TRANSFORM/PUBLISHlineage blast radius: 15 catalog assetsThe field rename may be intentional, but until the producer/consumer semantics are resolved it is breaking for the current contract. The safe response is not silent coercion or guessing that the new name means the same thing.
6. Incident fingerprint C: semantic error with green infrastructure
pipeline status: SUCCESSwarehouse gross_revenue_usd: 820.00local dashboard result: 665.00difference: -155.00 (-18.90%)row count / schema: unchangeddownstream semantic blast: 9 assets
The pipeline and schema are healthy. The failure is that a
dashboard-local filter excludes sales but reuses
the governed metric label. Reconciliation and cross-tool
semantic tests expose what infrastructure monitoring cannot.
7. Distribution monitoring needs declared semantics
| Channel | Canonical revenue | Share |
|---|---|---|
| web | 305 USD | 37.2% |
| mobile | 360 USD | 43.9% |
| sales | 155 USD | 18.9% |
| total | 820 USD | 100% |
A sudden zero share for sales could signal missing
data or a deliberate business change. Distribution evidence
should point the operator to a question; contracts/owners
determine whether the change is valid.
8. Lineage turns detection into blast-radius analysis
The schema incident reaches 15 assets: source field, normalization job, integrated fact column, governed measure, revenue/profit metrics, three certified marts, three reports, and three consumer groups. The semantic bug affects 9 assets downstream of the revenue metric. This difference matters: schema drift may block the entire dependent path before publish, while a semantic defect may require invalidating/rebuilding only products that consumed the wrong metric version or cached result.
9. Controlled failure: alert on every variance
Wrong: page an operator whenever row count differs from the previous run or any channel share moves. That produces noise and trains people to ignore alerts. Repair: distinguish hard invariants/contracts from anomaly signals, include expected seasonal/event context, and map alert severity to consumer risk. A blocker schema mismatch is categorically different from a small valid distribution shift.
10. What a useful alert should carry
| Field | Why |
|---|---|
| dataset/metric + partition | Defines the affected object and time window |
| observed vs expected | Shows the actual contract/SLO deviation |
| run/source IDs | Supports replay/audit |
| lineage blast radius | Names models/reports/consumers at risk |
| owner/runbook | Routes the incident to an accountable response |
| last known good / certification status | Prevents consumers from assuming fresh data |
11. Production judgment and bridge
Detectability: monitor consumer-relevant data properties, not every available counter. Signal noise: separate hard contracts from learned/heuristic baselines. Freshness/history: timestamps must identify the partition and as-of state. Security: distribution/lineage metadata can reveal sensitive categories or system topology; govern access. Testing: observability complements deterministic tests rather than replacing them. Next: Lesson 3 formalizes the most important signals as end-to-end SLOs from source availability to certified data products.
12. Verification checklist
- Confirm the late incident has normal controls but a 48-minute certification breach.
- Confirm schema drift is blocked before publish and reports 15 impacted assets.
- Confirm the semantic bug reports 665 vs 820 USD with unchanged schema/row count.
- Confirm baseline values are labeled as observations, not universal thresholds.
- Confirm alert evidence includes owner and blast radius.
Knowledge check
Acceptance questions
- Why can schema observability miss a semantic bug?
- What is the difference between a hard contract and an anomaly baseline?
- Why does lineage belong in an alert?
- Why is “every variance” a poor alert policy?
Review the answers
1. A query/filter/business definition can be wrong while every column and type remains unchanged.
2. A contract is an explicit required condition; a baseline describes observed behavior and usually needs contextual interpretation.
3. It identifies which models, metrics, reports, and consumers may be unsafe.
4. Normal business variation creates noise, causing alert fatigue and masking genuinely actionable failures.
Authoritative references
- OpenTelemetry — Metrics specificationAuthoritative metric concepts and separation of instrumentation from collection/export behavior.
- OpenTelemetry — Metrics data modelUseful for reasoning about time-series measurements, aggregation, temporality, and preserving metric semantics.
- Google SRE Book — Service Level ObjectivesDefines SLIs/SLOs and emphasizes selecting indicators from user-relevant service behavior rather than everything that is easy to measure.
- Google SRE Book — Monitoring Distributed SystemsBackground on actionable monitoring signals and avoiding monitoring that produces noise without operator decisions.
- SQLite — TransactionsSupports the local demonstration of atomic repair/backfill state and safe commit boundaries.
- Python — statisticsUsed for small deterministic baseline examples; production anomaly detection needs workload-specific statistical validation.
- Python — hashlibUsed to fingerprint deterministic incident evidence and repaired outputs.
13. Lab cleanup/reset
Each incident scenario is isolated against the same synthetic canonical state. Rerun from a clean process to reproduce the three fingerprints without carrying repaired state from one scenario into another.