Chapter 13 · Data Quality Engineering: Validation, Standardization, Matching, Reconciliation, and Quarantine
Define Data Quality Dimensions: Completeness, Validity, Uniqueness, Consistency, Timeliness, and Accuracy
Turn data quality into explicit, measurable evidence across completeness, validity, uniqueness, consistency, timeliness, and accuracy, with scoped thresholds and owners rather than a vague “clean data” label.
Learning outcomes
Define completeness, validity, uniqueness, consistency, timeliness, and accuracy as separate measurable claims.
Attach each quality rule to grain, scope, owner, severity, evidence, and consumer risk instead of a global “quality score”.
Distinguish observed conformance from accuracy, which usually needs trusted external evidence.
Demonstrate why row count alone cannot establish source-to-warehouse correctness.
Build and interpret the AtlasMart quality-test batch without changing the accepted 690-USD production continuity.
1. Quality begins with a decision, not a cleanup function
AtlasMart receives a new Q-prefixed source batch. One customer has a blank name, one order timestamp lacks an offset, one unit code is CASE without a contract conversion, and one order-line event is redelivered. A pipeline can parse most of these rows, but parsing is not the decision surface. Finance needs revenue that is not doubled; operations needs timestamps comparable in UTC; customer analytics needs identities that are neither silently merged nor silently split.
Data quality engineering is the system of explicit rules, measurements, evidence, exception handling, ownership, and reconciliation that determines whether data is fit for a declared use. It does not mean “make every field look tidy.” A field can be syntactically valid but semantically wrong, or complete but inaccurate.
Continuity contract
Chapter 13 preserves the accepted AtlasMart production continuity from Chapters 01–12: eight current paid order-line facts, five paid orders, ten units, 690 USD paid GMV, 425 USD cost-at-sale, 265 USD gross profit, and the governed inventory snapshot of 137 units. Chapter 12 established source contracts and readiness gates; this chapter does not silently change those values. Instead, it introduces a separate synthetic quality-test batch (IDs prefixed Q) so malformed records can be injected, quarantined, repaired, deduplicated, and reconciled without pretending they were already part of the accepted warehouse.
2. Six quality dimensions are different claims
| Dimension | Question | AtlasMart evidence | What it does not prove |
|---|---|---|---|
| Completeness | Is required evidence present at the declared grain? | 3 of 4 raw customer rows have a nonblank required name. | A present value can still be wrong. |
| Validity | Does a value satisfy its declared syntax/domain/unit rule? | 4 of 5 order timestamps carry an explicit offset; 4 of 5 order units are in the approved item aliases. | Validity does not prove the event really occurred. |
| Uniqueness | Does the business key occur no more than allowed? | QO1/1 appears twice and must resolve as one business event. | Unique rows can still represent duplicate real-world entities under different keys. |
| Consistency | Do related fields/datasets agree under a shared rule? | Customer identity must resolve before an accepted order line can reference it. | Two sources can consistently repeat the same wrong value. |
| Timeliness | Is evidence available within the consumer freshness contract? | Compare source event/updated time with arrival/certification time under a declared SLO. | Fresh data can be invalid or inaccurate. |
| Accuracy | How closely does the value match trusted reality/reference truth? | Requires producer confirmation, authoritative reference data, or sampled ground truth. | Accuracy generally cannot be inferred from internal shape alone. |
These are not interchangeable percentages. A dataset may be 100% complete and 0% valid for a specific unit contract. Conversely, a nullable optional field can be incomplete without violating the consumer contract.
3. A quality rule is a contract with a denominator
Every metric needs a population. “Timestamp validity = 80%” is meaningful only if the denominator is the five rows in this isolated training batch and the rule is “parseable with explicit offset.” Production thresholds must be negotiated per field and consumer; this chapter does not invent a universal 99% target.
| Rule ID | Population / grain | Predicate | Observed result | Owner / action |
|---|---|---|---|---|
| DQ-COMP-001 | one raw customer row | name is present after whitespace trim | 3/4 = 75% | Customer Platform; block missing required name |
| DQ-TIME-001 | one raw order-line event | order_ts has explicit offset and parses | 4/5 = 80% | Order Platform; quarantine ambiguous time |
| DQ-UNIT-001 | one raw order-line event | unit maps to canonical item under versioned reference | 4/5 = 80% | Commerce Data Steward; CASE requires conversion evidence |
| DQ-UNI-001 | one business event key (order_id,line_no) | at most one accepted event revision | QO1/1 redelivered once | Analytics Engineering; deterministic dedupe |
4. Controlled failure: “the row counts match, therefore quality is good”
A developer loads all five order rows and checks only that the target has five rows. That check can pass while revenue is doubled by QO1 redelivery, QO2 is assigned to the wrong business day after guessing a time zone, and QO5 treats CASE as one item. The mechanism failure is that row count observes transport volume, not semantic validity, uniqueness, or unit/time interpretation.
from __future__ import annotationsimport copy, hashlib, json, re, unicodedatafrom datetime import datetime, timezonedef canonical_json(obj): return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)def sha(obj): return hashlib.sha256(canonical_json(obj).encode("utf-8")).hexdigest()def clean_space(value): return re.sub(r"\s+", " ", value.strip())def text_nfc(value): return unicodedata.normalize("NFC", clean_space(value))def match_token(value): return text_nfc(value).casefold()def parse_aware(value): if value.endswith("Z"): value = value[:-1] + "+00:00" dt = datetime.fromisoformat(value) if dt.tzinfo is None: raise ValueError("timestamp has no UTC offset") return dt.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")quality = { "customer_required_name_completeness": 3/4, "order_timestamp_validity": 4/5, "order_unit_validity": 4/5, "raw_order_rows": 5, "accepted_unique_before_repair": 2, "quarantined_before_repair": 2, "duplicates_before_repair": 1,}assert quality["raw_order_rows"] == ( quality["accepted_unique_before_repair"] + quality["quarantined_before_repair"] + quality["duplicates_before_repair"])print(quality)The repaired control is a partition: every input event must be explainably accepted, quarantined, or classified as duplicate. Matching a target row count to source transport count would be the wrong invariant because duplicates should not become facts and invalid semantics should not be guessed.
5. Accuracy is a special boundary
Completeness and validity can often be tested from the dataset and contract. Accuracy asks whether a value agrees with reality or a trusted authority. AtlasMart cannot prove that “North” is the customer’s true region merely because North is an allowed code. It needs a trusted system of record, producer attestation, physical count, ledger, or sampled verification. Therefore the quality report should say not yet measured rather than converting validity into an accuracy claim.
6. Lab assumptions and verification checklist
| Contract item | Chapter 13 assumption |
|---|---|
| Runtime | Python 3 standard library only; executable fixture is local and synthetic. |
| Accepted warehouse continuity | Chapter 12 production controls remain 8 paid lines / 5 paid orders / 10 units / 690 USD GMV; the Q-prefixed batch is isolated quality-test evidence. |
| Time | Source event timestamps require RFC 3339/ISO-style explicit offsets; canonical output is UTC. Naive timestamps are rejected, not guessed. |
| Units/currency | Order quantity canonical unit is item; EA/each are approved aliases. CASE has no conversion factor in the contract and is quarantined. Currency is USD. |
| Identity | Exact normalized email is a deliberately strong deterministic key only for this synthetic lab. Production identity rules require governance and privacy review. |
| Raw evidence | Raw payloads remain immutable; canonical fields are derived alongside raw hashes and rule versions. |
| Security | Synthetic identities only. Production PII must follow authorized access, retention, purpose, deletion, and incident policies. |
| Engine/storage | No cloud service or proprietary DQ product is required; vendor-specific enforcement/performance is outside the mandatory lab. |
- Verify production continuity is described separately from the Q-prefixed test batch.
- Verify all six dimensions have different predicates/evidence.
- Verify QO1/1 redelivery is not accepted twice.
- Verify naive QO2 time and CASE QO5 unit are quarantined rather than guessed.
- Verify the report never labels syntactic validity as accuracy.
7. Production judgment and bridge
Quality rules need consumers, owners, severity, and failure behavior. A rule can warn for an optional marketing attribute yet block a finance measure when currency semantics are unknown. Reruns must be idempotent, rejects observable, and rule-version changes reviewed because they can restate historical acceptance. Performance optimizations must not remove evidence needed for reconciliation. Lesson 2 makes representation deterministic without confusing standardization with truth.
Knowledge check
Check your understanding
- Why can a 100% complete field still be low quality?
- Why is accuracy harder to measure than validity?
- What must happen to every source event in this lab?
- Why is a single overall quality percentage risky?
- Does Chapter 13 change the accepted AtlasMart 690-USD production control?
Review the answers
1. Completeness only says required values are present; those values may violate domains, conflict with references, be stale, or be inaccurate.
2. Accuracy usually needs trusted external truth or verified observations, while validity can often be checked against an internal contract.
3. It must be explainably accepted, quarantined, or classified as a duplicate; silent disappearance is not allowed.
4. It can hide a blocking failure such as unknown currency/unit semantics behind many easy passing checks.
5. No. The malformed Q-prefixed batch is isolated training evidence unless an explicit governed migration says otherwise.
Authoritative references
- ISO/IEC 25012 — Data quality modelOfficial ISO catalogue entry for a general data-quality model; use it as background, not as a substitute for AtlasMart-specific acceptance rules.
- Unicode Standard Annex #15 — Unicode Normalization FormsAuthoritative normalization guidance used to explain NFC normalization without transliterating or overwriting the raw value.
- RFC 3339 — Date and Time on the InternetAuthoritative timestamp syntax reference supporting the rule that source timestamps carry an explicit offset before normalization to UTC.
- IANA — Time Zone DatabaseAuthoritative source for named civil time-zone identifiers when business semantics require a region-based zone rather than a fixed offset.
- Python documentation — unicodedataStandard-library Unicode database access used by the free/local lab.
- Python documentation — hashlibStandard-library hashing used for evidence fingerprints; hashes demonstrate identity of serialized evidence, not semantic correctness.