Chapter 13 · Data Quality Engineering: Validation, Standardization, Matching, Reconciliation, and Quarantine
Quarantine Bad Records, Preserve Raw Evidence, Repair/Reprocess, and Avoid Silent Coercion
Quarantine invalid records with immutable evidence, repair them through an authorized manifest, and prove that reprocessing is idempotent instead of silently coercing or dropping data.
Learning outcomes
Quarantine records that cannot safely enter certified warehouse state while preserving immutable raw evidence.
Record reason codes, rule versions, ownership, hashes, and status for each reject.
Apply an authorized repair manifest instead of silent defaults or manual source-file edits.
Reprocess repaired evidence idempotently and leave unresolved records quarantined.
Explain how quarantine protects correctness without becoming a permanent unowned data swamp.
1. Quarantine is an explicit state, not a trash bin
QO2 has a timestamp with no offset. QO5 uses CASE with no conversion rule. Both are structurally readable, but accepting either would require guessing. Quarantine isolates the record from certified downstream use while preserving the exact raw payload, the rules that failed, and the evidence needed to repair or reject it permanently.
| Key | Reason | Why no default? |
|---|---|---|
| QO2/1 | DQ-TIME-001:order_ts_requires_offset | The missing semantics can change event date, quantity, or measure meaning. |
| QO5/1 | DQ-UNIT-001:unknown_unit_or_conversion | The missing semantics can change event date, quantity, or measure meaning. |
2. Minimum reject record
| Field | Example / purpose |
|---|---|
| raw_hash | 0b99cb58cda5838a73… |
| source identity | order line key + source/batch/sequence |
| rule IDs | DQ-TIME-001 or DQ-UNIT-001 |
| rule version | dq-v1.0 |
| first seen / last attempted | supports aging and retry observability |
| owner / disposition | producer, steward, or engineering queue |
| repair_id | null until authorized correction exists |
| raw payload | immutable access-controlled evidence; may be referenced rather than duplicated in production |
Quarantine tables can contain sensitive raw data. Access must be at least as restrictive as the source, often more restrictive because invalid rows may contain unexpected content.
3. Controlled failure: “make the pipeline pass”
Replacing QO2’s missing offset with the application server time zone and converting QO5 CASE to one item would make the batch load. It would also fabricate semantics. Silent coercion hides uncertainty from consumers and prevents later reconciliation. Dropping the rows is no better: finance sees lower totals with no explanation.
Safe failure behavior
Certified facts exclude unresolved rows, while quarantine counts/amounts remain observable and tied to owners. Consumers can see that certified completeness is constrained by rejects.
4. Authorized repair uses evidence and scope
The synthetic producer confirms that QO2’s naive timestamp was
intended as UTC. An authorized repair manifest pins the exact
raw hash and changes only order_ts. If the raw hash
differs, the manifest must not apply. This prevents a generic
“fix QO2” instruction from mutating a later, different payload.
repair = { "repair_id": "R-2026-09-24-001", "approved_by": "Synthetic Data Steward", "target_raw_hash": "<exact QO2 raw SHA-256>", "changes": {"order_ts": "2026-09-24T12:00:00Z"}, "reason": "producer confirmed source timestamp is UTC", "rule_version": "dq-v1.0",}# Algorithm:# 1. assert hash(raw_payload) == target_raw_hash# 2. derive a repaired candidate; never mutate immutable raw evidence# 3. rerun the same standardization + validation + dedupe rules# 4. record repair_id on the accepted/rejected lineage# 5. replay: accepted semantic checksum must remain unchangedAfter this repair, QO2 becomes valid. QO5 remains quarantined because no CASE conversion was authorized. Accepted test-batch amount rises from 100 USD to 150 USD; this is an explicit change in the isolated lab, not a silent rewrite of the Chapter 12 production continuity.
5. Idempotent reprocessing
Reprocessing the same authorized QO2 repair twice must not create two facts. The business event key is still QO2/1, and deduplication plus deterministic canonicalization produces the same accepted checksum on replay.
| State | Accepted unique rows | Accepted USD | Remaining quarantine | Meaning |
|---|---|---|---|---|
| Before repair | 2 | 100 | 2 | QO1 + QO6 accepted; QO2 time and QO5 unit blocked. |
| After authorized QO2 repair | 3 | 150 | 1 | QO2 enters certified state; QO5 stays blocked. |
| Replay same repair | 3 | 150 | 1 | No duplicate fact; semantic checksum is identical to repaired state. |
6. Repair/reprocess observability
Track reject age, reason distribution, repair throughput, reprocess attempts, permanent disposition, and downstream completeness impact. A growing quarantine with no owner is not a quality system; it is delayed data loss. Alerts should distinguish source regressions from expected exceptional data, and rule-version deployments should report how many historical rows would change classification before promotion.
7. Cleanup/reset and bridge
- Delete any locally materialized reject/repair JSON files.
- Recreate the Q fixtures from the lesson and verify the same raw hashes.
- Apply only R-2026-09-24-001 to QO2.
- Confirm QO5 remains quarantined.
- Replay QO2 and verify the repaired semantic checksum is unchanged.
Lesson 5 closes the chapter by reconciling every row and semantic amount across source, quality states, and warehouse facts.
Knowledge check
Check your understanding
- Why is quarantine better than dropping an invalid row?
- Why pin a repair to a raw hash?
- Should a repair mutate the raw source payload?
- What proves replay idempotency here?
- Why does QO5 stay quarantined?
Review the answers
1. It protects certified data while preserving evidence, ownership, and the ability to repair/reprocess or explain missing completeness.
2. It ensures the authorization applies to the exact evidence reviewed, not to a later payload that merely shares a key.
3. No. Raw evidence remains immutable; the repair creates a derived candidate with lineage to the manifest.
4. The same accepted business keys, values, totals, and semantic checksum after replay.
5. No governed CASE-to-item conversion is available, so converting it would invent measurement semantics.
Summary and next step
This lesson established the mechanism and production boundaries for Quarantine Bad Records, Preserve Raw Evidence, Repair/Reprocess, and Avoid Silent Coercion while preserving AtlasMart’s declared grain, governed metrics, history, and reconciliation evidence. Continue to Build Reconciliation Tests from Source Counts/Sums/Checksums to Warehouse Facts and Dimensions with those contracts unchanged unless an explicit, tested migration says otherwise.
Authoritative references
- ISO/IEC 25012 — Data quality modelOfficial ISO catalogue entry for a general data-quality model; use it as background, not as a substitute for AtlasMart-specific acceptance rules.
- Unicode Standard Annex #15 — Unicode Normalization FormsAuthoritative normalization guidance used to explain NFC normalization without transliterating or overwriting the raw value.
- RFC 3339 — Date and Time on the InternetAuthoritative timestamp syntax reference supporting the rule that source timestamps carry an explicit offset before normalization to UTC.
- IANA — Time Zone DatabaseAuthoritative source for named civil time-zone identifiers when business semantics require a region-based zone rather than a fixed offset.
- Python documentation — unicodedataStandard-library Unicode database access used by the free/local lab.
- Python documentation — hashlibStandard-library hashing used for evidence fingerprints; hashes demonstrate identity of serialized evidence, not semantic correctness.