Standardize Names, Codes, Units, Time Zones, Encodings, and Reference Data

Learning outcomes

1. Standardization narrows representation, not meaning

AtlasMart sees " Ada Retail ", "ADA RETAIL", region values such as " north ", status values such as "PAID", and item-unit aliases EA/each. These differences can prevent joins or inflate distinct counts. Standardization maps multiple approved representations to a governed canonical representation while retaining raw evidence. It must never invent missing business meaning.

2. Names and encodings: normalize carefully

Unicode permits visually equivalent text to have different code-point sequences. The lab normalizes to NFC and collapses whitespace. The decomposed Béna Shop becomes Béna Shop. For matching, a separate case-folded token can be created. The display name should not be forced to uppercase, stripped of accents, or transliterated as a general rule: those transformations can destroy meaningful distinctions across languages.

from __future__ import annotationsimport copy, hashlib, json, re, unicodedatafrom datetime import datetime, timezonedef canonical_json(obj):    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)def sha(obj):    return hashlib.sha256(canonical_json(obj).encode("utf-8")).hexdigest()def clean_space(value):    return re.sub(r"\s+", " ", value.strip())def text_nfc(value):    return unicodedata.normalize("NFC", clean_space(value))def match_token(value):    return text_nfc(value).casefold()def parse_aware(value):    if value.endswith("Z"):        value = value[:-1] + "+00:00"    dt = datetime.fromisoformat(value)    if dt.tzinfo is None:        raise ValueError("timestamp has no UTC offset")    return dt.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")raw_name = "Be\u0301na   Shop"canonical = text_nfc(raw_name)match_key = match_token(raw_name)print(canonical)  # Béna Shopprint(match_key)  # béna shopassert canonical == "Béna Shop"

3. Codes and reference data need versions and owners

A reference-data table is part of lineage. If a steward later adds CASE → 12 item, that is not a cosmetic update: it can change quantities and historical measures. Version it, test it, and decide whether prior quarantined rows may be reprocessed.

4. Time zones: reject ambiguity, then normalize the instant

2026-09-24T10:00:00+00:00 and 2026-09-24T10:00:00Z identify the same UTC instant. 2026-09-24 12:00 does not state an offset or named zone. The lab rejects it. Assuming the server’s local time would make results machine-dependent and could shift dates around daylight-saving transitions.

from __future__ import annotationsimport copy, hashlib, json, re, unicodedatafrom datetime import datetime, timezonedef canonical_json(obj):    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)def sha(obj):    return hashlib.sha256(canonical_json(obj).encode("utf-8")).hexdigest()def clean_space(value):    return re.sub(r"\s+", " ", value.strip())def text_nfc(value):    return unicodedata.normalize("NFC", clean_space(value))def match_token(value):    return text_nfc(value).casefold()def parse_aware(value):    if value.endswith("Z"):        value = value[:-1] + "+00:00"    dt = datetime.fromisoformat(value)    if dt.tzinfo is None:        raise ValueError("timestamp has no UTC offset")    return dt.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")print(parse_aware("2026-09-24T10:00:00+00:00"))try:    parse_aware("2026-09-24 12:00")except ValueError as exc:    print("QUARANTINE:", exc)

5. End-to-end canonicalization evidence

6. Controlled failure: defaults and “helpful” coercion

A common shortcut replaces an unparseable quantity with zero, a missing region with Unknown, or CASE with item. These choices make the pipeline green but change business facts. Unknown members are appropriate only when the model’s policy says the business value is genuinely unknown; they are not a license to convert invalid source evidence into valid facts. In this lab, unresolved required semantics are quarantined with reason codes.

7. Lab verification and reset

• Q-C007 normalizes from decomposed Unicode to NFC Béna Shop.
• QO1 status/channel/unit normalize to paid/web/item.
• QO2 remains quarantined because its time zone is not stated.
• QO5 remains quarantined because CASE has no conversion factor.
• Delete generated local artifacts to reset; raw fixtures remain reproducible in the lesson script.

8. Production judgment and bridge

Canonical fields simplify joins and rules, but they increase governance surface: reference maps, Unicode policy, time-zone interpretation, and unit conversion all need ownership. Observability should count each normalization path so a sudden rise in aliases or rejects is visible. Lesson 3 now separates true duplicate delivery from uncertain entity resolution.

Knowledge check

Authoritative references

• ISO/IEC 25012 — Data quality modelOfficial ISO catalogue entry for a general data-quality model; use it as background, not as a substitute for AtlasMart-specific acceptance rules.
• Unicode Standard Annex #15 — Unicode Normalization FormsAuthoritative normalization guidance used to explain NFC normalization without transliterating or overwriting the raw value.
• RFC 3339 — Date and Time on the InternetAuthoritative timestamp syntax reference supporting the rule that source timestamps carry an explicit offset before normalization to UTC.
• IANA — Time Zone DatabaseAuthoritative source for named civil time-zone identifiers when business semantics require a region-based zone rather than a fixed offset.
• Python documentation — unicodedataStandard-library Unicode database access used by the free/local lab.
• Python documentation — hashlibStandard-library hashing used for evidence fingerprints; hashes demonstrate identity of serialized evidence, not semantic correctness.