Auditing, Data Classification, Retention, Legal Holds, Deletion, and Backup Governance

1. Governance follows copies, not table names

AtlasMart customer data may exist in the system of record, read replicas, search indexes, caches, CDC streams, application logs, analytics stores, snapshots, and backups. Data classification assigns sensitivity and handling requirements such as public, internal, confidential, payment-related, or personally identifiable information (PII). Retention defines how long a class may or must be kept. A legal hold suspends normal deletion for identified data under an authorized process. None of these rules automatically propagate because one primary row expires.

2. One deletion request becomes a distributed workflow

A governance inventory should name each copy, its owner, reason for existence, source version/lineage, retention policy, deletion mechanism, backup policy, legal-hold behavior, and proof of completion. Derived systems need deletion events or reconciliation; caches may need explicit invalidation plus TTL; CDC/log retention may have legitimate security/operational requirements; backups may be immutable until a defined expiry. The policy must describe these boundaries instead of claiming immediate physical erasure everywhere when the architecture cannot provide it.

3. Audit events need integrity and useful semantics

An audit record should identify actor/service identity, action, subject/resource, timestamp/order evidence, authorization context, outcome, and correlation identifier while excluding reusable secrets. Append-only or tamper-evident techniques can make unauthorized edits detectable. A hash chain, for example, links each event digest to the prior event. It can reveal later modification if the trusted head/checkpoint is protected, but it does not prove the original event was truthful or that no event was omitted before logging.

Audit availability also matters: placing all audit data on the same mutable datastore controlled by the same administrator weakens independent evidence. Consider write-once/append-only destinations, restricted access, and monitored export pipelines appropriate to the risk.

4. Deliberately wrong approach: TTL the primary record and declare deletion complete

The lab deletes customer c-7 from the primary store while leaving search, cache, CDC, and backup copies. This is a concrete governance gap. It is not fixed by shortening the primary TTL. The system needs a deletion contract that reaches active derived copies, a documented retention boundary for streams/backups, and verification that the intended state actually converged.

A legal hold is the opposite constraint: an automated deletion workflow must stop or reroute covered data according to authorized policy. “Privacy deletion job always wins” can itself violate governance.

5. AtlasMart lab: propagate deletion and verify audit integrity

import hashlib, json

stores={
    "primary": {"c-7":{"email":"c7@example.test","class":"PII","hold":False}},
    "search": {"c-7":{"email":"c7@example.test","source_version":3}},
    "cache": {"c-7":{"email":"c7@example.test","expires_at":200}},
    "cdc": {"c-7":{"email":"c7@example.test","offset":991}},
    "backup-2026-08-01": {"c-7":{"email":"c7@example.test","immutable_until":260}},
}

print("DELIBERATELY INCOMPLETE DELETE")
stores["primary"].pop("c-7")
print({name:("c-7" in rows) for name,rows in stores.items()})
print("primary deletion alone did not remove search/cache/CDC/backup copies")

print("\nGOVERNANCE PLAN")
plan={
    "primary":"delete now if no legal hold",
    "search":"propagate delete/tombstone and reconcile",
    "cache":"explicit invalidate plus bounded TTL",
    "cdc":"retain per governed stream policy; restrict access",
    "backup-2026-08-01":"expire according to backup retention; preserve legal hold if applicable",
}
for k,v in plan.items(): print(k, "->", v)

print("\nLEGAL HOLD")
held={"customer":"c-8","hold":True,"reason":"case-42"}
print("delete request for c-8:", "BLOCKED" if held["hold"] else "ALLOWED", held["reason"])

print("\nAUDIT HASH CHAIN")
events=[
    {"seq":1,"actor":"privacy-job","action":"delete-primary","subject":"c-7"},
    {"seq":2,"actor":"search-sync","action":"delete-search","subject":"c-7"},
    {"seq":3,"actor":"cache-sync","action":"invalidate-cache","subject":"c-7"},
]
prev="GENESIS"; chain=[]
for e in events:
    payload=json.dumps(e,sort_keys=True,separators=(",",":"))
    digest=hashlib.sha256((prev+payload).encode()).hexdigest()
    chain.append((e,digest)); prev=digest
print("final audit hash:", chain[-1][1][:16])

# Tamper with event 2 and prove recomputation diverges.
tampered=[dict(e) for e in events]; tampered[1]["subject"]="c-999"
prev="GENESIS"; tampered_hashes=[]
for e in tampered:
    payload=json.dumps(e,sort_keys=True,separators=(",",":"))
    digest=hashlib.sha256((prev+payload).encode()).hexdigest(); tampered_hashes.append(digest); prev=digest
print("tamper detected:", tampered_hashes[-1] != chain[-1][1])

print("\nRECONCILE ACTIVE DERIVED STORES")
for name in ["search","cache"]:
    stores[name].pop("c-7",None)
print({name:("c-7" in rows) for name,rows in stores.items()})
print("backup and CDC remain governed by their own retention/access policies; erase claims must describe those boundaries precisely")

6. Backup governance is part of deletion governance

Backups exist specifically to restore older state, so they conflict with simplistic “delete instantly everywhere” assumptions. Organizations commonly use bounded backup retention, encryption, restricted restore access, deletion-on-expiry, documented legal-hold exceptions, and restore-time reapplication of deletion/tombstone records. The exact policy depends on law, contracts, threat model, and product capabilities.

Observe deletion queue age, derived-store mismatches, retained backup inventory, legal-hold overrides, audit-chain/export health, access to sensitive logs, and restore tests that verify governance metadata survives disaster recovery. Chapter 22 will go deeper into backup/restore and disaster-recovery engineering.

References

Foundational claims use standards, specifications, primary research, or current official documentation where practical. Product references are optional implementation anchors; the mandatory labs are vendor-neutral.

• NIST Privacy Framework — Risk-management framework for privacy engineering and governance.
• NIST SP 800-53 Rev. 5 — Audit/accountability, media protection, access control, retention-related and system-integrity controls.
• NIST SP 800-92 — Security log-management guidance.
• OWASP Logging Cheat Sheet — Security-event logging, sensitive-data exclusion, and log-protection guidance.