Replication Is Not Backup: Operator Error, Corruption, Bad Deployments, and Ransomware

1. Replication copies current state—including bad state

A replica exists primarily to preserve availability/durability against selected component failures and to serve reads or failover according to a replication protocol. If an authenticated operator deletes every order, an application bug writes corrupt totals, or compromised credentials encrypt/overwrite the dataset, replication may faithfully reproduce the destructive state everywhere. That is correctness for replication, not a backup failure.

The relevant distinction is failure independence: can the event that damages live state also modify or delete the recovery copy?

2. Recovery copies need different failure and administrative domains

An independent backup can be stored outside the primary cluster, account/project, credential set, or writable control path. Immutability means the backup cannot be altered/deleted until policy permits; an offline or logically isolated copy reduces exposure to live credentials and ransomware. No single label guarantees safety: an immutable bucket whose root credential is continuously exposed may still share a dangerous administrative failure domain.

3. Blast-radius worksheet

4. AtlasMart lab: perfectly replicated destruction

from copy import deepcopy

seed = {
    "o-1":{"status":"paid","total":80},
    "o-2":{"status":"created","total":45},
    "o-3":{"status":"shipped","total":120},
}
replicas = {name:deepcopy(seed) for name in ("A","B","C")}
immutable_backup = deepcopy(seed)
mutable_snapshot_same_admin = deepcopy(seed)

print("HEALTHY REPLICATION")
for n,d in replicas.items(): print(n, sorted(d))

print("\nDESTRUCTIVE OPERATOR COMMAND PROPAGATES PERFECTLY")
for d in replicas.values():
    d.clear()
for n,d in replicas.items(): print(n, d)
print("replica agreement after damage:", len({tuple(sorted(d)) for d in replicas.values()}) == 1)

print("\nSAME ADMIN DOMAIN IS NOT INDEPENDENCE")
mutable_snapshot_same_admin.clear()  # compromised credential deleted it too
print("mutable snapshot:", mutable_snapshot_same_admin)

print("\nIMMUTABLE/SEPARATE RECOVERY COPY")
print("backup still contains:", sorted(immutable_backup))
restored = deepcopy(immutable_backup)
print("restored:", sorted(restored))

print("\nCORRUPTION ALSO REPLICATES")
for d in replicas.values():
    d.update(deepcopy(restored))
    d["o-2"]["total"] = -999
print("live copies corrupted total:", [replicas[n]["o-2"]["total"] for n in replicas])
print("independent backup total:", immutable_backup["o-2"]["total"])
print("lesson: availability replicas and recovery copies solve different failure classes")

5. Backup security and ransomware judgment

Protect recovery copies with separate identities, least privilege, write-once/retention controls where appropriate, encryption, key recovery, access logs, delete protection, and restore-network isolation. Test the credentials used during disaster recovery: an offline copy that requires an expired, undocumented, or destroyed key is not operationally recoverable.

6. Production judgment

Design live redundancy and recovery separately. Measure replica health for availability, but measure backup age, independent-copy count, immutability retention, delete permissions, restore test age, and recovery-key availability for recoverability. Do not use “three replicas” as evidence of ransomware/operator-error recovery. The next lesson converts these copies into explicit RPO/RTO and dependency-order decisions.

References

Foundational claims use standards, specifications, primary research, or current official documentation where practical. Product references are optional implementation anchors; the mandatory labs are vendor-neutral.

• PostgreSQL 18 — Backup and Restore — Official backup guidance; regular backups remain necessary even when other durability mechanisms exist.
• NIST SP 800-209 — Security guidance for storage infrastructure, including recovery and protection boundaries.
• CISA StopRansomware Guide — Official guidance emphasizing resilient backups and ransomware preparation.
• Apache Cassandra — Repair — Shows repair handles replica divergence but does not protect against operator-error or corruption failure classes.