Backup Strategies, Retention, and Restore Testing

1. Start with recovery requirements, not a backup command

A backup system exists to recover defined assets after defined failure scenarios. Inventory data, configuration, secrets, package state, identities, boot requirements, databases, object stores, and external dependencies. Name owners and acceptable loss. A command that copies everything nightly may still fail the business if the database is inconsistent or the required key is unavailable.

flowchart TD
  B["Business service and failure scenarios"] --> O["RPO, RTO, retention, compliance"]
  O --> S["Data scope and consistency method"]
  S --> A["Backup architecture and independent storage"]
  A --> V["Integrity checks and monitoring"]
  V --> T["Scheduled restore tests"]
  T --> E["Recovery evidence and design improvement"]

Define recovery units precisely. “Restore the server” can mean rebuilding the OS from code and restoring only state, or recovering a complete image. Immutable infrastructure usually favors reproducible rebuild plus data restore; legacy systems may require image-level recovery as well.

2. RPO and RTO express different limits

The recovery point objective (RPO) is the maximum acceptable data-loss interval. The recovery time objective (RTO) is the target time to restore service after the recovery decision. They are objectives, not guarantees, and must include dependencies, approvals, data transfer, validation, and traffic restoration.

A four-hour backup interval does not prove a four-hour RPO if jobs fail silently. Measure the age of the newest verified recoverable point, not merely the age of the newest file in a repository.

3. Combine backup methods according to change and recovery behavior

Most production designs combine methods: periodic full or synthetic-full backups, frequent incrementals or transaction logs, local snapshots for rapid rollback, and an independently retained off-site copy for destructive events.

4. Filesystem consistency is not application consistency

A crash-consistent snapshot represents what storage might contain after sudden power loss. Journaling can repair filesystem structure, but an application may still need transaction recovery. Multi-volume or distributed systems require coordination. Use database-native dumps, physical backup tools, write-ahead-log shipping, application quiescing, or orchestrated snapshots according to vendor guidance.

# Example pattern: create a PostgreSQL logical dump with explicit failure handling.
set -Eeuo pipefail
umask 077
stamp=$(date -u +%Y%m%dT%H%M%SZ)
out="/var/lib/backup/postgresql/app-$stamp.dump"

sudo -u postgres pg_dump \
  --format=custom \
  --compress=6 \
  --file="$out" \
  appdb

pg_restore --list "$out" >"$out.contents"
sha256sum "$out" "$out.contents" >"$out.sha256"

A successful dump exit status is necessary but not sufficient. Restore into an isolated instance, apply required roles and extensions, validate row-level and application invariants, and record the time and exact backup identifier.

5. Separate copies across failure and trust domains

The 3-2-1 guideline—three copies, on two media or systems, with one off-site—remains a useful starting point, but modern threats require more detail. Separate administrative credentials, accounts, regions, storage technologies, and deletion paths. Use immutability or object lock where appropriate. A ransomware operator who controls the production account should not be able to erase every recovery point.

Example recovery layout

Production:
  /srv/app and database cluster

Fast recovery tier:
  local encrypted snapshots, short retention, same region

Backup repository:
  deduplicated encrypted backups, separate service account

Recovery vault:
  replicated immutable copy, separate account/region,
  restricted deletion and key administration

Offline evidence:
  periodic exported manifest, configuration, and recovery runbook

Document correlated failures: region outage, account compromise, malicious administrator, key loss, software defect, accidental recursive deletion, and legal hold. Geographic distance alone is not independence when one credential can delete both locations.

6. Retention is a recoverability policy, not “keep everything”

Retention should cover operational rollback, delayed corruption discovery, security investigations, legal requirements, and cost. A grandfather-father-son pattern might retain daily, weekly, monthly, and annual points, but the exact policy must match data change and discovery windows. Expiration should be automated, reviewed, and protected from deleting the last known-good generation.

Measure real repository growth. Databases, virtual-machine images, encrypted files, and compaction can produce change rates much larger than business transaction volume suggests.

7. Encryption is only as recoverable as its keys

Encrypt in transit and at rest, but keep key management independent and tested. A backup encrypted with a key stored only on the failed host is not recoverable. Rotate credentials without invalidating old recovery points. Restrict restore permissions because backups often contain production secrets and historical personal data.

Key-control checklist
- Encryption algorithm and tool version documented
- Key owner and recovery approvers named
- Offline or separately controlled recovery copy available
- Rotation procedure tested against old backup generations
- Revocation and compromise response documented
- Restore operators can obtain keys during an outage
- Audit logs cover reads, restores, deletion, and policy changes

8. Verification has several layers

1. Job verification: the backup command completed and uploaded the intended scope.
2. Repository verification: manifests, chunks, indexes, and checksums are internally consistent.
3. Restore verification: files or databases can be reconstructed in isolation.
4. Application verification: the recovered service starts and satisfies business invariants.
5. Objective verification: observed data loss and elapsed recovery meet RPO and RTO.

Automated repository checks are valuable, but only a restore exercises permissions, keys, dependencies, naming, capacity, and human procedure. Sample restores should include both recent and older generations.

9. A restore test should produce auditable evidence

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077

backup=${1:?Usage: restore-test BACKUP_ARCHIVE}
case_id="restore-$(date -u +%Y%m%dT%H%M%SZ)"
root="$HOME/devops-academy/linux/chapter18/lesson03/$case_id"
restore="$root/restore"
mkdir -p "$restore"

started=$(date +%s)
sha256sum "$backup" | tee "$root/source.sha256"
tar -tzf "$backup" >"$root/archive-list.txt"
tar -xzf "$backup" -C "$restore" --no-same-owner

find "$restore" -xdev -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum >"$root/restored-files.sha256"

# Example recovery invariants.
test -s "$restore/config/app.conf"
grep -q '^mode=' "$restore/config/app.conf"
test -s "$restore/data/records.txt"

ended=$(date +%s)
printf 'case=%s\nbackup=%s\nduration_s=%d\nresult=pass\n' \
  "$case_id" "$backup" "$((ended-started))" \
  | tee "$root/RESULT.txt"

For a production exercise, add isolated provisioning, application startup, dependency substitution, synthetic transactions, reconciliation, security controls, and teardown. Preserve logs and exceptions without copying secrets into a broadly accessible report.

10. Hands-on lab: define a backup policy and test a restore

Create a policy for one fictional service with a 15-minute RPO and two-hour RTO. Identify state, consistency method, backup intervals, retention, copy locations, encryption, monitoring, restore test frequency, and owners. Then create a small tar archive and run the restore-test pattern above.

11. Common backup mistakes

12. Knowledge check

13. Summary

• Backup architecture begins with failure scenarios, data scope, RPO, RTO, retention, and ownership.
• Full, incremental, differential, snapshot, replication, and application-native methods serve different recovery needs.
• Independent credentials, regions, media, and deletion controls reduce correlated loss.
• Capacity depends on change rate and retention, not only source size.
• Routine isolated restore tests are the definitive proof of recoverability.

14. Further reading

• CISA StopRansomware Guide
• PostgreSQL backup and restore documentation
• MySQL backup and recovery
• restic documentation
• BorgBackup documentation