Automation Runbooks and Disaster-Recovery Exercises

1. A runbook is an executable operational contract

A useful runbook tells an authorized operator when to act, what evidence is required, which systems and identities are in scope, how to perform each step, how to validate success, when to stop, and how to communicate. It must work under pressure for someone other than its author. Links to changing dashboards or undocumented tribal knowledge are dependencies that need owners.

flowchart TD
  I["Declare incident and recovery authority"] --> A["Assess scope and select recovery point"]
  A --> P["Provision isolated recovery environment"]
  P --> R["Restore data, configuration, and dependencies"]
  R --> V["Validate integrity, security, and service behavior"]
  V --> G{"Go / no-go decision"}
  G -- no --> B["Rollback, preserve evidence, revise plan"]
  G -- yes --> T["Restore traffic in controlled stages"]
  T --> M["Monitor, reconcile, communicate, improve runbook"]
  B --> A

Keep business decisions outside shell scripts. A script can verify that a backup exists and restore it to a target; an incident commander should authorize which recovery point to use when newer data might be corrupt.

2. Every runbook needs context before commands

Runbook: Recover the application service

Purpose and scenarios:
  Region loss, destructive change, corrupted database, failed upgrade

Trigger and authority:
  Who may declare recovery and who approves data-loss trade-offs

Objectives:
  RPO 15 minutes; RTO 120 minutes; critical transactions reconciled

Prerequisites:
  Backup repository access, encryption keys, clean account/region,
  infrastructure code, DNS authority, vendor contacts

Inputs:
  Incident ID, selected recovery point, target environment, change ticket

Safety gates:
  Confirm production writes stopped; preserve evidence; validate target is isolated

Procedure:
  Provision, restore, configure, validate, approve, shift traffic, reconcile

Stop and rollback conditions:
  Integrity failure, wrong recovery point, unexpected outbound traffic,
  unresolved schema mismatch, security-control failure

Evidence and communications:
  Commands, timestamps, logs, approvers, customer updates, final metrics

Version the runbook with the service. Record last review and last successful exercise. A stale runbook should fail a readiness check before an incident exposes it.

3. Recovery depends on more than the protected data

Map identity providers, DNS, certificates, secrets, package repositories, container registries, time synchronization, network routes, firewall policy, observability, CI/CD, external APIs, licenses, and people. Determine which dependencies must be available, which can be substituted, and which credentials remain usable when the primary environment is compromised.

4. Automate deterministic steps and expose irreversible decisions

Automation reduces typing errors and makes timing repeatable, but it can accelerate the wrong decision. Require explicit target, incident ID, recovery point, and confirmation for destructive operations. Default to dry-run. Validate preconditions and write an evidence directory before changing state. Make reruns safe or detect partial completion.

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077

usage() { printf 'Usage: %s --incident ID --backup PATH --target DIR [--execute]\n' "$0"; }
incident= backup= target= mode=dry-run
while (($#)); do
  case $1 in
    --incident) incident=${2:?}; shift 2 ;;
    --backup) backup=${2:?}; shift 2 ;;
    --target) target=${2:?}; shift 2 ;;
    --execute) mode=execute; shift ;;
    *) usage; exit 64 ;;
  esac
done
[[ $incident =~ ^INC-[0-9]+$ ]] || { printf 'invalid incident ID\n' >&2; exit 64; }
[[ -f $backup ]] || { printf 'backup not found\n' >&2; exit 66; }
[[ $target == /srv/recovery/* ]] || { printf 'target outside recovery root\n' >&2; exit 64; }
[[ ! -e $target ]] || { printf 'target already exists\n' >&2; exit 73; }

evidence="/var/log/recovery/$incident-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -m 0700 -p "$evidence"
sha256sum "$backup" | tee "$evidence/backup.sha256"
tar -tzf "$backup" >"$evidence/archive-list.txt"

if [[ $mode == dry-run ]]; then
  printf 'validated only; rerun with --execute after approval\n' | tee "$evidence/RESULT"
  exit 0
fi

read -r -p "Type $incident to restore into $target: " confirmation
[[ $confirmation == "$incident" ]] || exit 1
mkdir -m 0700 -p "$target"
tar -xzf "$backup" -C "$target" --no-same-owner
printf 'restore_complete_utc=%s\n' "$(date -u --iso-8601=seconds)" \
  | tee "$evidence/RESULT"

For mature automation, use signed artifacts, policy checks, short-lived credentials, immutable logs, structured output, and a workflow engine that records approvals. Do not embed broad production credentials in the runbook repository.

5. Exercise depth should match the question

Use a progression. Tabletop exercises identify obvious ownership gaps cheaply; technical restores prove recoverability; controlled failovers validate the complete operating model. Production fault injection requires explicit risk approval and rollback.

6. Build scenarios with injects, constraints, and success criteria

A scenario should force specific decisions rather than announce the answer. Example: at 09:00 UTC, monitoring reports database checksum errors following a compromised administrative account. The newest three backup generations may be suspect. The primary region is available but must be treated as untrusted. The exercise team receives evidence in stages and must select a recovery point, establish a clean environment, rotate credentials, restore, validate, and communicate.

Exercise injects
T+00  Declare suspected destructive compromise
T+10  Primary identity provider becomes unavailable
T+20  Latest backup manifest fails signature validation
T+35  Clean-room network cannot reach the package repository
T+50  Business owner asks for an estimated data-loss window
T+70  Restored database starts but one reconciliation invariant fails
T+95  Approval requested for partial traffic restoration

Success criteria
- Recovery authority and roles established within 10 minutes
- Trusted recovery point selected with evidence
- Independent credentials and keys obtained
- No connection from clean room to untrusted production
- Application invariants pass before traffic
- Observed RPO and RTO calculated and communicated

7. Measure the complete recovery, not only data copy time

Also measure time to obtain authority, time to access keys, failed steps, manual interventions, alert quality, dependency substitutions, data reconciliation, security-control status, and communication cadence. Report objective misses as design findings, not operator blame.

8. Recovery validation must cover data, service, and security

1. Verify backup identity, manifest, signature, and selected point.
2. Validate filesystem and database consistency.
3. Check schema, migrations, roles, extensions, and application configuration.
4. Run synthetic reads and writes in isolation.
5. Reconcile counts, balances, sequence numbers, and domain invariants.
6. Confirm TLS, secrets, authentication, authorization, firewall, logging, and time.
7. Confirm monitoring, alerting, backups, and scheduled jobs in the recovered environment.
8. Shift a small portion of traffic, observe, then increase deliberately.

A process that starts is not necessarily a recovered service. Define business-level acceptance checks before the incident.

9. Preserve an exercise evidence bundle

exercise="DR-$(date -u +%Y%m%dT%H%M%SZ)"
root="$HOME/devops-academy/linux/chapter18/lesson05/$exercise"
mkdir -m 0700 -p "$root"/{commands,logs,validation}

{
  printf 'exercise=%s\n' "$exercise"
  printf 'started_utc=%s\n' "$(date -u --iso-8601=seconds)"
  printf 'host=%s\n' "$(hostname -f 2>/dev/null || hostname)"
  printf 'operator=%s\n' "$(id -un)"
} >"$root/MANIFEST"

script -q -f "$root/commands/operator.typescript"
# Run the approved exercise steps, then type exit.

journalctl --since '2 hours ago' --no-pager >"$root/logs/journal.txt"
systemctl --failed --no-pager >"$root/validation/failed-units.txt"
ss -lntup >"$root/validation/listeners.txt"
find "$root" -type f ! -name SHA256SUMS -print0 \
  | sort -z | xargs -0 sha256sum >"$root/SHA256SUMS"
chmod -R go-rwx "$root"

Terminal transcripts can contain secrets. Review and redact a copy for distribution while preserving the restricted original according to policy. Record all human decisions and approvals with timestamps.

10. Close the exercise with owned improvement work

Conduct a blameless review soon after the exercise. Separate observed facts from interpretations. Identify which controls detected problems, which steps were ambiguous, which dependencies failed, where access was too broad or unavailable, and which objectives were missed. Every action needs an owner, priority, due date, verification method, and link back to the runbook or automation change.

Finding:
  Recovery key retrieval required an unavailable primary-region approver.

Impact:
  Added 38 minutes to observed RTO.

Action:
  Establish two-region quorum and quarterly break-glass test.

Owner / due date:
  Security Platform / 2026-09-15

Verification:
  Complete a clean-room key retrieval with primary region disabled;
  attach audit evidence and update runbook section 4.2.

11. Capstone lab: run a bounded recovery exercise

Use the archive created in Lesson 2 or the rsync generation from Lesson 4. Assume the current source is corrupted. In a new directory, select a recovery point, record its checksum, restore it, validate expected files and metadata, measure duration, and write a short go/no-go decision. Do not overwrite the source.

set -Eeuo pipefail
backup=${1:?Usage: dr-lab BACKUP.tar.gz}
lab="$HOME/devops-academy/linux/chapter18/lesson05/dr-lab-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -m 0700 -p "$lab/restore"
start=$(date +%s)

sha256sum "$backup" | tee "$lab/selected-backup.sha256"
tar -tzf "$backup" | tee "$lab/archive-list.txt"
tar -xzf "$backup" -C "$lab/restore" --no-same-owner

test -d "$lab/restore/config"
test -s "$lab/restore/config/app.conf"
find "$lab/restore" -type f -print0 | sort -z | xargs -0 sha256sum \
  >"$lab/restored-files.sha256"

end=$(date +%s)
printf 'decision=GO\nobserved_rto_s=%d\nvalidated_utc=%s\n' \
  "$((end-start))" "$(date -u --iso-8601=seconds)" \
  | tee "$lab/DECISION.txt"

12. Common recovery-program mistakes

13. Knowledge check

14. Summary

• Runbooks define authority, prerequisites, steps, safety gates, validation, rollback, evidence, and communications.
• Automation should accelerate deterministic work without hiding irreversible decisions.
• Exercises progress from review and tabletop to technical restore and controlled failover.
• Measure complete recovery time, data loss, correctness, security, dependencies, and coordination.
• Recovery capability improves only when findings become owned, verified changes.

15. Further reading

• NIST SP 800-34 Rev. 1 — Contingency Planning Guide
• CISA incident response planning resources
• Google SRE — Managing Incidents
• Google SRE Workbook — Postmortem Culture
• systemd.service