Operational Runbooks, Rollback, and Incident-Friendly Automation

1. Automation should help during incidents

Incident-friendly scripts expose current state, support safe diagnosis, preserve evidence, fail predictably, and have a tested rollback or recovery path.

flowchart TD
  A["automation change"] --> V{"verification"}
  V -->|"pass"| D["done"]
  V -->|"fail"| R["rollback or recovery"]
  R --> C["collect diagnostics"]
  C --> H["runbook / handoff"]

2. A runbook documents the operational contract

Document prerequisites, exact commands, expected outputs, health checks, failure modes, rollback, diagnostics, and escalation points.

3. Dry-run should describe planned mutations

if [[ $dry_run == true ]]; then
  printf 'PLAN service=%s version=%s action=deploy\n'                 "$service" "$version"
  return 0
fi

perform_deploy

4. Rollback is designed before rollout

rollback_release() {
  local previous_version=$1
  deploy_version "$previous_version" || return $?
  verify_health || return $?
}

5. Preserve the primary failure

if deploy_new_version; then
  :
else
  primary_status=$?

  if ! rollback_previous; then
    rollback_status=$?
    printf 'rollback_failed status=%d\n' "$rollback_status" >&2
  fi

  exit "$primary_status"
fi

6. Provide read-only status and diagnose modes

case ${1:-} in
  status) show_release_status ;;
  diagnose) collect_diagnostics ;;
  deploy) perform_deploy ;;
esac

Operators should not have to mutate production merely to discover what state it is in.

7. Error messages should suggest the next safe action

printf 'error: rollout health check failed\n' >&2
printf 'next: run "%s status" and inspect %s\n'               "$0" "$diag_dir" >&2

8. Hands-on lab: visible release state

mkdir -p "$HOME/devops-academy/bash/chapter20/lesson04"
cd "$HOME/devops-academy/bash/chapter20/lesson04"

cat > release-state.sh <<'EOF'
#!/usr/bin/env bash
set -u

state_dir=./state
mkdir -p -- "$state_dir"

status_cmd() {
  if [[ -f $state_dir/current ]]; then
    printf 'current=%s\n' "$(cat "$state_dir/current")"
  else
    printf 'current=unknown\n'
  fi
}

deploy_cmd() {
  local version=$1
  printf '%s\n' "$version" > "$state_dir/pending"
  printf '%s\n' "$version" > "$state_dir/current"
  rm -f -- "$state_dir/pending"
  printf 'deployed=%s\n' "$version"
}

case ${1:-status} in
  status) status_cmd ;;
  deploy) deploy_cmd "${2:?version required}" ;;
  *) printf 'usage: %s {status|deploy VERSION}\n' "$0" >&2; exit 64 ;;
esac
EOF

chmod u+x release-state.sh
./release-state.sh status
./release-state.sh deploy 1.0.0
./release-state.sh status

9. Knowledge check

10. Summary

Incident-friendly automation exposes state, supports safe inspection, preserves evidence, bounds recovery actions, and treats rollback as a tested workflow rather than an improvised emergency command.

11. Further reading

• Google Site Reliability Engineering — incident response and release engineering.
• Operational runbook practices.
• GNU Bash Reference Manual — traps and exit status.
• Change-management and observability guidance.