Safe Temporary State, Checkpoints, and Recovery

1. Production automation often needs resumable state

Long workflows may download artifacts, validate inputs, update several targets, or perform costly remote actions. If the script fails halfway through, restarting from zero can be wasteful or unsafe. Checkpoints make progress explicit.

flowchart LR
  S1["step 1"] --> C1["checkpoint 1"]
  C1 --> S2["step 2"]
  S2 --> C2["checkpoint 2"]
  C2 --> S3["step 3"]

2. Separate ephemeral temporary state from recovery state

3. Use a private workspace for one run

workdir=$(mktemp -d) || exit 1
trap 'rm -rf -- "$workdir"' EXIT

Scratch files can live under a unique run directory so concurrent invocations do not collide.

4. Checkpoints should be written only after success

if upload_artifact; then
  printf '%s\n' "$artifact_id" > "$state_dir/upload.done"
else
  exit $?
fi

A checkpoint means the step completed according to its contract. Never create it before the corresponding operation is confirmed.

5. Write checkpoint metadata atomically

tmp=$(mktemp "$state_dir/.checkpoint.XXXXXX") || exit 1

printf 'artifact_id=%s\n' "$artifact_id" > "$tmp"
chmod 0600 -- "$tmp"
mv -- "$tmp" "$state_dir/upload.done"

Atomic-style replacement prevents readers from observing a half-written checkpoint file.

6. Resume logic should verify, not blindly trust markers

if [[ -f $state_dir/upload.done ]]; then
  if remote_artifact_still_exists; then
    printf 'resume: upload already complete\n'
  else
    printf 'checkpoint stale; reconciling\n' >&2
    rm -f -- "$state_dir/upload.done"
    upload_artifact
  fi
fi

7. Version checkpoint formats

cat > "$tmp" <<EOF
format_version=1
run_id=$run_id
artifact_id=$artifact_id
EOF

Once recovery data survives across script versions, it becomes an interface. Version it so future code can reject or migrate incompatible formats. For complex state, prefer JSON or another structured format.

8. Recovery state may contain sensitive identifiers

umask 077
mkdir -p -- "$state_dir"
chmod 0700 -- "$state_dir"

Checkpoint directories can contain credentials, remote object identifiers, URLs, or operational metadata. Protect them according to their contents.

9. Recovery can resume, reconcile, or rollback

10. Do not delete recovery evidence too early

An EXIT trap that deletes every state file can destroy the evidence needed to resume or diagnose a failed operation. Cleanup policies should distinguish disposable scratch data from durable recovery state.

11. Hands-on lab: resumable three-step workflow

mkdir -p "$HOME/devops-academy/bash/chapter12/lesson05"
cd "$HOME/devops-academy/bash/chapter12/lesson05"

state_dir=./state
mkdir -p -- "$state_dir"

step() {
  local number=$1
  local marker="$state_dir/step${number}.done"

  if [[ -f $marker ]]; then
    printf 'RESUME step=%s already complete\n' "$number"
    return 0
  fi

  printf 'RUN step=%s\n' "$number"
  sleep 1

  tmp=$(mktemp "$state_dir/.step${number}.XXXXXX") || return 1
  printf 'completed_at=%s\n' \
    "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" > "$tmp"
  mv -- "$tmp" "$marker"

  printf 'CHECKPOINT step=%s\n' "$number"
}

step 1
step 2
step 3

printf '%s\n' 'Run the script again: all three steps should resume as completed.'

12. Knowledge check

13. Summary

Reliable recovery separates disposable scratch state from durable checkpoints. Write checkpoints after confirmed success, update them atomically, version their format, protect sensitive metadata, and reconcile external state before trusting old markers.

14. Further reading

• GNU Coreutils manuals — mv, mkdir, chmod.
• GNU Bash Reference Manual — traps and shell parameters.
• Database and distributed-systems literature — checkpoints and write-ahead state.
• Site Reliability Engineering practices — recovery, reconciliation, and idempotency.