Checkpoint Lab — Failure Recovery, Reruns, Idempotency, Rollbacks, and Incident-Safe Automation

1. Checkpoint scenario and safety boundary

You will use a throwaway repository and a dedicated branch named incident-state as a fake durable deployment target. Attempt 1 writes one deterministic release and receipt to that branch, updates a current pointer, and then intentionally exits 23. Because the branch is remote repository state, the side effect survives the failed hosted runner. Before rerunning, you preserve attempt 1. Attempt 2 then either reconciles the existing deployment or rolls back to the exact previous pointer.

2. Current assumptions — timestamp them in your evidence

3. Repository/resource/credential preflight

1. Create a new disposable repository, for example gha-ch33-incident-drill. Do not use this Academy repository or any production repository.
2. Confirm gh auth status and that you have permission to push branches in this disposable repository.
3. Record gh repo view --json nameWithOwner,id,visibility,defaultBranchRef. The marker file will bind the state branch to the exact repository name and ID.
4. Use no real secrets. GITHUB_TOKEN is only used by checkout/git push under the job's narrow contents: write permission.
5. Ensure there is no pre-existing incident-state branch or inspect it before replacing anything. If it exists unexpectedly, stop.

gh auth status
gh repo view --json nameWithOwner,id,visibility,defaultBranchRef

git status --short
git branch --all --list '*incident-state*'

4. Seed the durable fake target branch

The branch contains an authorization marker plus the current target pointer. The workflow refuses mutation unless the marker exactly matches repository name, repository ID and lab purpose. This protects against copying the workflow into another repository and accidentally pushing an unexpected state branch.

# Run only in a throwaway repository dedicated to this lab.
# Replace the repository-id placeholder with: gh repo view --json id --jq .id

git switch --orphan incident-state
# Remove tracked files from the orphan worktree while preserving .git.
git rm -rf . 2>/dev/null || true
mkdir -p state/releases state/receipts
cat > ALLOW_INCIDENT_LAB <<EOF
repository=OWNER/REPO
repository_id=REPOSITORY_ID
purpose=gha-ch33-incident-drill
EOF
printf 'baseline release\n' > state/releases/baseline.txt
baseline_hex=$(sha256sum state/releases/baseline.txt | awk '{print $1}')
cat > state/current.env <<EOF
active_release=baseline
active_sha=0000000000000000000000000000000000000000
active_digest=sha256:$baseline_hex
origin_run_id=baseline
origin_attempt=0
EOF
git add ALLOW_INCIDENT_LAB state/current.env state/releases/baseline.txt
git commit -m 'lab: seed disposable incident target ledger'
git push -u origin incident-state
git switch main

Before continuing, switch back to main, replace the placeholders before the commit, and inspect the remote branch with git ls-remote origin refs/heads/incident-state.

5. Install the checkpoint workflow

Save the following as .github/workflows/ch33-incident-drill.yml on the default branch. It separates source checkout from the writable target checkout, derives one artifact digest/idempotency key, reads target state before mutation, and serializes staging updates. The always() steps are deliberately read-only except for uploading an evidence artifact; no rollback or delete operation hides inside failure cleanup.

name: Chapter 33 — Incident-safe deployment drill
run-name: "incident-drill / ${{ inputs.recovery_mode }} / ${{ github.sha }}"

on:
  workflow_dispatch:
    inputs:
      recovery_mode:
        description: Recovery after the injected first-attempt failure
        type: choice
        options: [reconcile, rollback]
        default: reconcile
        required: true
      inject_fault_once:
        description: Fail after the durable activation on attempt 1
        type: boolean
        default: true
        required: true

permissions: {}

concurrency:
  group: incident-lab-${{ github.repository_id }}-staging
  queue: max

jobs:
  deploy:
    name: guarded-fake-deployment
    runs-on: ubuntu-24.04
    timeout-minutes: 10
    permissions:
      contents: write
    env:
      STATE_BRANCH: incident-state
      STATE_DIR: target-state
      SOURCE_DIR: source
      ENVIRONMENT_NAME: staging
    steps:
      - name: Checkout exact source revision without persisted credentials
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: ${{ github.sha }}
          path: source
          persist-credentials: false

      - name: Checkout durable disposable target ledger
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: incident-state
          path: target-state
          fetch-depth: 1

      - name: Preflight exact disposable target guard
        shell: bash
        working-directory: target-state
        env:
          EXPECTED_REPOSITORY: ${{ github.repository }}
          EXPECTED_REPOSITORY_ID: ${{ github.repository_id }}
        run: |
          set -euo pipefail
          test -f ALLOW_INCIDENT_LAB
          grep -Fx "repository=$EXPECTED_REPOSITORY" ALLOW_INCIDENT_LAB
          grep -Fx "repository_id=$EXPECTED_REPOSITORY_ID" ALLOW_INCIDENT_LAB
          grep -Fx 'purpose=gha-ch33-incident-drill' ALLOW_INCIDENT_LAB
          git status --short

      - name: Build one deterministic fake artifact and operation identity
        id: identity
        shell: bash
        env:
          SOURCE_SHA: ${{ github.sha }}
          REPOSITORY_ID: ${{ github.repository_id }}
          RUN_ID: ${{ github.run_id }}
        run: |
          set -euo pipefail
          mkdir -p evidence "$SOURCE_DIR/dist"
          printf 'repository_id=%s\nsource_sha=%s\nenvironment=%s\n' \
            "$REPOSITORY_ID" "$SOURCE_SHA" "$ENVIRONMENT_NAME" \
            > "$SOURCE_DIR/dist/app.txt"
          hex=$(sha256sum "$SOURCE_DIR/dist/app.txt" | awk '{print $1}')
          digest="sha256:$hex"
          key=$(printf '%s|%s|%s' "$REPOSITORY_ID" "$ENVIRONMENT_NAME" "$digest" \
            | sha256sum | awk '{print $1}')
          printf 'digest=%s\nidempotency_key=%s\n' "$digest" "$key" >> "$GITHUB_OUTPUT"
          printf 'run_id=%s\nsource_sha=%s\ndigest=%s\nidempotency_key=%s\n' \
            "$RUN_ID" "$SOURCE_SHA" "$digest" "$key" | tee evidence/identity.txt

      - name: Inspect target before mutation
        shell: bash
        env:
          KEY: ${{ steps.identity.outputs.idempotency_key }}
        run: |
          set -euo pipefail
          mkdir -p evidence
          printf '%s\n' '--- current target before mutation ---' | tee evidence/before.txt
          cat "$STATE_DIR/state/current.env" | tee -a evidence/before.txt
          if [[ -f "$STATE_DIR/state/receipts/$KEY.env" ]]; then
            printf '%s\n' '--- existing receipt ---' | tee -a evidence/before.txt
            cat "$STATE_DIR/state/receipts/$KEY.env" | tee -a evidence/before.txt
          else
            echo 'receipt=absent' | tee -a evidence/before.txt
          fi

      - name: Activate once or reconcile existing receipt
        id: activate
        shell: bash
        env:
          KEY: ${{ steps.identity.outputs.idempotency_key }}
          DIGEST: ${{ steps.identity.outputs.digest }}
          SOURCE_SHA: ${{ github.sha }}
          RUN_ID: ${{ github.run_id }}
          ATTEMPT: ${{ github.run_attempt }}
        run: |
          set -euo pipefail
          cd "$STATE_DIR"
          receipt="state/receipts/$KEY.env"
          release="state/releases/${DIGEST#sha256:}.txt"
          mkdir -p state/receipts state/releases

          # Serialize with GitHub concurrency, then still pull exact latest ledger state.
          git pull --ff-only origin "$STATE_BRANCH"

          if [[ -f "$receipt" ]]; then
            echo 'Existing logical deployment found; reconcile instead of creating another.'
            grep -Fx "source_sha=$SOURCE_SHA" "$receipt"
            grep -Fx "digest=$DIGEST" "$receipt"
            first_run_id=$(sed -n 's/^first_run_id=//p' "$receipt")
            test "$first_run_id" = "$RUN_ID" || {
              echo 'Receipt belongs to another workflow run; refuse mutation.' >&2
              exit 41
            }
            test -f "$release"
            actual="sha256:$(sha256sum "$release" | awk '{print $1}')"
            test "$actual" = "$DIGEST"
            echo 'created=false' >> "$GITHUB_OUTPUT"
          else
            previous_release=$(sed -n 's/^active_release=//p' state/current.env)
            previous_sha=$(sed -n 's/^active_sha=//p' state/current.env)
            previous_digest=$(sed -n 's/^active_digest=//p' state/current.env)

            cp "../$SOURCE_DIR/dist/app.txt" "$release"
            cat > "$receipt" <<EOF
key=$KEY
environment=$ENVIRONMENT_NAME
source_sha=$SOURCE_SHA
digest=$DIGEST
first_run_id=$RUN_ID
first_attempt=$ATTEMPT
previous_release=$previous_release
previous_sha=$previous_sha
previous_digest=$previous_digest
status=activated
EOF
            cat > state/current.env <<EOF
active_release=${DIGEST#sha256:}
active_sha=$SOURCE_SHA
active_digest=$DIGEST
origin_run_id=$RUN_ID
origin_attempt=$ATTEMPT
EOF
            git config user.name 'gha-ch33-lab'
            git config user.email 'gha-ch33-lab@users.noreply.github.com'
            git add state
            git commit -m "lab: activate $KEY from run $RUN_ID attempt $ATTEMPT"
            git push origin HEAD:"$STATE_BRANCH"
            echo 'created=true' >> "$GITHUB_OUTPUT"
          fi

      - name: Inject one failure after the durable side effect
        if: ${{ inputs.inject_fault_once && github.run_attempt == '1' }}
        shell: bash
        env:
          KEY: ${{ steps.identity.outputs.idempotency_key }}
          DIGEST: ${{ steps.identity.outputs.digest }}
        run: |
          set -euo pipefail
          printf 'Injected failure after activation. key=%s digest=%s\n' "$KEY" "$DIGEST" \
            | tee evidence/first-failure.txt
          exit 23

      - name: Reconcile or roll back on a later attempt
        if: ${{ !inputs.inject_fault_once || github.run_attempt != '1' }}
        shell: bash
        env:
          KEY: ${{ steps.identity.outputs.idempotency_key }}
          DIGEST: ${{ steps.identity.outputs.digest }}
          RECOVERY_MODE: ${{ inputs.recovery_mode }}
          RUN_ID: ${{ github.run_id }}
          ATTEMPT: ${{ github.run_attempt }}
        run: |
          set -euo pipefail
          cd "$STATE_DIR"
          receipt="state/receipts/$KEY.env"
          git pull --ff-only origin "$STATE_BRANCH"
          test -f "$receipt"
          grep -Fx "first_run_id=$RUN_ID" "$receipt"

          current=$(sed -n 's/^active_digest=//p' state/current.env)
          if [[ "$RECOVERY_MODE" == 'reconcile' ]]; then
            test "$current" = "$DIGEST" || {
              echo 'Current target no longer equals failed deployment; stop.' >&2
              exit 42
            }
            printf 'recovered_attempt=%s\nstatus=healthy\n' "$ATTEMPT" >> "$receipt"
          elif [[ "$RECOVERY_MODE" == 'rollback' ]]; then
            test "$current" = "$DIGEST" || {
              echo 'Current target changed; refuse stale rollback.' >&2
              exit 43
            }
            prev_release=$(sed -n 's/^previous_release=//p' "$receipt")
            prev_sha=$(sed -n 's/^previous_sha=//p' "$receipt")
            prev_digest=$(sed -n 's/^previous_digest=//p' "$receipt")
            test -f "state/releases/$prev_release.txt"
            actual_prev="sha256:$(sha256sum "state/releases/$prev_release.txt" | awk '{print $1}')"
            test "$actual_prev" = "$prev_digest"
            cat > state/current.env <<EOF
active_release=$prev_release
active_sha=$prev_sha
active_digest=$prev_digest
origin_run_id=$RUN_ID
origin_attempt=$ATTEMPT
EOF
            printf 'recovered_attempt=%s\nstatus=rolled_back\nrollback_digest=%s\n' \
              "$ATTEMPT" "$prev_digest" >> "$receipt"
          else
            echo 'Unknown recovery mode' >&2
            exit 44
          fi

          git config user.name 'gha-ch33-lab'
          git config user.email 'gha-ch33-lab@users.noreply.github.com'
          git add state
          git commit -m "lab: $RECOVERY_MODE $KEY run $RUN_ID attempt $ATTEMPT"
          git push origin HEAD:"$STATE_BRANCH"

      - name: Verify final target state
        if: ${{ !inputs.inject_fault_once || github.run_attempt != '1' }}
        shell: bash
        env:
          KEY: ${{ steps.identity.outputs.idempotency_key }}
          DIGEST: ${{ steps.identity.outputs.digest }}
          RECOVERY_MODE: ${{ inputs.recovery_mode }}
        run: |
          set -euo pipefail
          git -C "$STATE_DIR" pull --ff-only origin "$STATE_BRANCH"
          receipt="$STATE_DIR/state/receipts/$KEY.env"
          test -f "$receipt"
          if [[ "$RECOVERY_MODE" == 'reconcile' ]]; then
            grep -Fx 'status=healthy' "$receipt"
            grep -Fx "active_digest=$DIGEST" "$STATE_DIR/state/current.env"
          else
            grep -Fx 'status=rolled_back' "$receipt"
            rollback_digest=$(sed -n 's/^rollback_digest=//p' "$receipt" | tail -1)
            grep -Fx "active_digest=$rollback_digest" "$STATE_DIR/state/current.env"
          fi
          cp "$STATE_DIR/state/current.env" evidence/final-current.env
          cp "$receipt" evidence/final-receipt.env

      - name: Preserve attempt evidence without another privileged mutation
        if: ${{ always() }}
        shell: bash
        env:
          RUN_ID: ${{ github.run_id }}
          ATTEMPT: ${{ github.run_attempt }}
          SOURCE_SHA: ${{ github.sha }}
          ACTOR: ${{ github.actor }}
          TRIGGERING_ACTOR: ${{ github.triggering_actor }}
        run: |
          set -euo pipefail
          mkdir -p evidence
          printf 'run_id=%s\nattempt=%s\nsource_sha=%s\nactor=%s\ntriggering_actor=%s\nrunner=%s\n' \
            "$RUN_ID" "$ATTEMPT" "$SOURCE_SHA" "$ACTOR" "$TRIGGERING_ACTOR" "$RUNNER_NAME" \
            > "evidence/run-$RUN_ID-attempt-$ATTEMPT.env"
          git -C "$STATE_DIR" log -3 --oneline > evidence/state-branch-log.txt || true
          git -C "$STATE_DIR" status --short > evidence/state-worktree-status.txt || true

      - name: Upload immutable attempt evidence
        if: ${{ always() }}
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
        with:
          name: incident-${{ github.run_id }}-${{ github.run_attempt }}
          path: evidence/
          retention-days: 7
          if-no-files-found: error

6. Predict state changes before run 1

Write at least two of these predictions in your lab notes before dispatching. The exercise is about proving state transitions, not only making the YAML green.

7. Dispatch attempt 1 and capture the deliberate failure

# REST 2026-03-10 returns the exact created workflow_run_id.
REF=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name')
RUN_ID=$(gh api -X POST   -H 'Accept: application/vnd.github+json'   -H 'X-GitHub-Api-Version: 2026-03-10'   'repos/{owner}/{repo}/actions/workflows/ch33-incident-drill.yml/dispatches'   -f ref="$REF"   -F return_run_details=true   -f 'inputs[recovery_mode]=reconcile'   -F 'inputs[inject_fault_once]=true'   --jq '.workflow_run_id')
printf 'exact run id=%s
' "$RUN_ID"

gh run watch "$RUN_ID" --exit-status || true
gh run view "$RUN_ID" --json databaseId,attempt,headSha,status,conclusion,url

Expected result: guarded-fake-deployment fails at “Inject one failure…” after the state branch commit has succeeded. The run is red, but the current pointer already references the new digest. That discrepancy is intentional and is the incident you must recover.

8. Mandatory evidence preservation before rerun

mkdir -p incident-evidence

gh run view "$RUN_ID" --attempt 1 --json   databaseId,attempt,headSha,event,status,conclusion,url,jobs   > "incident-evidence/run-$RUN_ID-attempt-1.json"

gh run view "$RUN_ID" --attempt 1 --log   > "incident-evidence/run-$RUN_ID-attempt-1.log"

# Read exact remote target state after the failure.
git fetch origin incident-state
git show origin/incident-state:state/current.env   > "incident-evidence/target-after-attempt-1.env"
git ls-tree -r --name-only origin/incident-state state   > "incident-evidence/state-tree-after-attempt-1.txt"

# Optional attempt-specific REST log request (follows redirect with -L when using curl).
gh api \
  -H 'Accept: application/vnd.github+json' \
  -H 'X-GitHub-Api-Version: 2026-03-10' \
  repos/{owner}/{repo}/actions/runs/$RUN_ID \
  > "incident-evidence/run-resource-before-rerun.json"

Do not rerun until these files exist and you have inspected the state branch. Your preserved evidence should show the side effect completed before the red conclusion.

9. Classify the incident

10. Rerun failed work using exact run identity

# Same RUN_ID; GitHub creates attempt 2.
gh run rerun "$RUN_ID" --failed

gh run watch "$RUN_ID" --exit-status
gh run view "$RUN_ID" --json databaseId,attempt,headSha,status,conclusion,url

# Compare both attempts explicitly.
gh run view "$RUN_ID" --attempt 1 --json attempt,headSha,conclusion,url
gh run view "$RUN_ID" --attempt 2 --json attempt,headSha,conclusion,url

Expected result for recovery_mode=reconcile: attempt 2 has the same run ID/SHA but attempt=2, finds the existing receipt owned by the same run ID, does not create another release, marks the receipt healthy and verifies the current digest.

11. Independently verify no duplicate side effect

git fetch origin incident-state

git show origin/incident-state:state/current.env
# Count digest-addressed releases and logical receipts.
git ls-tree -r --name-only origin/incident-state state/releases | wc -l
git ls-tree -r --name-only origin/incident-state state/receipts | wc -l

git log --oneline --decorate -5 origin/incident-state

For a brand-new lab there should be one new digest-addressed release and one logical receipt for the exercise. The state branch history should show an activation commit from attempt 1 and a reconcile commit from attempt 2. If counts or ownership do not match your predictions, stop and investigate rather than cleaning up immediately.

12. Optional rollback variant

Repeat the drill from a clean state branch with recovery_mode=rollback. Attempt 1 still activates and fails. Attempt 2 verifies that the current target is still the failed digest, restores the exact previous digest recorded in the receipt, marks the receipt rolled_back, and verifies the pointer. The guard refuses rollback if another actor changed current state after the failure.

# Use a newly reset/reseeded disposable state branch, then:
gh workflow run ch33-incident-drill.yml   -f recovery_mode=rollback   -f inject_fault_once=true
# Preserve attempt 1 again before gh run rerun RUN_ID --failed.

13. Prove the difference between rerun and a corrected new run

Now make a harmless source change on main, commit it, and dispatch again with fault injection disabled. This creates a new run ID and new source SHA, so the deterministic artifact digest/idempotency key also changes. Record the linkage: the new run is a follow-up to the incident, not attempt 3 of the old run.

printf '
follow-up marker
' >> README.md
git add README.md
git commit -m 'lab: follow-up source revision'
git push

REF=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name')
NEW_RUN_ID=$(gh api -X POST   -H 'Accept: application/vnd.github+json'   -H 'X-GitHub-Api-Version: 2026-03-10'   'repos/{owner}/{repo}/actions/workflows/ch33-incident-drill.yml/dispatches'   -f ref="$REF"   -F return_run_details=true   -f 'inputs[recovery_mode]=reconcile'   -F 'inputs[inject_fault_once]=false'   --jq '.workflow_run_id')
printf 'new run id=%s
' "$NEW_RUN_ID"

14. Required evidence packet

• Event/source: workflow dispatch inputs, original source SHA, workflow path/revision.
• Run identity: run ID, run number, attempts 1 and 2, actor and triggering actor where visible.
• Job/runner: job database ID, runner label/name/image/tool information from logs.
• Side-effect identity: repository ID, environment, artifact digest, idempotency key, receipt path.
• External target: state-branch before/after current pointer and commit history.
• First failure: attempt-1 logs plus injected exit code 23 and failed step name.
• Recovery: exact rerun command/scope; attempt-2 result; reconcile or rollback receipt.
• Artifacts: attempt-qualified evidence artifact names/IDs/digests if retained.
• Limitations: state branch is a faithful fake target, not a cloud deployment; real APIs need provider-specific idempotency/locking.
• Cleanup record: when the lab repository/state branch was deleted after evidence review.

15. Cleanup and rollback of the lab itself

After reviewing and exporting the evidence packet, delete only the disposable repository or exact incident-state branch you created. Do not use a generic command that deletes “latest” run/artifact/branch. If you keep the repository for study, remove the workflow's write permission by deleting/renaming the lab workflow and mark the state branch as archived.

# Exact branch cleanup example in the disposable lab only:
git push origin --delete incident-state
# Or delete the entire throwaway repository through your normal deliberate process.

16. What Chapter 33 adds — and the bridge to Chapter 34

Chapter 33 adds an incident-safe operating model: reruns are attempts of the same event/SHA, side effects have idempotency/receipt identity, first-failure evidence is preserved, rollback is guarded against exact current state, and recovery is reconciled independently of the green check. Chapter 34 moves outward from workflow design to enterprise governance: allowed actions, runner policy, organization controls and auditability that make these recovery guarantees enforceable at scale.

Knowledge check

Official references and version notes

• Re-running workflows and jobs — Current rerun window, attempt behavior, actor privileges, same ref/SHA semantics and targeted rerun options.
• Variables reference — Definitions of GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, GITHUB_RUN_NUMBER, GITHUB_ACTOR and GITHUB_TRIGGERING_ACTOR.
• Contexts reference — Current github context fields and rerun identity semantics.
• REST API: workflow runs — Attempt-specific logs, rerun/cancel endpoints and exact workflow-run resource state.
• GitHub CLI: gh run rerun — Current --failed, --job and --debug rerun controls; job reruns require the database ID.
• GitHub CLI: gh run view — Current --attempt, --log, --log-failed and JSON run/job evidence inspection.
• Concurrency — Current serialization, cancel-in-progress and queueing behavior.
• Workflow syntax: concurrency — Current workflow/job concurrency syntax including queue:max.
• Deployment environments — Environment gates, secret timing and deployment target separation.
• Deployments and environments — Protection rules and deployment governance boundaries.
• actions/checkout v7.0.1 — Full commit SHA used by executable examples.
• actions/upload-artifact v7.0.1 — Full commit SHA used for bounded attempt-specific evidence artifacts.
• upload-artifact behavior — Current immutable-artifact behavior, unique names, outputs and overwrite semantics.