Chapter 27Lesson 05~240 minutes

Checkpoint Lab — Infrastructure as Code, Terraform Plans, Policy Checks, and Deployment Workflows

Prove a disposable plan→policy→approval→apply pipeline that binds source, provider lock, plan digest and state identity and rejects one stale plan.

CheckpointPlan digestEnvironment gateStale rejectionEvidence

Learning objectives

  • Build a complete disposable plan→policy→environment→apply pipeline with explicit identity predictions.
  • Prove source SHA, provider lock and saved-plan digest before mutation.
  • Run a normal path and a stale-state path and preserve both outcomes independently.
  • Produce a compact evidence packet without exposing real cloud credentials, state or plan secrets.
  • Explain how the same contract maps to a remote backend and OIDC-bound cloud role.

1. Checkpoint scenario: one approved plan, two state histories

The checkpoint uses the same synthetic local-file fixture. Run it once in normal mode and once in stale mode. Both runs produce an exact saved plan and policy result. The normal run applies that reviewed plan. The stale run deliberately advances local state before attempting the saved plan, and success means Terraform rejects the obsolete plan while the workflow preserves the mismatch evidence.

2. Preflight and safety boundary

  • Use a disposable repository or branch; no customer or production state may be referenced.
  • Pin Terraform to 1.16.1 and setup-terraform v4.0.1 commit dfe3c3f87815947d99a8997f908cb6525fc44e9e.
  • Pin hashicorp/local to 2.9.0, run terraform init locally once, and commit .terraform.lock.hcl.
  • Create/inspect the iac-lab environment. If reviewer protection is unavailable for the chosen repository visibility/plan, record that the gate is a simulation rather than silently claiming approval occurred.
  • The lab never requests id-token: write, package/release write, cloud credentials or a self-hosted runner.
  • The only apply target is a runner-local file under infra/out/.

Do not replace the local backend with a production backend for this exercise. A stale-plan experiment against shared infrastructure is disruptive and unnecessary.

3. Predict important state changes before running

Prediction Normal mode Stale mode Independent proof
Source/tool identity same source SHA + Terraform/provider lock same checkout HEAD, version, lock hash
Plan identity one plan digest survives plan→apply boundary same SHA-256 before apply
Policy allow allow policy.txt tied to plan artifact
State before saved apply no competing local state transition intervening state exists state snapshot + marker content
Saved apply succeeds fails as stale/different-state context captured apply.log/exit code
Final marker checkpoint-approved intervening-state remains read file independently

4. Exact checkpoint workflow

name: IaC checkpoint — exact plan identity
on:
  workflow_dispatch:
    inputs:
      mode:
        description: "normal or stale"
        type: choice
        options: [normal, stale]
        required: true
        default: normal

permissions: {}

jobs:
  plan:
    runs-on: ubuntu-24.04
    permissions:
      contents: read
    outputs:
      plan_sha256: ${{ steps.ids.outputs.plan_sha256 }}
      lock_sha256: ${{ steps.ids.outputs.lock_sha256 }}
    defaults:
      run:
        working-directory: infra
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
        with:
          terraform_version: "1.16.1"
          terraform_wrapper: false
      - name: Preflight exact source/tool/provider lock
        shell: bash
        run: |
          set -euo pipefail
          test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
          terraform version | tee terraform-version.txt
          terraform fmt -check -diff
          terraform init -input=false -lockfile=readonly
          terraform validate
      - name: Produce the reviewed plan and bounded policy result
        shell: bash
        run: |
          set -euo pipefail
          terraform plan -input=false -out=tfplan -var 'marker_text=checkpoint-approved'
          terraform show -no-color tfplan > plan.txt
          terraform show -json tfplan > plan.json
          python policy.py | tee policy.txt
      - id: ids
        shell: bash
        run: |
          set -euo pipefail
          p=$(sha256sum tfplan | awk '{print $1}')
          l=$(sha256sum .terraform.lock.hcl | awk '{print $1}')
          printf 'run=%s
attempt=%s
source_sha=%s
plan_sha256=%s
lock_sha256=%s
'             "$GITHUB_RUN_ID" "$GITHUB_RUN_ATTEMPT" "$GITHUB_SHA" "$p" "$l" > identity.env
          echo "plan_sha256=$p" >> "$GITHUB_OUTPUT"
          echo "lock_sha256=$l" >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        if: ${{ always() }}
        with:
          name: checkpoint-plan-${{ github.run_id }}-${{ github.run_attempt }}
          path: |
            infra/tfplan
            infra/plan.txt
            infra/plan.json
            infra/policy.txt
            infra/identity.env
            infra/terraform-version.txt
            infra/.terraform.lock.hcl
          retention-days: 1

  apply:
    needs: plan
    runs-on: ubuntu-24.04
    environment: iac-lab
    concurrency:
      group: iac-checkpoint-local
      cancel-in-progress: false
    permissions:
      contents: read
    defaults:
      run:
        working-directory: infra
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
        with:
          terraform_version: "1.16.1"
          terraform_wrapper: false
      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
        with:
          name: checkpoint-plan-${{ github.run_id }}-${{ github.run_attempt }}
          path: infra/evidence
      - name: Assert reviewed identities before target access
        shell: bash
        run: |
          set -euo pipefail
          test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
          cp evidence/tfplan ./tfplan
          test "$(sha256sum tfplan | awk '{print $1}')" = "${{ needs.plan.outputs.plan_sha256 }}"
          test "$(sha256sum .terraform.lock.hcl | awk '{print $1}')" = "${{ needs.plan.outputs.lock_sha256 }}"
          terraform init -input=false -lockfile=readonly
      - name: Intentionally advance state in stale mode
        if: ${{ inputs.mode == 'stale' }}
        shell: bash
        run: |
          set -euo pipefail
          terraform apply -input=false -auto-approve -var 'marker_text=intervening-state'
          terraform state pull > first-failure-precondition-state.json
      - id: apply_saved
        name: Apply exact saved plan (expected to fail in stale mode)
        shell: bash
        run: |
          set +e
          terraform apply -input=false tfplan > apply.log 2>&1
          rc=$?
          cat apply.log
          echo "rc=$rc" >> "$GITHUB_OUTPUT"
          if [ "${{ inputs.mode }}" = stale ]; then
            test "$rc" -ne 0
          else
            test "$rc" -eq 0
          fi
      - name: Preserve state and target evidence before any repair
        if: ${{ always() }}
        shell: bash
        run: |
          set -euo pipefail
          mkdir -p checkpoint-evidence
          cp apply.log checkpoint-evidence/
          cp evidence/identity.env checkpoint-evidence/
          cp evidence/policy.txt checkpoint-evidence/
          terraform state pull > checkpoint-evidence/state-after-attempt.json 2>/dev/null || true
          if [ -f out/marker.txt ]; then cp out/marker.txt checkpoint-evidence/marker-after-attempt.txt; fi
          printf 'mode=%s
apply_rc=%s
' "${{ inputs.mode }}" "${{ steps.apply_saved.outputs.rc }}" > checkpoint-evidence/result.env
      - name: Verify normal target or stale rejection
        shell: bash
        run: |
          set -euo pipefail
          if [ "${{ inputs.mode }}" = normal ]; then
            test "$(cat out/marker.txt)" = checkpoint-approved
          else
            test "$(cat out/marker.txt)" = intervening-state
            grep -Ei 'stale|state.*changed|different state' apply.log >/dev/null
          fi
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        if: ${{ always() }}
        with:
          name: checkpoint-result-${{ github.run_id }}-${{ github.run_attempt }}
          path: infra/checkpoint-evidence/
          retention-days: 7

The stale path deliberately captures a non-zero Terraform exit without allowing the workflow to continue silently. The step asserts that failure is expected only in stale mode, then the following evidence step runs with always() to preserve non-privileged diagnostics. No repair is performed because the lesson is to retain the first failure and prove the approved plan became obsolete.

5. Normal run: prove reviewed bytes became target state

  1. Dispatch mode=normal.
  2. Record the plan job run/attempt/source and plan/lock hashes.
  3. After the environment boundary, confirm the apply job recomputes both hashes before init/apply.
  4. Verify out/marker.txt contains checkpoint-approved.
  5. Download the result evidence and map the Terraform state hash to the same run.

6. Stale run: preserve the safety rejection

  1. Dispatch mode=stale.
  2. The plan job creates a new reviewed plan for checkpoint-approved.
  3. The apply job intentionally creates intervening-state first, advancing local state.
  4. The exact saved plan is attempted and must return non-zero because its state context is obsolete.
  5. Before any cleanup, preserve apply.log, state and current marker content.
  6. Do not regenerate or auto-approve a replacement in the same step; a fresh plan would be a new review object.

7. Evidence packet

Field Required evidence
event/revision workflow_dispatch, source SHA, workflow path/revision, run ID/attempt
toolchain ubuntu-24.04, setup-terraform SHA, Terraform 1.16.1
provider dependency .terraform.lock.hcl hash + local provider 2.9.0 constraint
backend/workspace local backend/default workspace; limitation note that it is runner-ephemeral
plan binary plan SHA-256, human plan.txt, synthetic plan.json
policy policy checker source/version + POLICY=allow output
authorization iac-lab environment state/protection or explicit simulation note
apply exit code/log and normal or stale interpretation
target state snapshot/hash and marker content
limitations no cloud OIDC, no remote locking, no production secret/remote resource

8. Optional policy-violation variant

To test policy denial without touching target state, temporarily change the fixture so the planned filename is outside infra/out/marker.txt or propose a destroy. The policy step should exit before the apply job becomes eligible. Preserve the denied plan digest and policy reason, then restore the safe fixture. This is a review failure, not an infrastructure rollback event because no apply occurred.

9. Map the checkpoint to production without changing invariants

Lab field Production analogue
local backend/default workspace remote backend key/workspace with access policy and locking
no credential OIDC-federated plan/read role and separate apply/mutation role
local provider file cloud/Kubernetes/provider resources
marker content provider resource revision/health
GitHub environment protected production environment/approval rule
stale local state remote state serial/lineage advanced by another apply
result artifact protected audit record + provider-side change/deployment ID

10. Recovery rule: a new state requires a new reviewed plan

After the stale-mode run, the safest next action is not to force the old plan through. Inspect why state advanced. If the intervening change is legitimate, generate a new plan from the current state, rerun policy, obtain the required approval, and then apply the new plan. If the state change is unauthorized, incident response may take precedence over deployment.

11. Verification checklist

  • Exactly one source SHA is recorded per run.
  • Terraform/setup/provider versions are pinned and the lock file is hashed.
  • Plan hash is recomputed before apply.
  • Policy output is tied to the same plan evidence.
  • Environment state is recorded separately from Terraform state.
  • Normal mode proves the intended local target.
  • Stale mode preserves non-zero apply and unchanged intervening target state.
  • No cloud secret, real backend or production resource is used.

12. What Chapter 27 adds — and the bridge to Chapter 28

Chapter 27 adds state-aware change governance to the production Actions model. Source SHA and dependency lock identify IaC intent; plan digest and policy identify a proposed transition; environment/OIDC bound mutation authority; backend/state identity can still veto an obsolete plan; and post-apply evidence proves what changed. Chapter 28 will reuse these exact-revision ideas for monorepo path filters and changed-file detection, where the risk shifts from stale state to incomplete change selection.

13. Checkpoint summary

You now have an IaC pipeline that treats a stale plan as evidence rather than inconvenience. The same control structure scales from one local file to cloud infrastructure when backend, identity, locking and external verification are substituted explicitly.

Next lesson

Monorepos, Path Filters, Changed-File Detection, and Selective Pipelines: Core Concepts and Mental Model

Continue with the next lesson to build on the current concepts, evidence, security boundaries, and operational practices.

Knowledge check

Why does stale mode intentionally leave intervening-state in place?

What must happen before a replacement plan can be applied after state legitimately changes?

Why are plan SHA and lock SHA both required?

If the environment gate is unavailable on the chosen plan/visibility, what should the evidence say?

What is the Chapter 28 handoff?

Official references and version notes

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0Send only Ethereum/ERC-20 compatible assets to this address.