Chapter 29Lesson 05~280 minutes

Checkpoint Lab — Infrastructure-as-Code Pipelines, Terraform/OpenTofu Workflows, Plan Reviews, State Safety, and Drift Controls

Implement a disposable plan/apply operating model with state isolation and serialized mutation, prove the applied change matches the reviewed plan digest and source SHA, inject drift, and remove exactly the created resources.

CheckpointPlan reviewSerializationEvidenceCleanup

Learning objectives

Checkpoint objectives

  • Implement a disposable plan/apply workflow with exact source/tool/provider/plan identity.
  • Isolate the state and serialize the mutation domain.
  • Predict and verify state changes before execution.
  • Inject drift and prove detection without blind repair.
  • Produce an evidence packet that states what is proved and what remains outside the lab.

1. Checkpoint scenario

You own a disposable IaC repository glci-ch29-checkpoint. A plan proposes creating one local managed file. You must prove that the applied bytes correspond to the reviewed saved plan, not to a later re-plan. Then you will change the file outside OpenTofu, prove drift, and remove the resource using an independently reviewed destroy plan.

The GitLab mapping uses a resource_group named ch29-checkpoint-state. The mandatory execution can be performed locally; a real GitLab project may additionally use GitLab-managed state and authentic pipeline/job IDs.

2. Tool and platform assumptions

Item Checkpoint value
OpenTofu 1.12.6 standalone local lab
Provider hashicorp/local 2.9.0
GitLab optional component 4.8.1 with supported OpenTofu 1.12.5
State Local isolated directory for mandatory lab; GitLab-managed state optional Free path
Credentials None for local lab; never introduce real cloud credentials
Mutation target Only sandbox/checkpoint.txt inside the lab

3. Preflight and disposable-resource guard

LAB=glci-ch29-checkpoint
mkdir -p "$LAB" && cd "$LAB"
test "$(basename "$PWD")" = 'glci-ch29-checkpoint'
mkdir -p evidence sandbox
git init
printf '.terraform/\n*.tfstate\n*.tfstate.*\n*.cache\nsandbox/checkpoint.txt\n' > .gitignore
tofu version
printf 'target=%s\n' "$PWD/sandbox/checkpoint.txt" | tee evidence/target.txt

Stop if the path guard fails. The checkpoint must never be pointed at a production directory, remote cloud account or existing shared state.

4. Exact checkpoint configuration

terraform {
  required_version = "= 1.12.6"
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "= 2.9.0"
    }
  }
}

variable "message" {
  type = string
}

resource "local_file" "checkpoint" {
  filename        = "${path.module}/sandbox/checkpoint.txt"
  content         = var.message
  file_permission = "0644"
}

output "checkpoint_file" {
  value = local_file.checkpoint.filename
}

Save as main.tf, then initialize and commit the source plus provider lock file.

tofu fmt -check -diff
tofu init -input=false
tofu validate
git add main.tf .terraform.lock.hcl .gitignore
git commit -m 'ch29 checkpoint source'
git rev-parse HEAD | tee evidence/source-sha.txt
sha256sum .terraform.lock.hcl | tee evidence/provider-lock.sha256
tofu version -json > evidence/tofu-version.json

5. Predict before running the plan

Write predictions before mutation:

Prediction A: the plan will propose exactly one local_file.checkpoint create.
Prediction B: no sandbox/checkpoint.txt file exists before apply.
Prediction C: applying the reviewed plan will create the file with reviewed-checkpoint-v1.
Prediction D: out-of-band file modification will make detailed-exitcode return 2.
Prediction E: the destroy plan will remove only local_file.checkpoint.

Save these to evidence/predictions.txt. The point is to compare expected versus observed state rather than retroactively explaining whatever happened.

6. Create the review candidate

test ! -e sandbox/checkpoint.txt
tofu plan -input=false -out=checkpoint.cache   -var='message=reviewed-checkpoint-v1' -no-color   | tee evidence/plan-human.txt
sha256sum checkpoint.cache | tee evidence/plan.sha256
printf 'source_sha=%s\n' "$(cat evidence/source-sha.txt)" > evidence/plan-metadata.txt
printf 'provider_lock_sha256=%s\n' "$(cut -d' ' -f1 evidence/provider-lock.sha256)" >> evidence/plan-metadata.txt
printf 'plan_sha256=%s\n' "$(cut -d' ' -f1 evidence/plan.sha256)" >> evidence/plan-metadata.txt
printf 'state=local:terraform.tfstate\n' >> evidence/plan-metadata.txt

Verify the plan text contains the expected resource address and no unrelated resource. The binary plan stays local; the digest is the approval anchor.

7. Simulated review and authorization

cp evidence/plan-metadata.txt evidence/review.txt
cat >> evidence/review.txt <<'EOF'
review_status=approved
reviewer=fake-platform-reviewer@example.test
apply_scope=local:sandbox/checkpoint.txt
EOF
cat evidence/review.txt

In real GitLab, replace the synthetic reviewer line with authentic manual-job/approval/audit evidence. Do not use a mutable CI variable as the only proof of approval.

8. Serialization model

For a GitLab run, the apply job must use the same resource-group key as any other job that can mutate this state:

apply_checkpoint:
  stage: apply
  needs: [plan_checkpoint]
  resource_group: ch29-checkpoint-state
  when: manual
  allow_failure: false
  script:
    - sha256sum -c plan.sha256
    - grep -Fx "source_sha=$CI_COMMIT_SHA" plan-metadata.txt
    - tofu init -input=false -lockfile=readonly
    - tofu apply -input=false -auto-approve checkpoint.cache

The local execution is sequential by design; the GitLab mapping adds queue-level serialization while the backend must still provide locking.

9. Deliberate plan-substitution test

Before the real apply, make a copy and prove the digest guard detects tampering without destroying the original:

cp checkpoint.cache tampered.cache
printf 'not-a-plan\n' >> tampered.cache
cp evidence/plan.sha256 evidence/tampered.sha256
sed 's/checkpoint.cache/tampered.cache/' evidence/tampered.sha256 > evidence/tampered-check.sha256
set +e
sha256sum -c evidence/tampered-check.sha256 > evidence/tamper-result.txt 2>&1
rc=$?
set -e
test "$rc" -ne 0
printf 'tamper_guard=blocked\n' >> evidence/tamper-result.txt
rm -f tampered.cache evidence/tampered.sha256 evidence/tampered-check.sha256

The expected result is a checksum failure. The original checkpoint.cache and plan.sha256 remain untouched.

10. Apply the exact reviewed plan

test -z "$(git status --porcelain --untracked-files=no)"
sha256sum -c evidence/plan.sha256
grep -Fx "source_sha=$(git rev-parse HEAD)" evidence/review.txt
sha256sum .terraform.lock.hcl | tee evidence/provider-lock-before-apply.sha256
cmp evidence/provider-lock.sha256 evidence/provider-lock-before-apply.sha256
tofu init -input=false -lockfile=readonly
tofu apply -input=false -auto-approve checkpoint.cache   | tee evidence/apply.txt

If any guard fails, the checkpoint fails. Do not regenerate the plan under the same review record.

11. Verify state and external result independently

tofu state list | tee evidence/state-list.txt
test "$(cat sandbox/checkpoint.txt)" = 'reviewed-checkpoint-v1'
sha256sum sandbox/checkpoint.txt | tee evidence/external-file.sha256
tofu plan -input=false -detailed-exitcode   -var='message=reviewed-checkpoint-v1' -no-color   > evidence/post-apply-plan.txt
rc=$?
test "$rc" -eq 0
printf 'post_apply_match=yes\n' | tee evidence/post-apply-result.txt

This verifies IaC state address, external file content and a no-change refresh plan. These are distinct checks.

12. Inject and detect controlled drift

printf 'emergency-out-of-band-change\n' > sandbox/checkpoint.txt
sha256sum sandbox/checkpoint.txt | tee evidence/drifted-external.sha256
set +e
tofu plan -input=false -detailed-exitcode   -var='message=reviewed-checkpoint-v1' -no-color   > evidence/drift-plan.txt
rc=$?
set -e
printf 'drift_exit_code=%s\n' "$rc" | tee evidence/drift-result.txt
test "$rc" -eq 2

Checkpoint rule: stop here and review the drift evidence. Do not auto-apply from the drift detector.

13. Reconcile the disposable drift under explicit intent

Because this is a synthetic lab, you may authorize reconciliation. Generate a new plan and a new digest; do not reuse the original review record:

tofu plan -input=false -out=reconcile.cache   -var='message=reviewed-checkpoint-v1' -no-color   | tee evidence/reconcile-plan.txt
sha256sum reconcile.cache | tee evidence/reconcile.sha256
printf 'reconcile_review=approved-for-disposable-lab\n' > evidence/reconcile-review.txt
sha256sum -c evidence/reconcile.sha256
tofu apply -input=false -auto-approve reconcile.cache   | tee evidence/reconcile-apply.txt
test "$(cat sandbox/checkpoint.txt)" = 'reviewed-checkpoint-v1

14. Cleanup is another reviewed mutation

tofu plan -destroy -input=false -out=destroy.cache   -var='message=reviewed-checkpoint-v1' -no-color   | tee evidence/destroy-plan.txt
sha256sum destroy.cache | tee evidence/destroy.sha256
printf 'destroy_review=approved-for-exact-lab-target\n' > evidence/destroy-review.txt
sha256sum -c evidence/destroy.sha256
tofu apply -input=false -auto-approve destroy.cache   | tee evidence/destroy-apply.txt
test ! -e sandbox/checkpoint.txt
tofu state list | tee evidence/state-after-destroy.txt
test ! -s evidence/state-after-destroy.txt
printf 'cleanup=verified\n' | tee evidence/cleanup.txt

15. Required evidence packet

Keep the following non-secret evidence:

  • Pipeline source/ref/SHA and pipeline/job IDs when run in GitLab; local source SHA otherwise.
  • OpenTofu version and provider-lock digest.
  • Plan digest and review record.
  • State/backend identity (not raw state contents).
  • Mutation serialization key/queue evidence for GitLab runs.
  • Apply trace plus external file digest/content assertion.
  • Post-apply no-change plan result.
  • Drift exit code and drift plan.
  • Reconciliation and destroy plan digests/review records.
  • Cleanup verification and assumptions/limitations note.

Do not include terraform.tfstate, provider cache, access tokens or a raw plan containing real secrets in the shareable evidence packet.

16. Optional authentic GitLab execution

If you run the checkpoint in a throwaway GitLab project, capture:

printf 'source=%s ref=%s sha=%s pipeline=%s job=%s\n'   "$CI_PIPELINE_SOURCE" "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA"   "$CI_PIPELINE_ID" "$CI_JOB_ID"
tofu version
sha256sum .terraform.lock.hcl

Use GitLab-managed state or another disposable remote backend rather than transferring local state between ephemeral runners. Preserve real pipeline/job IDs and resource-group waiting evidence. If you use the OpenTofu component, pin a released version and only request an OpenTofu version that release actually supports.

17. Verification checklist

  • The repository/source SHA is recorded and provider lock is committed.
  • The plan is saved, hashed and reviewed before apply.
  • The tamper test proves checksum substitution is blocked.
  • The apply consumes the saved plan and does not run init -upgrade or an implicit re-plan.
  • The mutation domain is serialized in the GitLab mapping.
  • Post-apply external state and no-change plan both match.
  • Drift returns detailed exit code 2 and does not auto-apply.
  • Reconciliation has a new plan/review identity.
  • Destroy has its own plan digest/review and removes only the lab resource.
  • No raw state, real credential or production target appears in the evidence packet.

18. What this checkpoint proves—and does not prove

It proves plan/apply identity, provider-lock discipline, local state isolation, explicit review metadata, serialization design, external drift detection and reviewed cleanup. It does not prove a real cloud provider role is least-privileged, a GitLab-managed state backend is configured, production state recovery is tested, or organization approvals/policies are correct. Those require authorized environment-specific evidence.

Knowledge check

Why does reconciliation get a new plan digest instead of reusing the original approved plan?

What does the tamper test prove?

Why does the checkpoint inspect both tofu state list and the external file?

If two GitLab pipelines target different state names, must they always share one resource group?

What production control from this chapter most directly prevents “reviewed plan A, applied plan B”?

19. What Chapter 29 adds to the production operating model

You can now treat IaC as a controlled state transition rather than a shell command: exact source and provider identity, protected backend/state, reviewable saved plan, explicit authorization, serialized exact apply, external verification, drift evidence and reviewed recovery/cleanup.

Next chapter

Runner Fleets, Autoscaling, Docker Machine Migration, Kubernetes Runners, Ephemeral Workers, and Capacity Planning

Chapter 30 moves from infrastructure state to the compute fleet that executes pipelines: ephemeral runners, autoscaling capacity, executor isolation, Docker Machine migration, Kubernetes runner design and queue/cost evidence.

Version and compatibility note

GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.

Official references and version notes

Documentation verification date: 2026-09-12. GitLab-managed OpenTofu/Terraform state and merge-request plan integration are available on Free, Premium, and Ultimate. GitLab’s legacy Terraform CI/CD templates were removed in GitLab 18.0; current guidance is the versioned OpenTofu CI/CD component. GitLab-managed OpenTofu/Terraform state support in the current CLI flow was introduced in GitLab 18.3 and requires glab 1.66+. Plan files are not encrypted by GitLab or OpenTofu and can contain credentials or sensitive values; restrict artifact access and never treat plan output as safe by default. The current released OpenTofu component used for optional examples is 4.8.1; that release supports OpenTofu 1.12.5 and publishes signed rootless images. The standalone local lab pins OpenTofu 1.12.6 (latest stable on the verification date) and hashicorp/local 2.9.0. For a real GitLab-managed state checkpoint, add state name/version/lock evidence, authentic actor/role evidence and backend recovery testing without copying sensitive state into logs.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.