Checkpoint Lab — Infrastructure-as-Code Pipelines, Terraform/OpenTofu Workflows, Plan Reviews, State Safety, and Drift Controls
Implement a disposable plan/apply operating model with state isolation and serialized mutation, prove the applied change matches the reviewed plan digest and source SHA, inject drift, and remove exactly the created resources.
Learning objectives
Checkpoint objectives
- Implement a disposable plan/apply workflow with exact source/tool/provider/plan identity.
- Isolate the state and serialize the mutation domain.
- Predict and verify state changes before execution.
- Inject drift and prove detection without blind repair.
- Produce an evidence packet that states what is proved and what remains outside the lab.
1. Checkpoint scenario
You own a disposable IaC repository
glci-ch29-checkpoint. A plan proposes creating one
local managed file. You must prove that the applied bytes correspond
to the reviewed saved plan, not to a later re-plan. Then you will
change the file outside OpenTofu, prove drift, and remove the
resource using an independently reviewed destroy plan.
The GitLab mapping uses a resource_group named
ch29-checkpoint-state. The mandatory execution can be
performed locally; a real GitLab project may additionally use
GitLab-managed state and authentic pipeline/job IDs.
2. Tool and platform assumptions
| Item | Checkpoint value |
|---|---|
| OpenTofu | 1.12.6 standalone local lab |
| Provider | hashicorp/local 2.9.0 |
| GitLab optional component | 4.8.1 with supported OpenTofu 1.12.5 |
| State | Local isolated directory for mandatory lab; GitLab-managed state optional Free path |
| Credentials | None for local lab; never introduce real cloud credentials |
| Mutation target |
Only sandbox/checkpoint.txt inside the lab
|
3. Preflight and disposable-resource guard
LAB=glci-ch29-checkpoint
mkdir -p "$LAB" && cd "$LAB"
test "$(basename "$PWD")" = 'glci-ch29-checkpoint'
mkdir -p evidence sandbox
git init
printf '.terraform/\n*.tfstate\n*.tfstate.*\n*.cache\nsandbox/checkpoint.txt\n' > .gitignore
tofu version
printf 'target=%s\n' "$PWD/sandbox/checkpoint.txt" | tee evidence/target.txt
Stop if the path guard fails. The checkpoint must never be pointed at a production directory, remote cloud account or existing shared state.
4. Exact checkpoint configuration
terraform {
required_version = "= 1.12.6"
required_providers {
local = {
source = "hashicorp/local"
version = "= 2.9.0"
}
}
}
variable "message" {
type = string
}
resource "local_file" "checkpoint" {
filename = "${path.module}/sandbox/checkpoint.txt"
content = var.message
file_permission = "0644"
}
output "checkpoint_file" {
value = local_file.checkpoint.filename
}
Save as main.tf, then initialize and commit the source
plus provider lock file.
tofu fmt -check -diff
tofu init -input=false
tofu validate
git add main.tf .terraform.lock.hcl .gitignore
git commit -m 'ch29 checkpoint source'
git rev-parse HEAD | tee evidence/source-sha.txt
sha256sum .terraform.lock.hcl | tee evidence/provider-lock.sha256
tofu version -json > evidence/tofu-version.json
5. Predict before running the plan
Write predictions before mutation:
Prediction A: the plan will propose exactly one local_file.checkpoint create.
Prediction B: no sandbox/checkpoint.txt file exists before apply.
Prediction C: applying the reviewed plan will create the file with reviewed-checkpoint-v1.
Prediction D: out-of-band file modification will make detailed-exitcode return 2.
Prediction E: the destroy plan will remove only local_file.checkpoint.
Save these to evidence/predictions.txt. The point is to
compare expected versus observed state rather than retroactively
explaining whatever happened.
6. Create the review candidate
test ! -e sandbox/checkpoint.txt
tofu plan -input=false -out=checkpoint.cache -var='message=reviewed-checkpoint-v1' -no-color | tee evidence/plan-human.txt
sha256sum checkpoint.cache | tee evidence/plan.sha256
printf 'source_sha=%s\n' "$(cat evidence/source-sha.txt)" > evidence/plan-metadata.txt
printf 'provider_lock_sha256=%s\n' "$(cut -d' ' -f1 evidence/provider-lock.sha256)" >> evidence/plan-metadata.txt
printf 'plan_sha256=%s\n' "$(cut -d' ' -f1 evidence/plan.sha256)" >> evidence/plan-metadata.txt
printf 'state=local:terraform.tfstate\n' >> evidence/plan-metadata.txt
Verify the plan text contains the expected resource address and no unrelated resource. The binary plan stays local; the digest is the approval anchor.
7. Simulated review and authorization
cp evidence/plan-metadata.txt evidence/review.txt
cat >> evidence/review.txt <<'EOF'
review_status=approved
reviewer=fake-platform-reviewer@example.test
apply_scope=local:sandbox/checkpoint.txt
EOF
cat evidence/review.txt
In real GitLab, replace the synthetic reviewer line with authentic manual-job/approval/audit evidence. Do not use a mutable CI variable as the only proof of approval.
8. Serialization model
For a GitLab run, the apply job must use the same resource-group key as any other job that can mutate this state:
apply_checkpoint:
stage: apply
needs: [plan_checkpoint]
resource_group: ch29-checkpoint-state
when: manual
allow_failure: false
script:
- sha256sum -c plan.sha256
- grep -Fx "source_sha=$CI_COMMIT_SHA" plan-metadata.txt
- tofu init -input=false -lockfile=readonly
- tofu apply -input=false -auto-approve checkpoint.cache
The local execution is sequential by design; the GitLab mapping adds queue-level serialization while the backend must still provide locking.
9. Deliberate plan-substitution test
Before the real apply, make a copy and prove the digest guard detects tampering without destroying the original:
cp checkpoint.cache tampered.cache
printf 'not-a-plan\n' >> tampered.cache
cp evidence/plan.sha256 evidence/tampered.sha256
sed 's/checkpoint.cache/tampered.cache/' evidence/tampered.sha256 > evidence/tampered-check.sha256
set +e
sha256sum -c evidence/tampered-check.sha256 > evidence/tamper-result.txt 2>&1
rc=$?
set -e
test "$rc" -ne 0
printf 'tamper_guard=blocked\n' >> evidence/tamper-result.txt
rm -f tampered.cache evidence/tampered.sha256 evidence/tampered-check.sha256
The expected result is a checksum failure. The original
checkpoint.cache and plan.sha256 remain
untouched.
10. Apply the exact reviewed plan
test -z "$(git status --porcelain --untracked-files=no)"
sha256sum -c evidence/plan.sha256
grep -Fx "source_sha=$(git rev-parse HEAD)" evidence/review.txt
sha256sum .terraform.lock.hcl | tee evidence/provider-lock-before-apply.sha256
cmp evidence/provider-lock.sha256 evidence/provider-lock-before-apply.sha256
tofu init -input=false -lockfile=readonly
tofu apply -input=false -auto-approve checkpoint.cache | tee evidence/apply.txt
If any guard fails, the checkpoint fails. Do not regenerate the plan under the same review record.
11. Verify state and external result independently
tofu state list | tee evidence/state-list.txt
test "$(cat sandbox/checkpoint.txt)" = 'reviewed-checkpoint-v1'
sha256sum sandbox/checkpoint.txt | tee evidence/external-file.sha256
tofu plan -input=false -detailed-exitcode -var='message=reviewed-checkpoint-v1' -no-color > evidence/post-apply-plan.txt
rc=$?
test "$rc" -eq 0
printf 'post_apply_match=yes\n' | tee evidence/post-apply-result.txt
This verifies IaC state address, external file content and a no-change refresh plan. These are distinct checks.
12. Inject and detect controlled drift
printf 'emergency-out-of-band-change\n' > sandbox/checkpoint.txt
sha256sum sandbox/checkpoint.txt | tee evidence/drifted-external.sha256
set +e
tofu plan -input=false -detailed-exitcode -var='message=reviewed-checkpoint-v1' -no-color > evidence/drift-plan.txt
rc=$?
set -e
printf 'drift_exit_code=%s\n' "$rc" | tee evidence/drift-result.txt
test "$rc" -eq 2
Checkpoint rule: stop here and review the drift evidence. Do not auto-apply from the drift detector.
13. Reconcile the disposable drift under explicit intent
Because this is a synthetic lab, you may authorize reconciliation. Generate a new plan and a new digest; do not reuse the original review record:
tofu plan -input=false -out=reconcile.cache -var='message=reviewed-checkpoint-v1' -no-color | tee evidence/reconcile-plan.txt
sha256sum reconcile.cache | tee evidence/reconcile.sha256
printf 'reconcile_review=approved-for-disposable-lab\n' > evidence/reconcile-review.txt
sha256sum -c evidence/reconcile.sha256
tofu apply -input=false -auto-approve reconcile.cache | tee evidence/reconcile-apply.txt
test "$(cat sandbox/checkpoint.txt)" = 'reviewed-checkpoint-v1
14. Cleanup is another reviewed mutation
tofu plan -destroy -input=false -out=destroy.cache -var='message=reviewed-checkpoint-v1' -no-color | tee evidence/destroy-plan.txt
sha256sum destroy.cache | tee evidence/destroy.sha256
printf 'destroy_review=approved-for-exact-lab-target\n' > evidence/destroy-review.txt
sha256sum -c evidence/destroy.sha256
tofu apply -input=false -auto-approve destroy.cache | tee evidence/destroy-apply.txt
test ! -e sandbox/checkpoint.txt
tofu state list | tee evidence/state-after-destroy.txt
test ! -s evidence/state-after-destroy.txt
printf 'cleanup=verified\n' | tee evidence/cleanup.txt
15. Required evidence packet
Keep the following non-secret evidence:
- Pipeline source/ref/SHA and pipeline/job IDs when run in GitLab; local source SHA otherwise.
- OpenTofu version and provider-lock digest.
- Plan digest and review record.
- State/backend identity (not raw state contents).
- Mutation serialization key/queue evidence for GitLab runs.
- Apply trace plus external file digest/content assertion.
- Post-apply no-change plan result.
- Drift exit code and drift plan.
- Reconciliation and destroy plan digests/review records.
- Cleanup verification and assumptions/limitations note.
Do not include terraform.tfstate, provider cache,
access tokens or a raw plan containing real secrets in the shareable
evidence packet.
16. Optional authentic GitLab execution
If you run the checkpoint in a throwaway GitLab project, capture:
printf 'source=%s ref=%s sha=%s pipeline=%s job=%s\n' "$CI_PIPELINE_SOURCE" "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA" "$CI_PIPELINE_ID" "$CI_JOB_ID"
tofu version
sha256sum .terraform.lock.hcl
Use GitLab-managed state or another disposable remote backend rather than transferring local state between ephemeral runners. Preserve real pipeline/job IDs and resource-group waiting evidence. If you use the OpenTofu component, pin a released version and only request an OpenTofu version that release actually supports.
17. Verification checklist
- The repository/source SHA is recorded and provider lock is committed.
- The plan is saved, hashed and reviewed before apply.
- The tamper test proves checksum substitution is blocked.
-
The apply consumes the saved plan and does not run
init -upgradeor an implicit re-plan. - The mutation domain is serialized in the GitLab mapping.
- Post-apply external state and no-change plan both match.
- Drift returns detailed exit code 2 and does not auto-apply.
- Reconciliation has a new plan/review identity.
- Destroy has its own plan digest/review and removes only the lab resource.
- No raw state, real credential or production target appears in the evidence packet.
18. What this checkpoint proves—and does not prove
It proves plan/apply identity, provider-lock discipline, local state isolation, explicit review metadata, serialization design, external drift detection and reviewed cleanup. It does not prove a real cloud provider role is least-privileged, a GitLab-managed state backend is configured, production state recovery is tested, or organization approvals/policies are correct. Those require authorized environment-specific evidence.
Knowledge check
Why does reconciliation get a new plan digest instead of reusing the original approved plan?
External state changed after the original apply. The reconciliation is a new proposed mutation and needs its own evidence/authorization.
What does the tamper test prove?
That apply can detect substitution/corruption of the saved plan before mutation when the reviewed digest is checked.
Why does the checkpoint inspect both
tofu state list and the external file?
State records IaC ownership, while the file is provider/external reality. Either one alone is incomplete evidence.
If two GitLab pipelines target different state names, must they always share one resource group?
No. Serialize the actual shared mutation domain. Independent state/targets can use different resource groups if they truly cannot conflict.
What production control from this chapter most directly prevents “reviewed plan A, applied plan B”?
Saved-plan identity: preserve the plan, hash it, bind review to source/provider/backend metadata, and verify the same digest before apply.
19. What Chapter 29 adds to the production operating model
You can now treat IaC as a controlled state transition rather than a shell command: exact source and provider identity, protected backend/state, reviewable saved plan, explicit authorization, serialized exact apply, external verification, drift evidence and reviewed recovery/cleanup.
Version and compatibility note
GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.
Official references and version notes
Documentation verification date: 2026-09-12.
GitLab-managed OpenTofu/Terraform state and merge-request plan
integration are available on Free, Premium, and Ultimate. GitLab’s
legacy Terraform CI/CD templates were removed in GitLab 18.0;
current guidance is the versioned OpenTofu CI/CD component.
GitLab-managed OpenTofu/Terraform state support in the current CLI
flow was introduced in GitLab 18.3 and requires
glab 1.66+. Plan files are not encrypted by GitLab or
OpenTofu and can contain credentials or sensitive values; restrict
artifact access and never treat plan output as safe by default. The
current released OpenTofu component used for optional examples is
4.8.1; that release supports OpenTofu
1.12.5 and publishes signed rootless images. The
standalone local lab pins OpenTofu 1.12.6 (latest
stable on the verification date) and
hashicorp/local 2.9.0. For a real GitLab-managed state
checkpoint, add state name/version/lock evidence, authentic
actor/role evidence and backend recovery testing without copying
sensitive state into logs.
- Infrastructure as Code with OpenTofu and GitLab — official reference.
- GitLab-managed Terraform/OpenTofu state — official reference.
- OpenTofu integration in merge requests — official reference.
- CI/CD artifacts reports — terraform — official reference.
- CI/CD YAML syntax — resource_group and artifacts access — official reference.
- Resource groups — official reference.
- GitLab IaC troubleshooting — official reference.
- GitLab OpenTofu CI/CD component — official reference.
- GitLab deprecations and removals — official reference.
- glab opentofu state — official reference.
- OpenTofu 1.12 documentation — official reference.
- OpenTofu releases — official reference.
- HashiCorp local provider — official reference.
Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.
Keep the academy open
Support free, practical DevOps education.
Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0
Send only Ethereum/ERC-20 compatible assets to this
address.