Infrastructure-as-Code Pipelines, Terraform/OpenTofu Workflows, Plan Reviews, State Safety, and Drift Controls: Concepts, Architecture, and Mental Model
Build a precise IaC mental model that separates source, tool/provider identity, state/lock, reviewed plan, apply authorization, state mutation, and drift evidence instead of treating a green plan as infrastructure truth.
Learning objectives
- Explain why IaC validation, planning, review, applying, state mutation, and drift detection are separate control points.
- Trace source change → fmt/validate/security → state read/lock → saved plan → review → apply exact plan → state update → drift/health verification.
- Distinguish repository configuration, compiled GitLab pipeline state, IaC tool/provider state, backend state, plan artifacts, and external provider state.
- Explain why saved plans and state files are sensitive evidence rather than ordinary harmless artifacts.
- Use read-only evidence before mutating state or infrastructure.
1. The practical problem: a green plan is not an applied, healthy infrastructure state
Chapter 28 separated GitLab deployment intent from Kubernetes runtime truth. Infrastructure as Code adds another chain of states. A file can be valid HCL while the provider cannot authenticate. A plan can be correct for commit A while an apply runs code from commit B. A plan can be reviewed while another pipeline mutates the same state first. An apply can exit successfully while the resulting service is unhealthy. A drift detector can find unexpected change without knowing whether that change was malicious, emergency repair, operator action, or an expected provider-side mutation.
The production question is therefore not “did Terraform/OpenTofu succeed?” It is: which exact source, tool/provider set, backend state, reviewed plan, actor, serialized mutation and external verification produced this result?
2. Mental model: from source change to reconciled state
Start with an exact Git source revision. Formatting, validation and IaC/security checks inspect configuration without intentionally changing infrastructure. Planning reads provider/backend state—usually under a state lock—and produces a proposed change set. Reviewers inspect that plan and its metadata. An authorized apply must consume the exact reviewed plan under serialized access. Only then does the backend state change. Afterward, read-only provider checks and later drift plans test whether reality still matches intent.
The arrows matter. If the source or provider lock changes after planning, the reviewed hypothesis has changed. If apply re-plans, the reviewer did not approve the thing being applied. If two applies run concurrently, each may have planned against a different predecessor state. If drift is automatically repaired without review, an emergency change can be erased before its cause is understood.
3. Define the state before changing it
| State layer | Evidence to capture | Why it matters |
|---|---|---|
| Source/revision |
CI_PIPELINE_SOURCE, ref,
CI_COMMIT_SHA, pipeline/job IDs
|
A plan is meaningful only for the exact source/configuration that produced it. |
| Compiled CI configuration | Merged YAML, rules decision, component/tag inputs, plan/apply job graph | Proves what GitLab actually scheduled, not what the repository file looks like in isolation. |
| IaC tool/provider identity |
OpenTofu/Terraform version,
.terraform.lock.hcl digest, provider
versions/checksums
|
Apply semantics can change when tool/provider selections change. |
| Backend/state | Backend address/name, state serial/lineage where available, lock owner/status | State is operational data and may contain secrets; it is not an ordinary build artifact. |
| Plan artifact | Saved plan filename, SHA-256 digest, source SHA, lockfile digest, creation job ID | The review target must be a concrete plan, not “whatever a later plan says.” |
| Authorization | Review/approval identity, manual gate, protected branch/environment or local simulation record | Authorization to mutate infrastructure is independent of a successful plan. |
| Mutation serialization |
State lock plus GitLab resource_group or
equivalent coordinator
|
Prevents two otherwise-valid jobs from racing against the same target/state. |
| External/provider state | Created/changed resource identity and read-only post-apply verification | A successful apply process is not the same as target health or intended real-world state. |
| Drift | Plan exit code, drift plan, changed resource identities, policy result | Drift is an observation that requires review; it is not an instruction to auto-fix blindly. |
| Governance/evidence | Plan/review/apply linkage, exceptions, cleanup/destroy plan and verification | Makes mutation and recovery independently auditable. |
4. Terminology without the folklore
| Term | Precise meaning | Common mistake |
|---|---|---|
| Configuration | HCL plus modules, variables, provider requirements and backend declaration. | Calling configuration “state.” |
| Provider | Plugin that reads/mutates a target API or local resource. | Assuming the CLI version uniquely determines provider behavior. |
| State | IaC’s recorded mapping/attributes for managed objects and metadata such as lineage/serial. | Committing or publishing it as a normal artifact. |
| Backend | Storage/locking mechanism for state. | Treating backend authentication as ordinary application credentials. |
| Lock | Mutual exclusion protecting a state operation. | Disabling locking because a pipeline is blocked. |
| Plan | Proposed actions computed from configuration, provider logic and observed state. | Treating it as a timeless approval. |
| Saved plan | Binary plan file later consumable by apply. | Regenerating it after review and calling it the same plan. |
| Drift | Difference between desired/configured expectations and refreshed external reality. | Automatically correcting every drift event. |
5. Read-only inspection first
Before planning or applying, prove the source and local tool/provider intent. In GitLab, also record the pipeline source/SHA and IDs. Avoid state downloads unless the investigation actually requires them because state can contain secrets.
printf 'source=%s sha=%s pipeline=%s job=%s\n' \
"${CI_PIPELINE_SOURCE:-local}" "${CI_COMMIT_SHA:-$(git rev-parse HEAD)}" \
"${CI_PIPELINE_ID:-local}" "${CI_JOB_ID:-local}"
git status --short
git rev-parse HEAD
tofu version
tofu providers
sha256sum .terraform.lock.hcl 2>/dev/null || true
tofu state list 2>/dev/null || true
tofu state list lists addresses but does not print full
state values. If a backend is remote, record backend/state identity
and lock status through the least-privileged interface rather than
copying state into logs.
6. Where IaC sits in the GitLab pipeline lifecycle
GitLab first compiles YAML/components and decides whether a pipeline/job exists. Only a queued job selected by a runner can execute OpenTofu. OpenTofu then initializes providers/backend, validates configuration, reads state/provider APIs and computes a plan. The plan artifact can be uploaded even though nothing has been applied. A manual/approval control can authorize a later job. That job must still obtain the state lock, load the same provider selections and successfully mutate the target. Finally, GitLab job success and external/provider health remain separate observations.
This prevents a common causal error: an MR plan widget is pipeline feedback, not evidence that the infrastructure changed.
7. Current GitLab/OpenTofu integration boundaries
GitLab’s current IaC guidance is OpenTofu-first. The old
Terraform.gitlab-ci.yml family was removed in GitLab
18.0. For reusable GitLab-managed workflows, use a released,
versioned OpenTofu CI/CD component. GitLab-managed
OpenTofu/Terraform state and MR plan report integration are
available on all tiers.
As of this chapter’s verification date, GitLab’s OpenTofu component
release 4.8.1 supports OpenTofu 1.12.5.
The local lab uses the independently released OpenTofu
1.12.6; do not request a component image version that
the selected component release did not publish.
8. State is operational memory, not source code
GitLab-managed state is encrypted at rest by GitLab, but users with sufficient project roles can download it. State can still contain secret material depending on providers/resources. The safer model is to keep state in a protected backend, minimize who can read/write it, and avoid transporting it between jobs as a generic artifact.
Current GitLab permissions distinguish read from lock/write operations. With GitLab-managed state, read-only plan access can be given more broadly than apply/lock permissions. That maps naturally to separation of duties: reviewers can inspect plan evidence while only a narrower identity can mutate state.
9. Plan files are sensitive
A saved plan can include configuration, values and backend-related
data. GitLab documentation explicitly warns that
plan.cache/plan.json are not encrypted by
default. A plan should therefore have short retention, restricted
artifact access and synthetic data in course labs.
plan:
stage: plan
script:
- tofu plan -input=false -out=plan.cache
- sha256sum plan.cache > plan.sha256
artifacts:
access: developer
expire_in: 1 day
paths:
- plan.cache
- plan.sha256
artifacts:access limits UI/API artifact downloads but
does not replace project visibility, job-token boundaries or backend
authorization. A production team may need tighter controls than
developer.
10. Review the plan identity, not just its prose
The review packet should bind the plan to at least: source SHA,
pipeline/job ID, IaC CLI version, provider-lock digest,
backend/state name, plan digest and intended apply environment.
Human-readable plan text is useful, but the binary plan digest is
the anchor that proves which file later reached apply.
tofu plan -input=false -out=plan.cache
sha256sum plan.cache > evidence/plan.sha256
sha256sum .terraform.lock.hcl > evidence/provider-lock.sha256
{
printf 'source_sha=%s\n' "$(git rev-parse HEAD)"
printf 'tofu=%s\n' "$(tofu version -json | jq -r .terraform_version)"
} > evidence/plan-metadata.txt
11. Apply exactly the reviewed plan
A safe plan/apply split avoids tofu apply without a
saved plan after review because that command computes a new plan.
Before applying, verify the saved plan digest and metadata,
initialize with the committed provider lock file, obtain the state
lock, then apply the saved plan.
sha256sum -c evidence/plan.sha256
grep -Fx "source_sha=$(git rev-parse HEAD)" evidence/plan-metadata.txt
tofu init -input=false -lockfile=readonly
tofu apply -input=false -auto-approve plan.cache
-auto-approve is appropriate here only because the
human/policy gate occurred before the exact saved plan reached this
command. It is not a substitute for review.
12. State locking and GitLab resource groups solve related but different races
The backend lock prevents unsafe concurrent state operations at the
IaC layer. GitLab resource_group serializes selected
jobs across pipelines before they reach the backend. Use the same
logical key for jobs that mutate the same state/target.
apply_lab:
stage: apply
resource_group: tofu-lab-state
when: manual
allow_failure: false
script:
- tofu apply -input=false -auto-approve plan.cache
The default resource-group process mode is unordered;
current GitLab also supports oldest_first,
newest_first and newest_ready_first,
configured via the resource-group API. Ordering is an operational
policy decision, while state locking remains a correctness boundary.
13. Drift detection is observation, not auto-remediation
tofu plan -detailed-exitcode can distinguish no change
(0), error (1) and proposed changes (2). In a drift-only job, code 2
should create evidence and a review path; it should not immediately
run apply.
set +e
tofu plan -input=false -detailed-exitcode -no-color > evidence/drift-plan.txt
rc=$?
set -e
case "$rc" in
0) printf 'drift=none\n' ;;
2) printf 'drift=detected-review-required\n' ;;
*) printf 'drift_check=error rc=%s\n' "$rc" >&2; exit "$rc" ;;
esac
14. GitLab plan report is a review surface, not the saved plan
artifacts:reports:terraform can feed an OpenTofu plan
summary into the merge request widget. That improves review
ergonomics, but the report is not the binary plan and it does not
authorize apply. Preserve the underlying plan/evidence according to
your security model and keep sensitive values out of the report.
15. Misconceptions to remove now
| Claim | Why it is wrong | Safer replacement |
|---|---|---|
| “Validate passed, so apply is safe.” | Validation checks configuration structure; it does not prove credentials, state freshness, policy, review or target health. | Treat validation as one non-mutating gate in a longer evidence chain. |
| “The MR widget is the approved plan.” | The widget is a GitLab-rendered report; the apply input is a saved plan or a newly computed plan. | Bind review to source/provider/plan identities and verify before apply. |
| “State locking makes resource_group unnecessary.” | The backend may lock correctly but queued pipeline ordering and side effects outside state can still conflict. | Use backend locking plus job-level serialization for the same mutation domain. |
| “Drift should be auto-fixed.” | Drift can represent emergency repair, compromise, provider mutation or accepted manual action. | Preserve drift evidence and review causality before reconciliation. |
| “State is encrypted, so it is safe to download.” | Encryption at rest does not remove sensitive contents or access-control risk. | Minimize state readers and avoid unnecessary copies. |
Knowledge check
A reviewer approves a plan, but the apply job runs
tofu apply without the saved plan. What
changed?
Apply computed a new plan, so the approved artifact is no longer the mutation input. The control failure is plan/apply identity, even if the new plan looks similar.
Why record .terraform.lock.hcl or its
digest?
It binds provider selections/checksums to the plan/apply evidence. Provider changes can alter planning and apply behavior even when HCL is unchanged.
Does a backend state lock make two GitLab apply jobs equivalent to one serialized job?
No. The backend may block one operation, but job ordering, external side effects and operator intent still need pipeline-level coordination such as a resource group.
What does tofu plan -detailed-exitcode code 2
mean?
A plan succeeded and proposes changes. It is not a generic failure code and should normally create drift/change evidence for review.
Why is an MR plan report not sufficient rollback evidence?
It does not prove which plan file was applied, the exact state predecessor, provider identity, or the external result. Preserve the plan/apply/state linkage separately.
Version and compatibility note
GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.
Official references and version notes
Documentation verification date: 2026-09-12.
GitLab-managed OpenTofu/Terraform state and merge-request plan
integration are available on Free, Premium, and Ultimate. GitLab’s
legacy Terraform CI/CD templates were removed in GitLab 18.0;
current guidance is the versioned OpenTofu CI/CD component.
GitLab-managed OpenTofu/Terraform state support in the current CLI
flow was introduced in GitLab 18.3 and requires
glab 1.66+. Plan files are not encrypted by GitLab or
OpenTofu and can contain credentials or sensitive values; restrict
artifact access and never treat plan output as safe by default. The
current released OpenTofu component used for optional examples is
4.8.1; that release supports OpenTofu
1.12.5 and publishes signed rootless images. The
standalone local lab pins OpenTofu 1.12.6 (latest
stable on the verification date) and
hashicorp/local 2.9.0. The local lab deliberately
avoids cloud credentials. Production backends/providers add
authentication and external failure modes but do not change the
state-separation model.
- Infrastructure as Code with OpenTofu and GitLab — official reference.
- GitLab-managed Terraform/OpenTofu state — official reference.
- OpenTofu integration in merge requests — official reference.
- CI/CD artifacts reports — terraform — official reference.
- CI/CD YAML syntax — resource_group and artifacts access — official reference.
- Resource groups — official reference.
- GitLab IaC troubleshooting — official reference.
- GitLab OpenTofu CI/CD component — official reference.
- GitLab deprecations and removals — official reference.
- glab opentofu state — official reference.
- OpenTofu 1.12 documentation — official reference.
- OpenTofu releases — official reference.
- HashiCorp local provider — official reference.
Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.
Keep the academy open
Support free, practical DevOps education.
Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0
Send only Ethereum/ERC-20 compatible assets to this
address.