Chapter 29Lesson 01~195 minutes

Infrastructure-as-Code Pipelines, Terraform/OpenTofu Workflows, Plan Reviews, State Safety, and Drift Controls: Concepts, Architecture, and Mental Model

Build a precise IaC mental model that separates source, tool/provider identity, state/lock, reviewed plan, apply authorization, state mutation, and drift evidence instead of treating a green plan as infrastructure truth.

IaCOpenTofuState safetySaved planDrift

Learning objectives

  • Explain why IaC validation, planning, review, applying, state mutation, and drift detection are separate control points.
  • Trace source change → fmt/validate/security → state read/lock → saved plan → review → apply exact plan → state update → drift/health verification.
  • Distinguish repository configuration, compiled GitLab pipeline state, IaC tool/provider state, backend state, plan artifacts, and external provider state.
  • Explain why saved plans and state files are sensitive evidence rather than ordinary harmless artifacts.
  • Use read-only evidence before mutating state or infrastructure.

1. The practical problem: a green plan is not an applied, healthy infrastructure state

Chapter 28 separated GitLab deployment intent from Kubernetes runtime truth. Infrastructure as Code adds another chain of states. A file can be valid HCL while the provider cannot authenticate. A plan can be correct for commit A while an apply runs code from commit B. A plan can be reviewed while another pipeline mutates the same state first. An apply can exit successfully while the resulting service is unhealthy. A drift detector can find unexpected change without knowing whether that change was malicious, emergency repair, operator action, or an expected provider-side mutation.

The production question is therefore not “did Terraform/OpenTofu succeed?” It is: which exact source, tool/provider set, backend state, reviewed plan, actor, serialized mutation and external verification produced this result?

Operating rule: plan is evidence of proposed change, apply is an authorized mutation, state is sensitive operational memory, and drift is a diagnostic signal. None of these should be silently substituted for another.

2. Mental model: from source change to reconciled state

Start with an exact Git source revision. Formatting, validation and IaC/security checks inspect configuration without intentionally changing infrastructure. Planning reads provider/backend state—usually under a state lock—and produces a proposed change set. Reviewers inspect that plan and its metadata. An authorized apply must consume the exact reviewed plan under serialized access. Only then does the backend state change. Afterward, read-only provider checks and later drift plans test whether reality still matches intent.

flowchart TD A[Source ref + SHA] --> B[fmt / validate / security] B --> C[Backend state read + lock] C --> D[Saved plan + digest] D --> E[Review / approval] E --> F[Serialized apply of exact plan] F --> G[State update] G --> H[External verification] H --> I[Later drift detection] I --> J[Review / reconcile decision]

The arrows matter. If the source or provider lock changes after planning, the reviewed hypothesis has changed. If apply re-plans, the reviewer did not approve the thing being applied. If two applies run concurrently, each may have planned against a different predecessor state. If drift is automatically repaired without review, an emergency change can be erased before its cause is understood.

3. Define the state before changing it

State layer Evidence to capture Why it matters
Source/revision CI_PIPELINE_SOURCE, ref, CI_COMMIT_SHA, pipeline/job IDs A plan is meaningful only for the exact source/configuration that produced it.
Compiled CI configuration Merged YAML, rules decision, component/tag inputs, plan/apply job graph Proves what GitLab actually scheduled, not what the repository file looks like in isolation.
IaC tool/provider identity OpenTofu/Terraform version, .terraform.lock.hcl digest, provider versions/checksums Apply semantics can change when tool/provider selections change.
Backend/state Backend address/name, state serial/lineage where available, lock owner/status State is operational data and may contain secrets; it is not an ordinary build artifact.
Plan artifact Saved plan filename, SHA-256 digest, source SHA, lockfile digest, creation job ID The review target must be a concrete plan, not “whatever a later plan says.”
Authorization Review/approval identity, manual gate, protected branch/environment or local simulation record Authorization to mutate infrastructure is independent of a successful plan.
Mutation serialization State lock plus GitLab resource_group or equivalent coordinator Prevents two otherwise-valid jobs from racing against the same target/state.
External/provider state Created/changed resource identity and read-only post-apply verification A successful apply process is not the same as target health or intended real-world state.
Drift Plan exit code, drift plan, changed resource identities, policy result Drift is an observation that requires review; it is not an instruction to auto-fix blindly.
Governance/evidence Plan/review/apply linkage, exceptions, cleanup/destroy plan and verification Makes mutation and recovery independently auditable.

4. Terminology without the folklore

Term Precise meaning Common mistake
Configuration HCL plus modules, variables, provider requirements and backend declaration. Calling configuration “state.”
Provider Plugin that reads/mutates a target API or local resource. Assuming the CLI version uniquely determines provider behavior.
State IaC’s recorded mapping/attributes for managed objects and metadata such as lineage/serial. Committing or publishing it as a normal artifact.
Backend Storage/locking mechanism for state. Treating backend authentication as ordinary application credentials.
Lock Mutual exclusion protecting a state operation. Disabling locking because a pipeline is blocked.
Plan Proposed actions computed from configuration, provider logic and observed state. Treating it as a timeless approval.
Saved plan Binary plan file later consumable by apply. Regenerating it after review and calling it the same plan.
Drift Difference between desired/configured expectations and refreshed external reality. Automatically correcting every drift event.

5. Read-only inspection first

Before planning or applying, prove the source and local tool/provider intent. In GitLab, also record the pipeline source/SHA and IDs. Avoid state downloads unless the investigation actually requires them because state can contain secrets.

printf 'source=%s sha=%s pipeline=%s job=%s\n' \
  "${CI_PIPELINE_SOURCE:-local}" "${CI_COMMIT_SHA:-$(git rev-parse HEAD)}" \
  "${CI_PIPELINE_ID:-local}" "${CI_JOB_ID:-local}"
git status --short
git rev-parse HEAD
tofu version
tofu providers
sha256sum .terraform.lock.hcl 2>/dev/null || true
tofu state list 2>/dev/null || true

tofu state list lists addresses but does not print full state values. If a backend is remote, record backend/state identity and lock status through the least-privileged interface rather than copying state into logs.

6. Where IaC sits in the GitLab pipeline lifecycle

GitLab first compiles YAML/components and decides whether a pipeline/job exists. Only a queued job selected by a runner can execute OpenTofu. OpenTofu then initializes providers/backend, validates configuration, reads state/provider APIs and computes a plan. The plan artifact can be uploaded even though nothing has been applied. A manual/approval control can authorize a later job. That job must still obtain the state lock, load the same provider selections and successfully mutate the target. Finally, GitLab job success and external/provider health remain separate observations.

This prevents a common causal error: an MR plan widget is pipeline feedback, not evidence that the infrastructure changed.

7. Current GitLab/OpenTofu integration boundaries

GitLab’s current IaC guidance is OpenTofu-first. The old Terraform.gitlab-ci.yml family was removed in GitLab 18.0. For reusable GitLab-managed workflows, use a released, versioned OpenTofu CI/CD component. GitLab-managed OpenTofu/Terraform state and MR plan report integration are available on all tiers.

As of this chapter’s verification date, GitLab’s OpenTofu component release 4.8.1 supports OpenTofu 1.12.5. The local lab uses the independently released OpenTofu 1.12.6; do not request a component image version that the selected component release did not publish.

Version discipline: a component tag, the OpenTofu version requested by that component, the component image digest and the provider lock file are four different identities. Record all that materially affect the executable workflow.

8. State is operational memory, not source code

GitLab-managed state is encrypted at rest by GitLab, but users with sufficient project roles can download it. State can still contain secret material depending on providers/resources. The safer model is to keep state in a protected backend, minimize who can read/write it, and avoid transporting it between jobs as a generic artifact.

Current GitLab permissions distinguish read from lock/write operations. With GitLab-managed state, read-only plan access can be given more broadly than apply/lock permissions. That maps naturally to separation of duties: reviewers can inspect plan evidence while only a narrower identity can mutate state.

9. Plan files are sensitive

A saved plan can include configuration, values and backend-related data. GitLab documentation explicitly warns that plan.cache/plan.json are not encrypted by default. A plan should therefore have short retention, restricted artifact access and synthetic data in course labs.

plan:
  stage: plan
  script:
    - tofu plan -input=false -out=plan.cache
    - sha256sum plan.cache > plan.sha256
  artifacts:
    access: developer
    expire_in: 1 day
    paths:
      - plan.cache
      - plan.sha256

artifacts:access limits UI/API artifact downloads but does not replace project visibility, job-token boundaries or backend authorization. A production team may need tighter controls than developer.

10. Review the plan identity, not just its prose

The review packet should bind the plan to at least: source SHA, pipeline/job ID, IaC CLI version, provider-lock digest, backend/state name, plan digest and intended apply environment. Human-readable plan text is useful, but the binary plan digest is the anchor that proves which file later reached apply.

tofu plan -input=false -out=plan.cache
sha256sum plan.cache > evidence/plan.sha256
sha256sum .terraform.lock.hcl > evidence/provider-lock.sha256
{
  printf 'source_sha=%s\n' "$(git rev-parse HEAD)"
  printf 'tofu=%s\n' "$(tofu version -json | jq -r .terraform_version)"
} > evidence/plan-metadata.txt

11. Apply exactly the reviewed plan

A safe plan/apply split avoids tofu apply without a saved plan after review because that command computes a new plan. Before applying, verify the saved plan digest and metadata, initialize with the committed provider lock file, obtain the state lock, then apply the saved plan.

sha256sum -c evidence/plan.sha256
grep -Fx "source_sha=$(git rev-parse HEAD)" evidence/plan-metadata.txt
tofu init -input=false -lockfile=readonly
tofu apply -input=false -auto-approve plan.cache

-auto-approve is appropriate here only because the human/policy gate occurred before the exact saved plan reached this command. It is not a substitute for review.

12. State locking and GitLab resource groups solve related but different races

The backend lock prevents unsafe concurrent state operations at the IaC layer. GitLab resource_group serializes selected jobs across pipelines before they reach the backend. Use the same logical key for jobs that mutate the same state/target.

apply_lab:
  stage: apply
  resource_group: tofu-lab-state
  when: manual
  allow_failure: false
  script:
    - tofu apply -input=false -auto-approve plan.cache

The default resource-group process mode is unordered; current GitLab also supports oldest_first, newest_first and newest_ready_first, configured via the resource-group API. Ordering is an operational policy decision, while state locking remains a correctness boundary.

13. Drift detection is observation, not auto-remediation

tofu plan -detailed-exitcode can distinguish no change (0), error (1) and proposed changes (2). In a drift-only job, code 2 should create evidence and a review path; it should not immediately run apply.

set +e
tofu plan -input=false -detailed-exitcode -no-color > evidence/drift-plan.txt
rc=$?
set -e
case "$rc" in
  0) printf 'drift=none\n' ;;
  2) printf 'drift=detected-review-required\n' ;;
  *) printf 'drift_check=error rc=%s\n' "$rc" >&2; exit "$rc" ;;
esac

14. GitLab plan report is a review surface, not the saved plan

artifacts:reports:terraform can feed an OpenTofu plan summary into the merge request widget. That improves review ergonomics, but the report is not the binary plan and it does not authorize apply. Preserve the underlying plan/evidence according to your security model and keep sensitive values out of the report.

15. Misconceptions to remove now

Claim Why it is wrong Safer replacement
“Validate passed, so apply is safe.” Validation checks configuration structure; it does not prove credentials, state freshness, policy, review or target health. Treat validation as one non-mutating gate in a longer evidence chain.
“The MR widget is the approved plan.” The widget is a GitLab-rendered report; the apply input is a saved plan or a newly computed plan. Bind review to source/provider/plan identities and verify before apply.
“State locking makes resource_group unnecessary.” The backend may lock correctly but queued pipeline ordering and side effects outside state can still conflict. Use backend locking plus job-level serialization for the same mutation domain.
“Drift should be auto-fixed.” Drift can represent emergency repair, compromise, provider mutation or accepted manual action. Preserve drift evidence and review causality before reconciliation.
“State is encrypted, so it is safe to download.” Encryption at rest does not remove sensitive contents or access-control risk. Minimize state readers and avoid unnecessary copies.

Knowledge check

A reviewer approves a plan, but the apply job runs tofu apply without the saved plan. What changed?

Why record .terraform.lock.hcl or its digest?

Does a backend state lock make two GitLab apply jobs equivalent to one serialized job?

What does tofu plan -detailed-exitcode code 2 mean?

Why is an MR plan report not sufficient rollback evidence?

Next lesson

Continue to the guided workflow

Lesson 2 makes the model concrete with a synthetic local file resource, a pinned OpenTofu/provider toolchain, saved-plan hashing, review metadata, exact apply, controlled drift and reviewed cleanup.

Version and compatibility note

GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.

Official references and version notes

Documentation verification date: 2026-09-12. GitLab-managed OpenTofu/Terraform state and merge-request plan integration are available on Free, Premium, and Ultimate. GitLab’s legacy Terraform CI/CD templates were removed in GitLab 18.0; current guidance is the versioned OpenTofu CI/CD component. GitLab-managed OpenTofu/Terraform state support in the current CLI flow was introduced in GitLab 18.3 and requires glab 1.66+. Plan files are not encrypted by GitLab or OpenTofu and can contain credentials or sensitive values; restrict artifact access and never treat plan output as safe by default. The current released OpenTofu component used for optional examples is 4.8.1; that release supports OpenTofu 1.12.5 and publishes signed rootless images. The standalone local lab pins OpenTofu 1.12.6 (latest stable on the verification date) and hashicorp/local 2.9.0. The local lab deliberately avoids cloud credentials. Production backends/providers add authentication and external failure modes but do not change the state-separation model.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.