Production Capstone: Build, Secure, Scale, Observe, and Govern a Complete GitLab Delivery Platform: Security, Governance, and Reliability Validation

1. Validation asks “what prevents or detects the wrong thing?”

Lesson 2 proved that the happy path works. Production readiness asks harder questions. Can untrusted code reach a privileged runner? Can a moving include change the pipeline without a project commit? Can a broad token cross a project boundary? Can a release artifact be swapped between build and deploy? Can the person triggering a deployment also satisfy its approval policy? Can a policy be bypassed or an exception live forever? Can operators detect drift and recover without destroying evidence?

For each control, name the asset, threat or failure mode, preventive control, detective evidence, owner and rollback. A checkbox without observable proof is not validation.

2. Validate runner trust boundaries

GitLab Runner executes user-controlled scripts. Current Runner security guidance warns that privileged Docker effectively disables container security mechanisms and can expose the host to privilege escalation. Therefore, “runner online” is not a security property. You need explicit trust classes.

3. Validate reusable configuration and dependency identity

Copied YAML hides origin and makes patching inconsistent. Reusable configuration improves maintainability only if its revision is reviewable. Current GitLab pipeline-security guidance recommends specific refs for included configuration when possible. Components add a catalog/reuse contract and typed inputs, but a moving semantic selector or tag is still different from a commit SHA.

4. Validate identity by purpose, not convenience

CI_JOB_TOKEN, personal access tokens, deploy tokens, trigger tokens and OIDC ID tokens solve different problems. A production design should explain why each identity exists and which resource can accept it. Current job-token guidance uses an allowlist for cross-project access and still evaluates the triggering user’s permissions. OIDC id_tokens let a job present a short-lived signed identity to an external provider, but the provider’s trust policy remains an external control.

5. Validate artifact identity and supply-chain evidence

The build artifact and the deployment authorization are different states. A deployment may be approved for “release 1.4” while the actual bytes differ unless you retain and verify a digest. The mandatory lab already produced dist/SHA256SUMS and evidence/provenance.json. Verify both before deployment and again at the target boundary.

cd "${LAB:-${TMPDIR:-/tmp}/gitlab-ch38-capstone}"
sha256sum -c dist/SHA256SUMS
python - <<'PY'
import json, hashlib
p=json.load(open('evidence/provenance.json'))
actual=hashlib.sha256(open(p['subject'],'rb').read()).hexdigest()
print({'declared':p['sha256'],'actual':actual,'source_sha':p['source_sha']})
assert actual == p['sha256']
PY

GitLab’s native attestations API and glab attestation can verify provenance-shaped attestations, but current documentation marks the feature Experimental and Ultimate. Treat it as an optional integration, not the mandatory production foundation of this course.

6. Validate deployment authorization separately from target health

Protected environments and deployment approvals can restrict who may deploy and who must approve, but they are Premium/Ultimate. Even with approvals, GitLab explicitly separates approval from running the deployment job, and neither approval nor job success proves the external target is healthy.

7. Validate policy enforcement and exception lifecycle

Chapter 36 established the key rule: policy origin must remain visible. On Ultimate, pipeline execution policies can inject or override CI behavior from a security policy project. In the mandatory path, simulate the same control objective with a versioned policy file and an exception register.

{
  "policy_id": "capstone-release-v1",
  "policy_version": "git:deadbeef-replace-with-real-sha",
  "scope": "default-branch release",
  "required_controls": ["tests", "artifact-digest", "target-verification"],
  "exceptions": [
    {
      "id": "EX-2026-001",
      "owner": "platform-owner@example.invalid",
      "reason": "training example only",
      "expires_at": "2026-09-20T00:00:00Z",
      "scope": "synthetic security evidence only"
    }
  ]
}

An exception is not “turn control off.” It is explicit residual risk with owner, bounded scope, expiry and follow-up. If the product feature does not provide a generic waiver-expiry object, keep that lifecycle in a reviewed governance system rather than inventing a hidden field.

8. Validate SLOs and recovery readiness

Capture pipeline duration, queue time, job failure classes, runner saturation and deployment verification latency in production. The local lab cannot reproduce GitLab queue metrics, so write the assumptions explicitly. A good dashboard separates creation/compilation failures, pending/scheduling, runner/executor failures, script/tool/network failures, report/artifact ingestion, deployment authorization, deployment status and target health.

Recovery criteria should be binary enough to audit: configuration lints/compiles; the intended jobs exist; trusted work lands on trusted runners; artifact digest matches; deployment target reports the intended digest; the original failure remains preserved; policy/exception evidence points to a reviewed version; and SLOs return inside the accepted range.

9. Capstone design decision table

10. Production-readiness validation checklist

• Source, reusable config and policy versions are reviewable and recoverable.
• Pipeline inputs/variables have explicit ownership; secrets are not printed or embedded in artifacts.
• Untrusted and trusted workloads have justified runner isolation boundaries.
• Artifact identity survives promotion; deploy does not silently rebuild release bytes.
• Deployment authorization, deployment status and external target health are verified independently.
• Policy scope and exception lifecycle are visible; audit retention meets the organization’s requirement.
• Version/deprecation assumptions are timestamped and owned.
• Recovery playbooks preserve first-failure evidence and specify smallest safe rerun scope.

11. Lesson summary

The capstone is now more than a working pipeline. Its trust boundaries, identities, artifacts, deployment controls, governance and operational evidence have explicit failure tests. Lesson 4 deliberately breaks representative layers and uses the Chapter 37 evidence-first recovery method to prove that the operating model survives failure.

Knowledge check

Official references and version notes

Further reading — current official GitLab sources

Version-sensitive assumptions in this production capstone were checked against current official GitLab documentation on 2026-09-13. Re-check your exact GitLab, GitLab Runner, glab, executor, component, image/tool and external-provider versions before applying the operating model to production.

• GitLab Docs — GitLab 19.3 release notes
• GitLab Docs — Critical patch release 19.3.2
• GitLab Docs — Release and maintenance policy
• GitLab Docs — Deprecations and removals
• GitLab Docs — CI/CD YAML syntax reference
• GitLab Docs — Use CI/CD configuration from other files
• GitLab Docs — CI/CD inputs
• GitLab Docs — CI/CD components
• GitLab Docs — Pipeline security
• GitLab Docs — Validate CI/CD configuration / CI Lint
• GitLab Docs — Runner security
• GitLab Docs — CI/CD job token
• GitLab Docs — OIDC authentication using ID tokens
• GitLab Docs — Job artifacts
• GitLab Docs — CI/CD artifact report types
• GitLab Docs — Environments and deployments
• GitLab Docs — Protected environments
• GitLab Docs — Deployment approvals
• GitLab Docs — Pipeline execution policies
• GitLab Docs — Audit events
• GitLab Docs — Audit events API
• GitLab Docs — Attestations API (Experimental)
• GitLab Docs — glab attestation (Experimental)

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.