Production Capstone: Build, Secure, Scale, Observe, and Govern a Complete GitLab Delivery Platform: Implementation and Automation Build-Out

1. Build the platform incrementally

Atlas Relay needs a minimal delivery contract: validate source, test it, produce security/evidence records, build one immutable release bundle, and deploy that exact bundle to a simulated staging target. We will keep the YAML small and split reusable logic into a local include so configuration provenance is visible instead of copied into every project.

The local harness deliberately mirrors the pipeline stages. Each stage writes evidence before the next stage consumes it. This makes the lab useful even when no GitLab service is available.

2. Create a guarded disposable repository

LAB="${TMPDIR:-/tmp}/gitlab-ch38-capstone"
rm -rf "$LAB"
mkdir -p "$LAB"/{app,tests,ci,scripts,evidence,dist,target,governance}
cd "$LAB"
git init -q
git config user.name 'Chapter 38 Learner'
git config user.email 'learner@example.invalid'

cat > app/relay.py <<'PYAPP'
def normalize(message: str) -> str:
    return ' '.join(message.strip().split()).lower()

if __name__ == '__main__':
    print(normalize('  Atlas   Relay  '))
PYAPP

cat > tests/test_relay.py <<'PYTEST'
import unittest
from app.relay import normalize
class RelayTest(unittest.TestCase):
    def test_normalize(self):
        self.assertEqual(normalize('  Hello   CI  '), 'hello ci')
if __name__ == '__main__': unittest.main()
PYTEST

cat > governance/exception-register.json <<'JSON'
{"exceptions":[],"schema":"owner + reason + expires_at + approved_scope"}
JSON

git add .
git commit -qm 'capstone: establish Atlas Relay source'
printf 'source_sha=%s\n' "$(git rev-parse HEAD)" | tee evidence/00-source.env

3. Add a versioned reusable configuration contract

Current GitLab favors explicit reusable contracts: CI/CD components and spec:inputs are available across tiers, and inputs are typed/validated at pipeline creation. For this repository-local exercise we use include:local, because it is free, reviewable and does not require publishing a component. In production, a cross-project include or catalog component should be pinned to a reviewed tag or commit SHA where practical rather than a moving branch.

# ci/base.yml
.default_job:
  interruptible: true
  before_script:
    - printf 'source=%s pipeline=%s job=%s\n' "$CI_COMMIT_SHA" "$CI_PIPELINE_ID" "$CI_JOB_ID"

# .gitlab-ci.yml
include:
  - local: ci/base.yml

stages: [validate, test, evidence, package, deploy]

default:
  retry:
    max: 1
    when: [runner_system_failure]

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

validate-config:
  extends: .default_job
  stage: validate
  script:
    - python -m py_compile app/relay.py

unit-test:
  extends: .default_job
  stage: test
  script:
    - python -m unittest discover -s tests -v
  artifacts:
    when: always
    expire_in: 1 week
    paths: [evidence/]

build-release:
  extends: .default_job
  stage: package
  script:
    - ./scripts/build_release.sh
  artifacts:
    access: developer
    expire_in: 1 week
    paths: [dist/, evidence/]

deploy-staging:
  extends: .default_job
  stage: deploy
  needs: [build-release]
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
  script:
    - ./scripts/deploy_staging.sh
  environment:
    name: staging

State changed: the repository now contains the pipeline contract. workflow:rules controls pipeline creation; job stages control execution order; artifact settings control retained GitLab-owned evidence; the environment keyword creates GitLab deployment/environment state if run on GitLab. None of these settings proves a runner, external target, or policy exists.

4. Implement build-once and target verification scripts

The release script writes one tar archive, computes its SHA-256, and records source identity. The deployment script does not rebuild; it copies the already-built artifact identity into the simulated target and then reads it back. This is artifact promotion rather than “rebuild on deploy.”

cat > scripts/build_release.sh <<'SH'
#!/usr/bin/env sh
set -eu
mkdir -p dist evidence
SOURCE_SHA="${CI_COMMIT_SHA:-$(git rev-parse HEAD)}"
tar -czf dist/atlas-relay.tgz app tests
DIGEST="$(sha256sum dist/atlas-relay.tgz | awk '{print $1}')"
printf '%s  %s\n' "$DIGEST" dist/atlas-relay.tgz > dist/SHA256SUMS
cat > evidence/provenance.json <<EOF
{"subject":"dist/atlas-relay.tgz","sha256":"$DIGEST","source_sha":"$SOURCE_SHA","builder":"capstone-local-v1"}
EOF
printf 'artifact_digest=%s\n' "$DIGEST" | tee evidence/20-artifact.env
SH

cat > scripts/deploy_staging.sh <<'SH'
#!/usr/bin/env sh
set -eu
mkdir -p target evidence
DIGEST="$(awk '{print $1}' dist/SHA256SUMS)"
SOURCE_SHA="${CI_COMMIT_SHA:-$(git rev-parse HEAD)}"
cat > target/staging.json <<EOF
{"environment":"staging","source_sha":"$SOURCE_SHA","artifact_digest":"sha256:$DIGEST","healthy":true}
EOF
python - <<'PY'
import json
x=json.load(open('target/staging.json'))
assert x['healthy'] is True
assert x['artifact_digest'].startswith('sha256:')
print(x)
PY
cp target/staging.json evidence/30-staging-verified.json
SH
chmod +x scripts/*.sh

5. Generate test and synthetic security/SBOM evidence

A production system may use GitLab security templates, third-party scanners, SARIF, CycloneDX or registry metadata. Those feature and UI boundaries vary by tier. The mandatory lab instead creates small transparent files: a unittest trace, a dependency inventory and a synthetic finding document. The point is not to imitate a scanner; it is to prove the pipeline can retain evidence with source identity.

python -m unittest discover -s tests -v 2>&1 | tee evidence/10-tests.log
python - <<'PY'
import json, sys
json.dump({
  'python': sys.version.split()[0],
  'dependencies': [],
  'format': 'training-inventory-v1'
}, open('evidence/11-dependencies.json','w'), indent=2)
json.dump({
  'scanner': 'synthetic-capstone-check',
  'critical': 0,
  'high': 0,
  'note': 'training evidence; not a vulnerability scanner'
}, open('evidence/12-security.json','w'), indent=2)
PY

6. Run the end-to-end local pipeline simulation

Before running it, predict these changes: the source SHA should stay constant; tests should produce evidence but not mutate the target; build should create exactly one archive and digest; deploy should copy that identity into target state; the final verification must match source and digest independently.

set -eu
SOURCE_SHA="$(git rev-parse HEAD)"
printf 'prediction_source_sha=%s\n' "$SOURCE_SHA" > evidence/predictions.env

python -m py_compile app/relay.py
python -m unittest discover -s tests -v 2>&1 | tee evidence/10-tests.log
./scripts/build_release.sh
./scripts/deploy_staging.sh

ARTIFACT_DIGEST="$(awk '{print $1}' dist/SHA256SUMS)"
python - <<PY | tee evidence/40-final-verification.json
import json
x=json.load(open('target/staging.json'))
expected_sha='$SOURCE_SHA'
expected_digest='sha256:$ARTIFACT_DIGEST'
result={
 'source_matches': x['source_sha']==expected_sha,
 'digest_matches': x['artifact_digest']==expected_digest,
 'healthy': x['healthy'] is True,
 'target': x,
}
print(json.dumps(result,indent=2))
assert all([result['source_matches'],result['digest_matches'],result['healthy']])
PY

7. Map each local transition to real GitLab evidence

8. Challenge: choose the failing layer

The build job reports success and the SHA-256 file exists, but the staging target still reports the previous digest. Which layer should you inspect first? The correct answer is deployment/external-state verification, not source compilation, runner tags, or test rules. Preserve the successful build artifact and producer job evidence; then inspect the deployment record, target API/state and promotion logic before deciding whether to retry anything.

9. Bounded cleanup

The lab contains no external side effects. When you finish the full chapter, remove only the guarded workspace after archiving the evidence packet you want to keep.

# Run only after completing Lesson 5 and copying evidence you want to retain.
case "$LAB" in
  */gitlab-ch38-capstone) rm -rf "$LAB" ;;
  *) echo 'Refusing cleanup: unexpected LAB path' >&2; exit 2 ;;
esac

10. Lesson summary

You now have a coherent free/disposable delivery path: versioned repository configuration, reusable local CI logic, tests, synthetic security evidence, build-once artifact identity, provenance-shaped metadata, staging target state and independent verification. Lesson 3 subjects that design to security, governance and reliability validation rather than assuming “green pipeline” means “production ready.”

Knowledge check

Official references and version notes

Further reading — current official GitLab sources

Version-sensitive assumptions in this production capstone were checked against current official GitLab documentation on 2026-09-13. Re-check your exact GitLab, GitLab Runner, glab, executor, component, image/tool and external-provider versions before applying the operating model to production.

• GitLab Docs — GitLab 19.3 release notes
• GitLab Docs — Critical patch release 19.3.2
• GitLab Docs — Release and maintenance policy
• GitLab Docs — Deprecations and removals
• GitLab Docs — CI/CD YAML syntax reference
• GitLab Docs — Use CI/CD configuration from other files
• GitLab Docs — CI/CD inputs
• GitLab Docs — CI/CD components
• GitLab Docs — Pipeline security
• GitLab Docs — Validate CI/CD configuration / CI Lint
• GitLab Docs — Runner security
• GitLab Docs — CI/CD job token
• GitLab Docs — OIDC authentication using ID tokens
• GitLab Docs — Job artifacts
• GitLab Docs — CI/CD artifact report types
• GitLab Docs — Environments and deployments
• GitLab Docs — Protected environments
• GitLab Docs — Deployment approvals
• GitLab Docs — Pipeline execution policies
• GitLab Docs — Audit events
• GitLab Docs — Audit events API
• GitLab Docs — Attestations API (Experimental)
• GitLab Docs — glab attestation (Experimental)

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.