Checkpoint Lab — Pipeline Supply-Chain Security, Dependency Pinning, Image Digests, Signing, Provenance, and Trusted Builders

1. Scenario and assumptions

You are releasing a tiny internal artifact. Production policy wants proof that downstream consumers receive the exact reviewed bytes. You have no paid GitLab feature, no cloud account and no production signing service. Build a faithful local model using Git + Python + OpenSSL, then map the evidence to GitLab Runner/Sigstore options.

2. Preflight

git --version
python3 --version
openssl version
mkdir -p glci-ch32-checkpoint && cd glci-ch32-checkpoint
mkdir -p src dist evidence .keys inputs
printf 'root=%s\n' "$PWD"

Stop if any command points to a non-disposable directory. Never reuse a production signing key for this lab.

3. Create source, dependency lock and verifier

printf 'release payload v1\n' > src/message.txt
printf 'lib-a==2.0.4 sha256:training-a204\n' > deps.lock
cat > build.py <<'PYBUILD'
from pathlib import Path
import hashlib, json, os

src = Path('src/message.txt').read_bytes()
lock = Path('deps.lock').read_bytes()
out = Path('dist/app.bundle')
out.parent.mkdir(parents=True, exist_ok=True)
# Deterministic tiny artifact: explicit separators + exact input bytes.
payload = b'CH32-BUNDLE-V1\n' + src + b'\n--LOCK--\n' + lock
out.write_bytes(payload)
sha = hashlib.sha256(payload).hexdigest()
Path('evidence').mkdir(exist_ok=True)
Path('evidence/artifact.sha256').write_text(f'{sha}  dist/app.bundle\n', encoding='utf-8')
print(sha)
PYBUILD
cat > provenance.py <<'PYPROV'
from pathlib import Path
import hashlib, json, os

artifact = Path('dist/app.bundle')
artifact_sha = hashlib.sha256(artifact.read_bytes()).hexdigest()
lock_sha = hashlib.sha256(Path('deps.lock').read_bytes()).hexdigest()
stmt = {
  '_type': 'https://in-toto.io/Statement/v1',
  'subject': [{'name':'app.bundle','digest':{'sha256':artifact_sha}}],
  'predicateType': 'https://slsa.dev/provenance/v1',
  'predicate': {
    'buildDefinition': {
      'buildType': 'https://example.invalid/devops-academy/ch32/local-build/v1',
      'externalParameters': {
        'source_sha': os.environ.get('SOURCE_SHA','local-uncommitted'),
        'pipeline_source': os.environ.get('PIPELINE_SOURCE','local'),
      },
      'resolvedDependencies': [
        {'uri':'file:deps.lock','digest':{'sha256':lock_sha}}
      ]
    },
    'runDetails': {
      'builder': {'id':'urn:devops-academy:ch32:local-openssl-builder'},
      'metadata': {'invocationId': os.environ.get('INVOCATION_ID','ch32-local-001')}
    }
  }
}
Path('evidence/provenance.json').write_text(
    json.dumps(stmt, indent=2, sort_keys=True) + '\n', encoding='utf-8')
print(artifact_sha)
PYPROV
cat > verify_release.py <<'PYVERIFY'
from pathlib import Path
import hashlib, json, subprocess, sys

artifact = Path(sys.argv[1])
provenance = Path(sys.argv[2])
public_key = Path(sys.argv[3])
sig = Path(sys.argv[4])
prov_sig = Path(sys.argv[5])
expected_builder = sys.argv[6]
expected_key_fp = Path(sys.argv[7]).read_text().strip()

# 1. Verify expected public key identity first.
der = subprocess.check_output(['openssl','pkey','-pubin','-in',str(public_key),'-outform','DER'])
actual_fp = hashlib.sha256(der).hexdigest()
if actual_fp != expected_key_fp:
    raise SystemExit('DENY unexpected public-key fingerprint')

# 2. Verify signatures for artifact and provenance statement.
for target, signature in [(artifact,sig),(provenance,prov_sig)]:
    cp = subprocess.run(['openssl','pkeyutl','-verify','-pubin','-inkey',str(public_key),
                         '-rawin','-in',str(target),'-sigfile',str(signature)],
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    if cp.returncode != 0:
        raise SystemExit(f'DENY signature verification failed for {target}')

# 3. Bind candidate bytes to signed provenance subject and expected builder.
stmt = json.loads(provenance.read_text(encoding='utf-8'))
actual_digest = hashlib.sha256(artifact.read_bytes()).hexdigest()
subject_digest = stmt['subject'][0]['digest']['sha256']
builder = stmt['predicate']['runDetails']['builder']['id']
if actual_digest != subject_digest:
    raise SystemExit('DENY artifact digest does not match provenance subject')
if builder != expected_builder:
    raise SystemExit('DENY provenance builder identity does not match policy')
print(f'ALLOW digest={actual_digest} builder={builder} key_fp={actual_fp}')
PYVERIFY
printf '.keys/\ndist/\nevidence/\n' > .gitignore

4. Commit the exact checkpoint source

git init -q
git config user.name 'DevOps Academy Lab'
git config user.email 'lab@example.invalid'
git add src/message.txt deps.lock build.py provenance.py verify_release.py .gitignore
git commit -qm 'ch32 checkpoint source'
SOURCE_SHA=$(git rev-parse HEAD)
printf '%s\n' "$SOURCE_SHA" > inputs/source-sha.txt
printf 'local\n' > inputs/pipeline-source.txt

5. Write predictions before mutation

Record at least these predictions in evidence/predictions.txt:

P1: building twice from unchanged committed source + deps.lock yields the same artifact SHA-256.
P2: the signed provenance subject SHA-256 equals the artifact SHA-256.
P3: the downstream verifier ALLOWs the original artifact only with the expected key fingerprint and builder ID.
P4: appending one byte to a candidate copy causes DENY without changing expected evidence.
P5: changing deps.lock and rebuilding produces a new artifact digest and must be treated as a new release candidate.

6. Build the reviewed candidate and prove determinism

export SOURCE_SHA=$(cat inputs/source-sha.txt)
export PIPELINE_SOURCE=$(cat inputs/pipeline-source.txt)
export INVOCATION_ID=ch32-checkpoint-001
python3 build.py
cp dist/app.bundle /tmp/ch32-first.bundle
python3 build.py
cmp /tmp/ch32-first.bundle dist/app.bundle
sha256sum dist/app.bundle deps.lock > evidence/material-digests.txt
rm -f /tmp/ch32-first.bundle
python3 provenance.py

cmp proves deterministic output for this intentionally tiny build. Real builds need stronger reproducibility controls; deterministic output is not assumed merely because source SHA is stable.

7. Create the disposable signer and sign evidence

openssl genpkey -algorithm ED25519 -out .keys/checkpoint.pem
openssl pkey -in .keys/checkpoint.pem -pubout -out evidence/builder.pub.pem
openssl pkey -pubin -in evidence/builder.pub.pem -outform DER   | sha256sum | awk '{print $1}' > evidence/expected-key-fingerprint.txt
openssl pkeyutl -sign -inkey .keys/checkpoint.pem -rawin   -in dist/app.bundle -out evidence/app.bundle.sig
openssl pkeyutl -sign -inkey .keys/checkpoint.pem -rawin   -in evidence/provenance.json -out evidence/provenance.sig

8. Capture builder/tool identity without secrets

{
  git --version
  python3 --version
  openssl version
  printf 'source_sha=%s\n' "$(cat inputs/source-sha.txt)"
  printf 'pipeline_source=%s\n' "$(cat inputs/pipeline-source.txt)"
  printf 'builder=urn:devops-academy:ch32:local-openssl-builder\n'
  printf 'gitlab_fields=CI_PIPELINE_SOURCE,CI_COMMIT_SHA,CI_PIPELINE_ID,CI_JOB_ID,CI_RUNNER_ID\n'
} > evidence/build-context.txt

9. Downstream verification of original candidate

python3 verify_release.py   dist/app.bundle evidence/provenance.json evidence/builder.pub.pem   evidence/app.bundle.sig evidence/provenance.sig   urn:devops-academy:ch32:local-openssl-builder   evidence/expected-key-fingerprint.txt   | tee evidence/original-verification.txt

Expected: ALLOW plus artifact digest, builder ID and public-key fingerprint. That line is policy outcome evidence, not a replacement for the underlying signatures/provenance.

10. Inject one tampered candidate and preserve the denial

cp dist/app.bundle dist/app.tampered
printf 'X' >> dist/app.tampered
set +e
python3 verify_release.py   dist/app.tampered evidence/provenance.json evidence/builder.pub.pem   evidence/app.bundle.sig evidence/provenance.sig   urn:devops-academy:ch32:local-openssl-builder   evidence/expected-key-fingerprint.txt   > evidence/tamper-verification.txt 2>&1
status=$?
set -e
printf 'tamper_exit=%s\n' "$status" >> evidence/tamper-verification.txt
test "$status" -ne 0
cat evidence/tamper-verification.txt

Do not overwrite artifact.sha256, resign the tampered file, or regenerate provenance. The expected denial proves the existing verification contract detects candidate substitution.

11. Change one dependency input and show a new candidate identity

cp deps.lock inputs/deps.lock.original
printf 'lib-a==2.0.5 sha256:training-a205\n' > deps.lock
python3 build.py
sha256sum dist/app.bundle > evidence/rebuilt-after-lock-change.sha256
printf 'changed dependency requires a new review/sign/provenance cycle\n'   > evidence/dependency-change-note.txt
git checkout -- deps.lock
python3 build.py

This controlled mutation demonstrates why dependency locks/materials belong in provenance. Restoring deps.lock and rebuilding returns to the original training candidate; in a real release you would not silently replace the published artifact.

12. Map checkpoint evidence to GitLab fields

13. Final evidence packet

cat > evidence/limitations.txt <<'EOF'
PROVES:
- exact local source commit identifier was recorded
- artifact and lock digests were recorded
- artifact/provenance signatures validate against the retained expected public key
- provenance subject digest and builder string were policy-checked
- tampered candidate is denied
DOES NOT PROVE:
- the local workstation/runner is a hardened trusted builder
- the self-declared builder field cannot be forged by build code
- the artifact is vulnerability-free or functionally correct
- a production keyless/HSM identity was used
- GitLab native SLSA Level 3 requirements were satisfied
EOF
sha256sum evidence/* > evidence/evidence-packet.sha256
find evidence -maxdepth 1 -type f -printf '%f\n' | sort

The limitations file is part of the evidence, not an apology. Supply-chain systems become dangerous when metadata is allowed to imply guarantees it does not actually provide.

14. Verification checklist

• Source SHA and pipeline source were recorded before building.
• Build output is deterministic for unchanged synthetic inputs.
• Artifact and dependency-lock SHA-256 values are preserved.
• Private key never appears in retained evidence.
• Public-key fingerprint is explicitly treated as signer identity.
• Artifact and provenance signatures verify.
• Provenance subject digest equals candidate artifact digest.
• Expected builder identity is checked, not merely displayed.
• Tampered artifact is denied without changing policy evidence.
• Dependency change produces a new candidate identity and requires a new review/sign cycle.
• Limitations distinguish local training evidence from hardened platform provenance.

15. Cleanup and rollback

test "$(basename "$PWD")" = glci-ch32-checkpoint
rm -f .keys/checkpoint.pem
cd ..
rm -rf glci-ch32-checkpoint
printf 'cleanup=verified-disposable-checkpoint-removed\n'

If this were a real release, cleanup would destroy ephemeral signing material/worker state but retain required signatures, provenance, digest and audit evidence according to policy.

Knowledge check

16. What Chapter 32 adds to the production operating model

You can now treat the pipeline itself as a software supply chain. Configuration includes/components, execution images and tools are explicit materials; the builder is a trust boundary; the artifact digest is the candidate identity; signatures authenticate bytes only when verifier identity policy is explicit; provenance binds source/materials/builder claims to that digest; and promotion/deployment consumes the verified artifact rather than rebuilding mutable state.

Official references and version notes

Documentation verification date: 2026-09-12. GitLab/GitLab Runner 19.3.2 is the current patched 19.3 baseline used for version notes in this chapter; Runner tag v19.3.2 was published 2026-09-10. Current GitLab documentation states that include:integrity is available on Free/Premium/Ultimate and rejects a remote include whose Base64-encoded SHA-256 does not match; cross-project includes should use a full 40-character commit SHA when stable immutability is required; CI/CD component consumers should prefer a commit SHA or trusted release version over moving selectors; and job/service images can use name@sha256:digest. Runner can emit in-toto/SLSA provenance metadata with RUNNER_GENERATE_ARTIFACTS_METADATA=true. GitLab's SLSA CI/CD components are available across tiers for signing/verifying Runner-generated provenance, while native SLSA Level 3 attestations and the Attestations API remain Ultimate experimental capabilities. GitLab.com keyless Sigstore signing is available across tiers; Self-Managed requires self-hosted Sigstore infrastructure. The mandatory lab below uses only local OpenSSL/Python/Git and creates no registry, cloud, package, or production side effects. The checkpoint intentionally uses a disposable local key and self-declared builder to make trust boundaries visible. Production designs should prefer protected signing services or GitLab OIDC/Sigstore identities plus independently generated/verified provenance appropriate to their assurance target.

• CI/CD YAML syntax — include, include:integrity and image digests — official reference.
• CI/CD includes — official reference.
• CI/CD components and version pinning — official reference.
• Run jobs in Docker containers — image checksums — official reference.
• GitLab Runner artifact provenance metadata — official reference.
• GitLab SLSA guidance — official reference.
• GitLab SLSA Level 3 attestations — official reference.
• GitLab Attestations API — official reference.
• GitLab Sigstore keyless signing examples — official reference.
• GitLab Self-Managed Sigstore integration — official reference.
• Sigstore Cosign installation — official reference.
• SLSA provenance specification — official reference.
• in-toto attestation framework — official reference.
• GitLab Runner tags/releases — official reference.
• GitLab 19.3.2 patch release — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.