Pipeline Supply-Chain Security, Dependency Pinning, Image Digests, Signing, Provenance, and Trusted Builders: Guided Hands-On Workflow and Core Operations

1. Disposable scenario and safety boundary

This lab creates only files under glci-ch32-lab. It uses Git, Python 3.11+ and OpenSSL 3.x already installed on the workstation or disposable runner. There are no network calls, registries, cloud accounts, GitLab mutations, real signing identities or production credentials. The private key is generated inside .keys/, never uploaded as an artifact, and deleted during cleanup.

git --version
python3 --version
openssl version
mkdir -p glci-ch32-lab && cd glci-ch32-lab
printf 'lab_root=%s\n' "$PWD"

2. Create exact synthetic source and a lock input

The lock file represents a resolved dependency input. It is deliberately local, so the lab can show content binding without downloading packages.

mkdir -p src dist evidence .keys
printf 'hello supply chain\n' > src/message.txt
printf 'example-lib==1.4.2 sha256:7d9f-local-training-digest\n' > deps.lock
cat > build.py <<'PYBUILD'
from pathlib import Path
import hashlib, json, os

src = Path('src/message.txt').read_bytes()
lock = Path('deps.lock').read_bytes()
out = Path('dist/app.bundle')
out.parent.mkdir(parents=True, exist_ok=True)
# Deterministic tiny artifact: explicit separators + exact input bytes.
payload = b'CH32-BUNDLE-V1\n' + src + b'\n--LOCK--\n' + lock
out.write_bytes(payload)
sha = hashlib.sha256(payload).hexdigest()
Path('evidence').mkdir(exist_ok=True)
Path('evidence/artifact.sha256').write_text(f'{sha}  dist/app.bundle\n', encoding='utf-8')
print(sha)
PYBUILD
cat > provenance.py <<'PYPROV'
from pathlib import Path
import hashlib, json, os

artifact = Path('dist/app.bundle')
artifact_sha = hashlib.sha256(artifact.read_bytes()).hexdigest()
lock_sha = hashlib.sha256(Path('deps.lock').read_bytes()).hexdigest()
stmt = {
  '_type': 'https://in-toto.io/Statement/v1',
  'subject': [{'name':'app.bundle','digest':{'sha256':artifact_sha}}],
  'predicateType': 'https://slsa.dev/provenance/v1',
  'predicate': {
    'buildDefinition': {
      'buildType': 'https://example.invalid/devops-academy/ch32/local-build/v1',
      'externalParameters': {
        'source_sha': os.environ.get('SOURCE_SHA','local-uncommitted'),
        'pipeline_source': os.environ.get('PIPELINE_SOURCE','local'),
      },
      'resolvedDependencies': [
        {'uri':'file:deps.lock','digest':{'sha256':lock_sha}}
      ]
    },
    'runDetails': {
      'builder': {'id':'urn:devops-academy:ch32:local-openssl-builder'},
      'metadata': {'invocationId': os.environ.get('INVOCATION_ID','ch32-local-001')}
    }
  }
}
Path('evidence/provenance.json').write_text(
    json.dumps(stmt, indent=2, sort_keys=True) + '\n', encoding='utf-8')
print(artifact_sha)
PYPROV
cat > verify.py <<'PYVERIFY'
from pathlib import Path
import hashlib, json, subprocess, sys

artifact = Path(sys.argv[1])
prov = Path(sys.argv[2])
expected_builder = sys.argv[3]
statement = json.loads(prov.read_text(encoding='utf-8'))
actual = hashlib.sha256(artifact.read_bytes()).hexdigest()
subject = statement['subject'][0]['digest']['sha256']
builder = statement['predicate']['runDetails']['builder']['id']
if actual != subject:
    raise SystemExit(f'DENY digest mismatch actual={actual} provenance={subject}')
if builder != expected_builder:
    raise SystemExit(f'DENY unexpected builder {builder}')
print(f'ALLOW digest={actual} builder={builder}')
PYVERIFY

3. Establish a real source SHA before building

Use a fake local author and commit only synthetic files. The source commit gives the provenance statement a stable revision rather than a branch label.

git init -q
git config user.name 'DevOps Academy Lab'
git config user.email 'lab@example.invalid'
printf '.keys/\ndist/\nevidence/\n' > .gitignore
git add src/message.txt deps.lock build.py provenance.py verify.py .gitignore
git commit -qm 'ch32 synthetic source'
SOURCE_SHA=$(git rev-parse HEAD)
printf 'source_sha=%s\n' "$SOURCE_SHA"

4. Predict the state changes

5. Build and record digest

export SOURCE_SHA=$(git rev-parse HEAD)
export PIPELINE_SOURCE=local
export INVOCATION_ID=ch32-local-001
python3 build.py
sha256sum dist/app.bundle deps.lock
python3 provenance.py
python3 -m json.tool evidence/provenance.json >/dev/null

The artifact digest is the promotion identity. The lock digest is one material identity. The local provenance statement binds both, but because your build script generated it, it is not independently trustworthy until you add authenticated signing/builder controls.

6. Generate an ephemeral Ed25519 signing identity

The lab uses a local key only to teach the distinction between bytes, signature and signer identity. It is not a recommendation to store a long-lived private key in CI variables.

openssl genpkey -algorithm ED25519 -out .keys/ch32-ed25519.pem
openssl pkey -in .keys/ch32-ed25519.pem -pubout -out evidence/builder.pub.pem
openssl pkey -pubin -in evidence/builder.pub.pem -outform DER   | sha256sum | awk '{print $1}' > evidence/builder-key.sha256
printf 'public_key_fingerprint=' && cat evidence/builder-key.sha256

7. Sign the exact artifact and provenance bytes

openssl pkeyutl -sign -inkey .keys/ch32-ed25519.pem -rawin   -in dist/app.bundle -out evidence/app.bundle.sig
openssl pkeyutl -sign -inkey .keys/ch32-ed25519.pem -rawin   -in evidence/provenance.json -out evidence/provenance.sig
sha256sum evidence/app.bundle.sig evidence/provenance.json evidence/provenance.sig

The private key never moves into evidence/. The retained public key fingerprint becomes the expected identity for this disposable lab. A production key-based model would protect that key in a signing service/HSM; a keyless model would verify an OIDC-backed certificate identity and issuer instead.

8. Verify cryptography, subject digest and builder identity separately

openssl pkeyutl -verify -pubin -inkey evidence/builder.pub.pem -rawin   -in dist/app.bundle -sigfile evidence/app.bundle.sig
openssl pkeyutl -verify -pubin -inkey evidence/builder.pub.pem -rawin   -in evidence/provenance.json -sigfile evidence/provenance.sig
python3 verify.py dist/app.bundle evidence/provenance.json   urn:devops-academy:ch32:local-openssl-builder

Three independent checks just happened: the artifact signature matched the expected key; the provenance file signature matched that key; and the provenance subject/builder values matched policy. A production verifier should also check source/materials, policy version, trusted transparency evidence where applicable, and artifact location/digest.

9. Compare mutable and immutable dependency references

No registry is required to see the logic. Imagine a human label that changes its target:

reference=builder:stable -> sha256:111111...   # Monday
reference=builder:stable -> sha256:222222...   # Friday
immutable=builder@sha256:111111...             # still the Monday bytes

A tag is valuable for discovery and lifecycle management. A digest is valuable for exact content identity. In production, record both when useful, but make verification/promotion decisions on the digest.

10. Map the same controls into GitLab CI/CD

The following is a teaching skeleton. It uses an image digest to demonstrate immutable execution identity and keeps signing local to the job. A production project should use an organization-reviewed builder image that already contains pinned tools rather than installing mutable packages at runtime.

stages: [build, verify]

default:
  image: alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce

build-evidence:
  stage: build
  variables:
    RUNNER_GENERATE_ARTIFACTS_METADATA: "true"
  script:
    - printf 'source=%s sha=%s pipeline=%s job=%s runner=%s\n' "$CI_PIPELINE_SOURCE" "$CI_COMMIT_SHA" "$CI_PIPELINE_ID" "$CI_JOB_ID" "$CI_RUNNER_ID"
    - printf 'Build with an organization-pinned toolchain image in production.\n'
    - printf 'synthetic artifact for %s\n' "$CI_COMMIT_SHA" > app.bundle
    - sha256sum app.bundle > app.bundle.sha256
  artifacts:
    name: ch32-$CI_COMMIT_SHA
    paths: [app.bundle, app.bundle.sha256]
    expire_in: 1 day

verify-evidence:
  stage: verify
  needs:
    - job: build-evidence
      artifacts: true
  script:
    - sha256sum -c app.bundle.sha256
    - printf 'Verification job consumes retained bytes; it does not rebuild them.\n'

Runner provenance metadata is stored alongside the artifact when supported/configured. Do not confuse metadata ingestion/upload with a verification-policy decision; inspect and verify it explicitly before promotion.

11. Optional GitLab.com keyless signing path

GitLab's official pattern uses an ID token with audience sigstore, signs the artifact or image digest, preserves the Cosign bundle, and verifies both expected certificate identity and GitLab issuer. This path is Free/Premium/Ultimate on GitLab.com. Self-Managed needs its own Sigstore deployment.

keyless-signing-concept:
  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore
  script:
    - printf 'Use a pinned, verified Cosign 3.x tool in the real builder image.\n'
    - printf 'Verify exact certificate identity + OIDC issuer downstream.\n'

12. Challenge: choose the control layer

A pipeline uses a full source SHA and a pinned job image digest, but downloads scanner-linux-amd64 from a /latest/ URL and executes it without a checksum. Which layer is wrong? The source and image identities are fine; the tool dependency layer is mutable. Fix it with a reviewed tool version and independently obtained digest/signature, then preserve those values in evidence.

13. Cleanup

test "$(basename "$PWD")" = glci-ch32-lab
rm -f .keys/ch32-ed25519.pem
cd ..
rm -rf glci-ch32-lab
printf 'cleanup=local-lab-removed\n'

In a real pipeline, never delete provenance/signature evidence merely because an ephemeral signing key is destroyed. Key destruction and evidence retention serve different purposes.

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-12. GitLab/GitLab Runner 19.3.2 is the current patched 19.3 baseline used for version notes in this chapter; Runner tag v19.3.2 was published 2026-09-10. Current GitLab documentation states that include:integrity is available on Free/Premium/Ultimate and rejects a remote include whose Base64-encoded SHA-256 does not match; cross-project includes should use a full 40-character commit SHA when stable immutability is required; CI/CD component consumers should prefer a commit SHA or trusted release version over moving selectors; and job/service images can use name@sha256:digest. Runner can emit in-toto/SLSA provenance metadata with RUNNER_GENERATE_ARTIFACTS_METADATA=true. GitLab's SLSA CI/CD components are available across tiers for signing/verifying Runner-generated provenance, while native SLSA Level 3 attestations and the Attestations API remain Ultimate experimental capabilities. GitLab.com keyless Sigstore signing is available across tiers; Self-Managed requires self-hosted Sigstore infrastructure. The mandatory lab below uses only local OpenSSL/Python/Git and creates no registry, cloud, package, or production side effects. Executable local commands assume Git, Python 3.11+ and OpenSSL 3.x. Record the exact versions present in your environment; the chapter does not download a signing tool or create a long-lived key.

• CI/CD YAML syntax — include, include:integrity and image digests — official reference.
• CI/CD includes — official reference.
• CI/CD components and version pinning — official reference.
• Run jobs in Docker containers — image checksums — official reference.
• GitLab Runner artifact provenance metadata — official reference.
• GitLab SLSA guidance — official reference.
• GitLab SLSA Level 3 attestations — official reference.
• GitLab Attestations API — official reference.
• GitLab Sigstore keyless signing examples — official reference.
• GitLab Self-Managed Sigstore integration — official reference.
• Sigstore Cosign installation — official reference.
• SLSA provenance specification — official reference.
• in-toto attestation framework — official reference.
• GitLab Runner tags/releases — official reference.
• GitLab 19.3.2 patch release — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.