Chapter 32Lesson 01~205 minutes

Pipeline Supply-Chain Security, Dependency Pinning, Image Digests, Signing, Provenance, and Trusted Builders: Concepts, Architecture, and Mental Model

Secure the pipeline itself by pinning configuration, images and tools; separating builder identity from artifact identity; binding signatures and provenance to exact digests; and verifying the promoted bytes before deployment.

Supply chainPinningDigestsSigningProvenance

Learning objectives

  • Model pipeline dependencies as supply-chain inputs that can change independently of application source.
  • Pin CI configuration, components, container images and tools with the strongest immutable identity practical for each type.
  • Separate checksum, signature, signer identity, provenance, trusted-builder identity and deployment authorization.
  • Trace exact source/configuration → builder → artifact digest → signature/provenance → verification policy → promoted bytes.
  • Inspect current state before changing any dependency or signing policy.

1. The practical problem: a secure runner can execute compromised inputs perfectly

Chapter 31 hardened the runner boundary. That is necessary but not sufficient. A perfectly isolated runner can still pull tool:latest, compile a remote CI include from a moving branch, execute a compromised installer, or deploy a container tag that now points to different bytes. In those cases the runner is doing exactly what the pipeline requested; the supply-chain problem is that the request itself was mutable or insufficiently verified.

Chapter 32 therefore asks a different question: can a reviewer prove which configuration, image, tool, builder and artifact bytes participated in the result? The answer must survive time. A label like v1 or latest is useful to humans, but it is not enough evidence when the object behind that label can move.

Chapter invariant: promotion is allowed only for an artifact whose digest is known, whose producing configuration/dependencies are pinned or integrity-checked, whose signer/builder identity matches policy, and whose downstream consumer verifies the same immutable identity.

2. Mental model: trust is a chain of bindings, not one green checkmark

Start with trusted source and CI configuration. Resolve every external pipeline dependency to an immutable or integrity-checked reference. Execute it on a builder whose identity and isolation you understand. Hash the resulting artifact. Sign those exact bytes or an attestation bound to that digest. Generate provenance that names the subject digest and build context. Finally, verify the expected signer/builder and subject before promotion or deployment. Each arrow is a binding that can fail independently.

Supply-chain binding path
            flowchart TD
            A[Source SHA + reviewed CI config] --> B[Pinned includes/components/images/tools]
            B --> C[Trusted isolated builder]
            C --> D[Artifact bytes + SHA-256]
            D --> E[Signature + provenance]
            E --> F[Verification policy]
            F --> G[Promote/deploy exact digest]
            G --> H[Consumer re-verifies digest/identity]
          

A signature that verifies with some key does not prove the expected project signed it. Provenance that names an artifact but contains the wrong digest does not bind to the bytes. An image digest pins content but says nothing about whether the image is trustworthy. Build evidence and authorization remain separate controls.

3. Define the state before changing it

State layer Read-only evidence Supply-chain question
Source/revision CI_PIPELINE_SOURCE, ref, full CI_COMMIT_SHA, repository URL Which exact source revision requested the build?
Compiled CI configuration Merged configuration, project/local/remote/component references, integrity fields Which exact pipeline instructions were compiled, and were remote instructions immutable?
Reusable configuration Component/project include ref, component SHA/version, remote URL + integrity Can the referenced configuration change without a reviewed update here?
Execution image/tools Image name + digest, package/tool version, tool checksum/signature Which exact executable environment and tools ran?
Builder identity Runner ID/version/executor, job ID, project/ref context, trust class Which execution boundary produced the artifact, and why is that builder trusted?
Artifact identity Artifact path, size and SHA-256 digest Which exact bytes are being signed, promoted or deployed?
Signature identity Public key fingerprint or keyless certificate identity + OIDC issuer Who/what signed those bytes, and is that the expected signer?
Provenance Statement subject digest, builder ID, source/materials, invocation Does metadata bind the artifact to the claimed source/build context?
Verification policy Expected digest, signer identity, issuer, builder, dependency pins What must be true before promotion/deployment is allowed?
Promotion/external state Registry/package digest, release asset digest, deployed digest Did downstream systems consume the verified bytes rather than rebuild or retag ambiguously?

4. Different dependency types need different pinning strategies

Dependency Weak/moving reference Stronger reference What to verify
Cross-project CI include ref: main Full 40-character commit SHA Repository/project identity + SHA + reviewed file path
Remote CI include URL whose content can change include:integrity Base64 SHA-256, ideally immutable URL too Fetched content hash at pipeline compilation
CI/CD component @~latest or branch Commit SHA preferred, or trusted release version Component source/release governance and exact reference
Container job image alpine:latest alpine:3.22@sha256:… Registry/repository + digest; signature if policy requires
Downloaded tool Unversioned “latest” URL Versioned URL + published checksum/signature Digest and expected publisher identity
Language dependency Floating range without lock Lockfile + registry integrity metadata Resolved version/content hash and registry source
Produced artifact Filename/tag only SHA-256 digest + signature/provenance Bytes, signer identity, builder/source binding

5. GitLab configuration pinning: compilation can fail before any job exists

For a private project on the same GitLab instance, include:project can use a full commit SHA. GitLab explicitly recommends the full 40-character SHA when stability matters. For public remote YAML, include:integrity accepts a Base64-encoded SHA-256. If the fetched content does not match, GitLab refuses to process it and pipeline compilation fails—there is no runner job to debug.

include:
  - project: platform/ci-library
    ref: 0123456789abcdef0123456789abcdef01234567
    file: /templates/build.yml
  - remote: https://example.invalid/public-ci/scan.yml
    integrity: sha256-L3/GAoKaw0Arw6hDCKeKQlV1QPEgHYxGBHsH4zG1IY8=

The example remote domain is intentionally reserved and is not fetched in the lab. The point is state ownership: integrity is checked while GitLab resolves configuration, before the job graph and before runner assignment.

6. Components: semantic versions are governance labels; SHA is the strongest revision pin

Current GitLab components can be referenced by commit SHA, tag, branch, partial semantic version or ~latest. GitLab's security guidance recommends a specific commit SHA (preferred) or a release version tag and warns against moving selectors such as latest. A release tag can be operationally excellent when maintainers protect releases, but it is still a governance promise; a commit SHA identifies repository content directly.

include:
  - component: $CI_SERVER_FQDN/platform/secure-build/build@e3262fdd0914fa823210cdb79a8c421e2cef79d8
    inputs:
      stage: build

7. Image tag versus image digest

GitLab's image and services syntax accepts <name>@<digest>. A digest pins the manifest content that the runner requests. Keep a human-readable tag beside the digest when it helps operators understand the intended release, but treat the digest as the execution identity.

default:
  image: alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce

inspect-image-context:
  script:
    - printf 'pipeline=%s job=%s sha=%s\n' "$CI_PIPELINE_ID" "$CI_JOB_ID" "$CI_COMMIT_SHA"
    - cat /etc/alpine-release
Important: this digest is used as a concrete pinning example from the verified Docker Hub record, not as a recommendation that this historical image is vulnerability-free. Immutability and security are different properties.

8. Checksum, signature and identity answer different questions

Evidence Answers Does not answer
SHA-256 digest Are these the same bytes? Who produced or approved them
Signature verified by a public key Did the holder of the corresponding private key sign these bytes? Whether that key is the expected trusted identity
Keyless Sigstore certificate + bundle Did an identity authenticated by the configured OIDC issuer sign, with transparency evidence? Whether your policy selected the correct project/ref/workflow identity
Provenance statement What artifact digest/source/materials/builder does the statement claim? Whether the statement itself is authentic unless it is signed/trusted
Trusted-builder policy Is this builder allowed to create promotable artifacts? Whether artifact bytes match the verified provenance subject

9. GitLab provenance: useful metadata is not magic trust

GitLab Runner can generate artifact provenance metadata when RUNNER_GENERATE_ARTIFACTS_METADATA=true. Current docs describe an in-toto v0.1 Statement with a SLSA 1.0 provenance predicate, including artifact SHA-256 subjects, source, job entry point and Runner builder information. GitLab also provides SLSA CI/CD components for signing/verifying Runner-generated provenance and creating verification summary attestations.

Native GitLab SLSA Level 3 attestations are a different feature: current docs label them Ultimate, GitLab.com, experimental. Do not silently substitute an experimental hosted attestation feature for a production trust model. Likewise, a locally generated JSON file in this chapter is pedagogical evidence, not proof of a hardened SLSA builder.

10. Keyless Sigstore: identity becomes part of verification

On GitLab.com, GitLab documents keyless Cosign signing across tiers using an id_tokens token with audience sigstore. Verification should constrain both certificate identity and OIDC issuer. The identity includes project/config/ref context; accepting any valid Fulcio certificate is not enough. Current GitLab docs require Cosign 2.0.1 or newer and Sigstore's current documentation uses Cosign v3 syntax. Self-Managed instances use their own Sigstore infrastructure because the public service does not trust arbitrary self-managed GitLab issuers.

sign:
  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore
  script:
    - echo 'Use an organization-pinned Cosign release in real pipelines.'
    - echo 'Sign the artifact/image digest and preserve the verification bundle.'

11. Read-only inspection first

Before changing pins or signatures, capture non-secret identity:

printf 'source=%s ref=%s sha=%s\n' "$CI_PIPELINE_SOURCE" "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA"
printf 'pipeline=%s job=%s runner=%s runner_version=%s\n'   "$CI_PIPELINE_ID" "$CI_JOB_ID" "$CI_RUNNER_ID" "$CI_RUNNER_VERSION"
printf 'project=%s config=%s\n' "$CI_PROJECT_PATH" "$CI_CONFIG_PATH"
sha256sum dist/app.bundle 2>/dev/null || true

Outside the job, inspect merged CI configuration, resolved component/include references, registry digest, tool release/checksum source, protected ref rules and the public key or expected keyless certificate identity. Do not log private keys, OIDC tokens or credential values.

12. Common misconceptions

Misconception Correction
“The pipeline is green, so its dependencies were trusted.” Green means the configured jobs succeeded; it does not prove immutable inputs or expected signer identity.
“A digest means the image is safe.” A digest means content identity is stable. Vulnerability, provenance and publisher trust are separate.
“Any valid signature is enough.” Verification must bind to the expected key/certificate identity and issuer.
“Provenance proves the artifact is secure.” Provenance describes build relationships; it can be truthful or forged depending on how it is generated and authenticated.
“Rebuilding from the same commit reproduces the release.” Moving base images, tools, repositories or timestamps can change bytes. Promote verified bytes when exact identity matters.

Knowledge check

Why can a full commit SHA be stronger than a branch name for a CI include?

What does include:integrity change in the pipeline lifecycle?

A Cosign signature is cryptographically valid, but the certificate identity names another project. Promote?

Does Runner-generated provenance alone prove a hardened builder?

Why verify the artifact digest again at deployment?

Next lesson

Build, hash, sign and verify a tiny artifact

Lesson 2 turns the model into a disposable local workflow and then maps each evidence object to GitLab CI/CD.

Version and compatibility note

GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.

Official references and version notes

Documentation verification date: 2026-09-12. GitLab/GitLab Runner 19.3.2 is the current patched 19.3 baseline used for version notes in this chapter; Runner tag v19.3.2 was published 2026-09-10. Current GitLab documentation states that include:integrity is available on Free/Premium/Ultimate and rejects a remote include whose Base64-encoded SHA-256 does not match; cross-project includes should use a full 40-character commit SHA when stable immutability is required; CI/CD component consumers should prefer a commit SHA or trusted release version over moving selectors; and job/service images can use name@sha256:digest. Runner can emit in-toto/SLSA provenance metadata with RUNNER_GENERATE_ARTIFACTS_METADATA=true. GitLab's SLSA CI/CD components are available across tiers for signing/verifying Runner-generated provenance, while native SLSA Level 3 attestations and the Attestations API remain Ultimate experimental capabilities. GitLab.com keyless Sigstore signing is available across tiers; Self-Managed requires self-hosted Sigstore infrastructure. The mandatory lab below uses only local OpenSSL/Python/Git and creates no registry, cloud, package, or production side effects. The image digest example is intentionally evidence of immutability, not a vulnerability-free endorsement. The mandatory signing lab uses an ephemeral local Ed25519 key so no long-lived signing secret is introduced.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.