Pipeline Supply-Chain Security, Dependency Pinning, Image Digests, Signing, Provenance, and Trusted Builders: Concepts, Architecture, and Mental Model
Secure the pipeline itself by pinning configuration, images and tools; separating builder identity from artifact identity; binding signatures and provenance to exact digests; and verifying the promoted bytes before deployment.
Learning objectives
- Model pipeline dependencies as supply-chain inputs that can change independently of application source.
- Pin CI configuration, components, container images and tools with the strongest immutable identity practical for each type.
- Separate checksum, signature, signer identity, provenance, trusted-builder identity and deployment authorization.
- Trace exact source/configuration → builder → artifact digest → signature/provenance → verification policy → promoted bytes.
- Inspect current state before changing any dependency or signing policy.
1. The practical problem: a secure runner can execute compromised inputs perfectly
Chapter 31 hardened the runner boundary. That is necessary but not
sufficient. A perfectly isolated runner can still pull
tool:latest, compile a remote CI include from a moving
branch, execute a compromised installer, or deploy a container tag
that now points to different bytes. In those cases the runner is
doing exactly what the pipeline requested; the supply-chain problem
is that the request itself was mutable or insufficiently verified.
Chapter 32 therefore asks a different question:
can a reviewer prove which configuration, image, tool, builder
and artifact bytes participated in the result?
The answer must survive time. A label like v1 or
latest is useful to humans, but it is not enough
evidence when the object behind that label can move.
2. Mental model: trust is a chain of bindings, not one green checkmark
Start with trusted source and CI configuration. Resolve every external pipeline dependency to an immutable or integrity-checked reference. Execute it on a builder whose identity and isolation you understand. Hash the resulting artifact. Sign those exact bytes or an attestation bound to that digest. Generate provenance that names the subject digest and build context. Finally, verify the expected signer/builder and subject before promotion or deployment. Each arrow is a binding that can fail independently.
flowchart TD
A[Source SHA + reviewed CI config] --> B[Pinned includes/components/images/tools]
B --> C[Trusted isolated builder]
C --> D[Artifact bytes + SHA-256]
D --> E[Signature + provenance]
E --> F[Verification policy]
F --> G[Promote/deploy exact digest]
G --> H[Consumer re-verifies digest/identity]
A signature that verifies with some key does not prove the expected project signed it. Provenance that names an artifact but contains the wrong digest does not bind to the bytes. An image digest pins content but says nothing about whether the image is trustworthy. Build evidence and authorization remain separate controls.
3. Define the state before changing it
| State layer | Read-only evidence | Supply-chain question |
|---|---|---|
| Source/revision |
CI_PIPELINE_SOURCE, ref, full
CI_COMMIT_SHA, repository URL
|
Which exact source revision requested the build? |
| Compiled CI configuration | Merged configuration, project/local/remote/component references, integrity fields | Which exact pipeline instructions were compiled, and were remote instructions immutable? |
| Reusable configuration | Component/project include ref, component SHA/version, remote URL + integrity | Can the referenced configuration change without a reviewed update here? |
| Execution image/tools | Image name + digest, package/tool version, tool checksum/signature | Which exact executable environment and tools ran? |
| Builder identity | Runner ID/version/executor, job ID, project/ref context, trust class | Which execution boundary produced the artifact, and why is that builder trusted? |
| Artifact identity | Artifact path, size and SHA-256 digest | Which exact bytes are being signed, promoted or deployed? |
| Signature identity | Public key fingerprint or keyless certificate identity + OIDC issuer | Who/what signed those bytes, and is that the expected signer? |
| Provenance | Statement subject digest, builder ID, source/materials, invocation | Does metadata bind the artifact to the claimed source/build context? |
| Verification policy | Expected digest, signer identity, issuer, builder, dependency pins | What must be true before promotion/deployment is allowed? |
| Promotion/external state | Registry/package digest, release asset digest, deployed digest | Did downstream systems consume the verified bytes rather than rebuild or retag ambiguously? |
4. Different dependency types need different pinning strategies
| Dependency | Weak/moving reference | Stronger reference | What to verify |
|---|---|---|---|
| Cross-project CI include | ref: main |
Full 40-character commit SHA | Repository/project identity + SHA + reviewed file path |
| Remote CI include | URL whose content can change |
include:integrity Base64 SHA-256, ideally
immutable URL too
|
Fetched content hash at pipeline compilation |
| CI/CD component | @~latest or branch |
Commit SHA preferred, or trusted release version | Component source/release governance and exact reference |
| Container job image | alpine:latest |
alpine:3.22@sha256:… |
Registry/repository + digest; signature if policy requires |
| Downloaded tool | Unversioned “latest” URL | Versioned URL + published checksum/signature | Digest and expected publisher identity |
| Language dependency | Floating range without lock | Lockfile + registry integrity metadata | Resolved version/content hash and registry source |
| Produced artifact | Filename/tag only | SHA-256 digest + signature/provenance | Bytes, signer identity, builder/source binding |
5. GitLab configuration pinning: compilation can fail before any job exists
For a private project on the same GitLab instance,
include:project can use a full commit SHA. GitLab
explicitly recommends the full 40-character SHA when stability
matters. For public remote YAML,
include:integrity accepts a Base64-encoded SHA-256. If
the fetched content does not match, GitLab refuses to process it and
pipeline compilation fails—there is no runner job to debug.
include:
- project: platform/ci-library
ref: 0123456789abcdef0123456789abcdef01234567
file: /templates/build.yml
- remote: https://example.invalid/public-ci/scan.yml
integrity: sha256-L3/GAoKaw0Arw6hDCKeKQlV1QPEgHYxGBHsH4zG1IY8=
The example remote domain is intentionally reserved and is not fetched in the lab. The point is state ownership: integrity is checked while GitLab resolves configuration, before the job graph and before runner assignment.
6. Components: semantic versions are governance labels; SHA is the strongest revision pin
Current GitLab components can be referenced by commit SHA, tag,
branch, partial semantic version or ~latest. GitLab's
security guidance recommends a specific commit SHA (preferred) or a
release version tag and warns against moving selectors such as
latest. A release tag can be operationally excellent
when maintainers protect releases, but it is still a governance
promise; a commit SHA identifies repository content directly.
include:
- component: $CI_SERVER_FQDN/platform/secure-build/build@e3262fdd0914fa823210cdb79a8c421e2cef79d8
inputs:
stage: build
7. Image tag versus image digest
GitLab's image and services syntax accepts
<name>@<digest>. A digest pins the manifest
content that the runner requests. Keep a human-readable tag beside
the digest when it helps operators understand the intended release,
but treat the digest as the execution identity.
default:
image: alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce
inspect-image-context:
script:
- printf 'pipeline=%s job=%s sha=%s\n' "$CI_PIPELINE_ID" "$CI_JOB_ID" "$CI_COMMIT_SHA"
- cat /etc/alpine-release
8. Checksum, signature and identity answer different questions
| Evidence | Answers | Does not answer |
|---|---|---|
| SHA-256 digest | Are these the same bytes? | Who produced or approved them |
| Signature verified by a public key | Did the holder of the corresponding private key sign these bytes? | Whether that key is the expected trusted identity |
| Keyless Sigstore certificate + bundle | Did an identity authenticated by the configured OIDC issuer sign, with transparency evidence? | Whether your policy selected the correct project/ref/workflow identity |
| Provenance statement | What artifact digest/source/materials/builder does the statement claim? | Whether the statement itself is authentic unless it is signed/trusted |
| Trusted-builder policy | Is this builder allowed to create promotable artifacts? | Whether artifact bytes match the verified provenance subject |
9. GitLab provenance: useful metadata is not magic trust
GitLab Runner can generate artifact provenance metadata when
RUNNER_GENERATE_ARTIFACTS_METADATA=true. Current docs
describe an in-toto v0.1 Statement with a SLSA 1.0 provenance
predicate, including artifact SHA-256 subjects, source, job entry
point and Runner builder information. GitLab also provides SLSA
CI/CD components for signing/verifying Runner-generated provenance
and creating verification summary attestations.
Native GitLab SLSA Level 3 attestations are a different feature: current docs label them Ultimate, GitLab.com, experimental. Do not silently substitute an experimental hosted attestation feature for a production trust model. Likewise, a locally generated JSON file in this chapter is pedagogical evidence, not proof of a hardened SLSA builder.
10. Keyless Sigstore: identity becomes part of verification
On GitLab.com, GitLab documents keyless Cosign signing across tiers
using an id_tokens token with audience
sigstore. Verification should constrain both
certificate identity and OIDC issuer. The identity includes
project/config/ref context; accepting any valid Fulcio certificate
is not enough. Current GitLab docs require Cosign 2.0.1 or newer and
Sigstore's current documentation uses Cosign v3 syntax. Self-Managed
instances use their own Sigstore infrastructure because the public
service does not trust arbitrary self-managed GitLab issuers.
sign:
id_tokens:
SIGSTORE_ID_TOKEN:
aud: sigstore
script:
- echo 'Use an organization-pinned Cosign release in real pipelines.'
- echo 'Sign the artifact/image digest and preserve the verification bundle.'
11. Read-only inspection first
Before changing pins or signatures, capture non-secret identity:
printf 'source=%s ref=%s sha=%s\n' "$CI_PIPELINE_SOURCE" "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA"
printf 'pipeline=%s job=%s runner=%s runner_version=%s\n' "$CI_PIPELINE_ID" "$CI_JOB_ID" "$CI_RUNNER_ID" "$CI_RUNNER_VERSION"
printf 'project=%s config=%s\n' "$CI_PROJECT_PATH" "$CI_CONFIG_PATH"
sha256sum dist/app.bundle 2>/dev/null || true
Outside the job, inspect merged CI configuration, resolved component/include references, registry digest, tool release/checksum source, protected ref rules and the public key or expected keyless certificate identity. Do not log private keys, OIDC tokens or credential values.
12. Common misconceptions
| Misconception | Correction |
|---|---|
| “The pipeline is green, so its dependencies were trusted.” | Green means the configured jobs succeeded; it does not prove immutable inputs or expected signer identity. |
| “A digest means the image is safe.” | A digest means content identity is stable. Vulnerability, provenance and publisher trust are separate. |
| “Any valid signature is enough.” | Verification must bind to the expected key/certificate identity and issuer. |
| “Provenance proves the artifact is secure.” | Provenance describes build relationships; it can be truthful or forged depending on how it is generated and authenticated. |
| “Rebuilding from the same commit reproduces the release.” | Moving base images, tools, repositories or timestamps can change bytes. Promote verified bytes when exact identity matters. |
Knowledge check
Why can a full commit SHA be stronger than a branch name for a CI include?
The SHA identifies a specific repository revision; a branch is a mutable pointer that can move without changing the consuming project.
What does include:integrity change in the pipeline lifecycle?
It adds a content-hash check during remote include resolution. A mismatch stops configuration processing before runner execution.
A Cosign signature is cryptographically valid, but the certificate identity names another project. Promote?
No. Cryptographic validity is necessary but the signer identity/issuer must also match policy.
Does Runner-generated provenance alone prove a hardened builder?
No. It records build metadata and subject digests. Trust depends on how the provenance is generated, signed/verified and how strongly the builder is isolated.
Why verify the artifact digest again at deployment?
To prove the bytes being deployed are the same verified/promoted bytes rather than a rebuild or mutable reference.
Version and compatibility note
GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.
Official references and version notes
Documentation verification date: 2026-09-12.
GitLab/GitLab Runner 19.3.2 is the current patched 19.3
baseline used for version notes in this chapter; Runner tag
v19.3.2 was published 2026-09-10. Current GitLab
documentation states that include:integrity is
available on Free/Premium/Ultimate and rejects a remote include
whose Base64-encoded SHA-256 does not match; cross-project includes
should use a full 40-character commit SHA when stable immutability
is required; CI/CD component consumers should prefer a commit SHA or
trusted release version over moving selectors; and job/service
images can use name@sha256:digest. Runner can emit
in-toto/SLSA provenance metadata with
RUNNER_GENERATE_ARTIFACTS_METADATA=true. GitLab's SLSA
CI/CD components are available across tiers for signing/verifying
Runner-generated provenance, while native SLSA Level 3 attestations
and the Attestations API remain Ultimate experimental capabilities.
GitLab.com keyless Sigstore signing is available across tiers;
Self-Managed requires self-hosted Sigstore infrastructure. The
mandatory lab below uses only local OpenSSL/Python/Git and creates
no registry, cloud, package, or production side effects. The image
digest example is intentionally evidence of immutability, not a
vulnerability-free endorsement. The mandatory signing lab uses an
ephemeral local Ed25519 key so no long-lived signing secret is
introduced.
- CI/CD YAML syntax — include, include:integrity and image digests — official reference.
- CI/CD includes — official reference.
- CI/CD components and version pinning — official reference.
- Run jobs in Docker containers — image checksums — official reference.
- GitLab Runner artifact provenance metadata — official reference.
- GitLab SLSA guidance — official reference.
- GitLab SLSA Level 3 attestations — official reference.
- GitLab Attestations API — official reference.
- GitLab Sigstore keyless signing examples — official reference.
- GitLab Self-Managed Sigstore integration — official reference.
- Sigstore Cosign installation — official reference.
- SLSA provenance specification — official reference.
- in-toto attestation framework — official reference.
- GitLab Runner tags/releases — official reference.
- GitLab 19.3.2 patch release — official reference.
Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.
Keep the academy open
Support free, practical DevOps education.
Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0
Send only Ethereum/ERC-20 compatible assets to this
address.