Pipeline Supply-Chain Security, Dependency Pinning, Image Digests, Signing, Provenance, and Trusted Builders: Configuration, Design Choices, and Tradeoffs
Choose tag versus digest, branch versus commit-pinned reuse, key-based versus keyless signing, centralized versus project-local builders, and build-time versus deployment-time verification using explicit trust and portability tradeoffs.
Learning objectives
- Choose tag versus digest according to update and reproducibility requirements.
- Choose branch/release/commit references for reusable CI configuration and components.
- Compare key-based and keyless signing without conflating cryptographic validity with identity policy.
- Choose centralized trusted builders or project-local builders based on isolation, autonomy and auditability.
- Decide where verification must occur: build, release, promotion and deployment.
1. Start from the evidence contract, not from a favorite signing tool
A supply-chain design is easier to reason about when consumers state
what evidence they require. Example: “Production accepts only
artifact digest D when provenance binds D to source SHA S, the
signature matches project P's expected builder identity, all CI
configuration dependencies are pinned, and the artifact was not
rebuilt after verification.” In GitLab evidence, record
CI_PIPELINE_SOURCE and the full
CI_COMMIT_SHA alongside pipeline/job/builder identity
so the policy refers to one exact execution. Tools then implement
this contract.
2. Tag versus digest
| Choice | Advantages | Risks | Good use |
|---|---|---|---|
| Tag only | Human-friendly; update automation is simple | Mutable; same string can resolve to different bytes | Discovery/development where exact reproducibility is not required |
| Digest only | Immutable content identity | Harder for humans; update process must discover new digest | High-assurance execution/promotion |
| Tag + digest | Human intent plus immutable bytes | Must update both consistently | Recommended operational pattern for important CI images |
A digest does not replace vulnerability scanning or signature verification. It prevents silent content movement under the same reference.
3. Branch include versus release versus commit SHA
| Reference | Update behavior | Trust implication | Recommendation |
|---|---|---|---|
Branch such as main |
Moves on every merge | Consumer can change without consumer-repo diff | Avoid for high-trust reusable config |
Catalog release 1.4.2 |
Stable by maintainer convention/governance | Trusts component release process | Good when maintainer/release controls are trusted |
| Commit SHA | Exact repository content | Strongest Git revision identity | Preferred pin for critical third-party/project component reuse |
Partial semver / ~latest |
Automatically floats within selector | Convenient but future pipelines compile different code | Use only where automatic update risk is explicitly accepted |
GitLab component guidance explicitly recommends a commit SHA
(preferred) or trusted release version and warns against moving
latest-style selectors.
4. Project SHA versus remote integrity
include:project with a full commit SHA ties resolution
to Git history on the GitLab instance.
include:remote is just fetched content, so
include:integrity adds a content hash. These are not
identical controls: Git history provides repository identity/change
governance; integrity proves only that fetched bytes match the
expected digest.
# Compute the SRI-like value GitLab expects for include:integrity.
python3 - <<'PY'
import base64, hashlib, pathlib
b = pathlib.Path('remote-template.yml').read_bytes()
print('sha256-' + base64.b64encode(hashlib.sha256(b).digest()).decode())
PY
5. Key-based versus keyless signing
| Model | Identity source | Operational burden | Main verification requirement |
|---|---|---|---|
| Long-lived key | Possession/control of private key | Key generation, HSM/vault, rotation, revocation | Pin expected public key/certificate and protect signing service |
| Ephemeral lab key | Disposable local key | Minimal; not production identity | Verify key fingerprint; destroy private key after lab |
| Sigstore keyless | OIDC identity → short-lived certificate + transparency evidence | No long-lived signing key, but depends on issuer/Fulcio/Rekor trust | Constrain certificate identity and OIDC issuer; verify bundle/transparency evidence |
GitLab.com keyless signing uses GitLab OIDC. A cryptographically valid certificate from the wrong project/ref/config path must be rejected. For Self-Managed, public Sigstore does not accept arbitrary GitLab issuers; GitLab documents a self-hosted Sigstore architecture.
6. Centralized trusted builder versus per-project build
| Design | Benefits | Costs/risks | Evidence emphasis |
|---|---|---|---|
| Per-project builder | Team autonomy, simple ownership | Many toolchains/policies; larger trust surface | Runner identity, pinned builder image, project policy |
| Central trusted builder service/pool | Consistent hardening, controlled signing identity, easier evidence policy | Platform bottleneck; tenancy/isolation must be strong | Builder pool identity, source authorization, immutable inputs |
| Separate build and signer | Private signing material isolated from build step | Requires secure digest handoff and authorization | Artifact digest is the interface; signer never rebuilds |
Chapter 31's runner isolation still applies. “Trusted builder” is not a label on a runner; it is a verifiable combination of controlled configuration, identity, isolation, allowed source context and evidence generation.
7. Verify at build or deploy? Usually both, for different reasons
Build-time verification protects the builder from compromised dependencies. Promotion/release verification prevents unverified outputs from entering a distribution channel. Deployment-time verification proves the exact candidate still satisfies policy and has not been substituted. The checks can reuse evidence, but each protects a different transition.
flowchart TD
A[Pinned inputs] -->|verify inputs| B[Build]
B --> C[Artifact digest]
C -->|sign + provenance| D[Release candidate]
D -->|verify identity/digest| E[Promoted artifact]
E -->|verify again| F[Deployment]
8. GitLab provenance choices and tier boundaries
| Capability | Current availability | Use |
|---|---|---|
| Runner artifact provenance metadata |
Runner feature; generated with
RUNNER_GENERATE_ARTIFACTS_METADATA=true
|
Automatically bind artifact SHA-256 to GitLab job/builder/source metadata |
| GitLab SLSA CI/CD components | Free/Premium/Ultimate | Sign Runner provenance and create/verify supply-chain attestations in reusable jobs |
| GitLab.com keyless Sigstore examples | Free/Premium/Ultimate on GitLab.com | Sign artifacts/images with OIDC-backed identity |
| Self-Managed keyless Sigstore | Free/Premium/Ultimate with self-hosted Sigstore infrastructure | Same identity pattern under organization-controlled Fulcio/Rekor/CT |
| Native SLSA Level 3 attestations + Attestations API | Ultimate; experimental; hosted support varies as documented | Platform-generated higher-assurance attestations; do not make mandatory lab depend on it |
9. SLSA is a requirements framework, not a badge you get from one JSON file
A SLSA provenance-shaped statement is useful only when its generation and verification satisfy the relevant level's requirements. The local statement in Lesson 2 is deliberately SLSA-shaped training data: the build process can forge its own builder field. Runner-generated and signed provenance improves this, and hardened platform-generated attestations improve isolation further. Always state which guarantees you actually have.
10. Worked verification policy
PROMOTION POLICY ch32/v1
1. source SHA must equal the reviewed release commit
2. compiled CI config must use approved include/component refs
3. builder image and critical tools must have immutable identity
4. artifact SHA-256 must equal provenance subject SHA-256
5. signature must validate against expected project/builder identity
6. provenance builder/source/material policy must pass
7. promoted registry/package reference must resolve to the same artifact digest
8. deployment verifies the digest again; no rebuild allowed
11. Decision table
| Scenario | Recommended approach | Tier/offering prerequisite | Observable evidence |
|---|---|---|---|
| Small Free project, no public registry | Commit-SHA includes + image digests + local/organization key signing | Free/local | Merged config refs, image digest, artifact SHA, key fingerprint/signature |
| GitLab.com public/releasable artifact | Pinned inputs + GitLab OIDC keyless Cosign + verification bundle | Free/Premium/Ultimate GitLab.com | Certificate identity/issuer, bundle, artifact digest, source/job evidence |
| Self-Managed regulated environment | Pinned inputs + protected isolated builders + self-hosted Sigstore/HSM policy | Self-Managed; infrastructure required | Builder identity, internal trust root, transparency/signature records |
| Experiment with GitLab native Level 3 attestations | Use only as optional evaluation path | Ultimate GitLab.com; experimental current docs | Attestation record/API + documented feature status/limitations |
12. Pinning needs an update process or it becomes neglect
Immutability should not mean “never update.” A healthy process discovers new releases, verifies their publisher/signature/checksum, tests them in a branch/MR, records the old and new digest/SHA, and merges the pin update through review. Automation may open the update MR; policy decides whether humans or tests approve it.
Knowledge check
When is a release tag acceptable instead of a component commit SHA?
When you trust the maintainer and protected release process and accept governance-based immutability. A commit SHA remains the strongest Git revision identity.
Why is keyless signing not “no trust”?
Trust moves from a long-lived private key to the OIDC issuer, Fulcio/transparency infrastructure, certificate identity claims and verification policy.
Why can build-time verification not replace deployment-time verification?
Build-time checks inputs during production; deployment-time verification protects the later promotion/deployment boundary against substitution or wrong candidate selection.
What does include:integrity not prove?
It proves fetched content matches a hash. It does not prove who authored that content or whether the content is safe.
What is the risk of never updating a digest pin?
You preserve reproducibility but can freeze known vulnerabilities or obsolete tools. Pin updates need an intentional review/testing process.
Version and compatibility note
GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.
Official references and version notes
Documentation verification date: 2026-09-12.
GitLab/GitLab Runner 19.3.2 is the current patched 19.3
baseline used for version notes in this chapter; Runner tag
v19.3.2 was published 2026-09-10. Current GitLab
documentation states that include:integrity is
available on Free/Premium/Ultimate and rejects a remote include
whose Base64-encoded SHA-256 does not match; cross-project includes
should use a full 40-character commit SHA when stable immutability
is required; CI/CD component consumers should prefer a commit SHA or
trusted release version over moving selectors; and job/service
images can use name@sha256:digest. Runner can emit
in-toto/SLSA provenance metadata with
RUNNER_GENERATE_ARTIFACTS_METADATA=true. GitLab's SLSA
CI/CD components are available across tiers for signing/verifying
Runner-generated provenance, while native SLSA Level 3 attestations
and the Attestations API remain Ultimate experimental capabilities.
GitLab.com keyless Sigstore signing is available across tiers;
Self-Managed requires self-hosted Sigstore infrastructure. The
mandatory lab below uses only local OpenSSL/Python/Git and creates
no registry, cloud, package, or production side effects. The design
table intentionally keeps paid/experimental features optional. The
core controls—Git SHAs, hashes, local signing, image digests and
evidence comparison—remain free and portable.
- CI/CD YAML syntax — include, include:integrity and image digests — official reference.
- CI/CD includes — official reference.
- CI/CD components and version pinning — official reference.
- Run jobs in Docker containers — image checksums — official reference.
- GitLab Runner artifact provenance metadata — official reference.
- GitLab SLSA guidance — official reference.
- GitLab SLSA Level 3 attestations — official reference.
- GitLab Attestations API — official reference.
- GitLab Sigstore keyless signing examples — official reference.
- GitLab Self-Managed Sigstore integration — official reference.
- Sigstore Cosign installation — official reference.
- SLSA provenance specification — official reference.
- in-toto attestation framework — official reference.
- GitLab Runner tags/releases — official reference.
- GitLab 19.3.2 patch release — official reference.
Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.
Keep the academy open
Support free, practical DevOps education.
Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0
Send only Ethereum/ERC-20 compatible assets to this
address.