Chapter 32Lesson 03~225 minutes

Pipeline Supply-Chain Security, Dependency Pinning, Image Digests, Signing, Provenance, and Trusted Builders: Configuration, Design Choices, and Tradeoffs

Choose tag versus digest, branch versus commit-pinned reuse, key-based versus keyless signing, centralized versus project-local builders, and build-time versus deployment-time verification using explicit trust and portability tradeoffs.

DesignSigstoreComponentsTrusted builderTradeoffs

Learning objectives

  • Choose tag versus digest according to update and reproducibility requirements.
  • Choose branch/release/commit references for reusable CI configuration and components.
  • Compare key-based and keyless signing without conflating cryptographic validity with identity policy.
  • Choose centralized trusted builders or project-local builders based on isolation, autonomy and auditability.
  • Decide where verification must occur: build, release, promotion and deployment.

1. Start from the evidence contract, not from a favorite signing tool

A supply-chain design is easier to reason about when consumers state what evidence they require. Example: “Production accepts only artifact digest D when provenance binds D to source SHA S, the signature matches project P's expected builder identity, all CI configuration dependencies are pinned, and the artifact was not rebuilt after verification.” In GitLab evidence, record CI_PIPELINE_SOURCE and the full CI_COMMIT_SHA alongside pipeline/job/builder identity so the policy refers to one exact execution. Tools then implement this contract.

2. Tag versus digest

Choice Advantages Risks Good use
Tag only Human-friendly; update automation is simple Mutable; same string can resolve to different bytes Discovery/development where exact reproducibility is not required
Digest only Immutable content identity Harder for humans; update process must discover new digest High-assurance execution/promotion
Tag + digest Human intent plus immutable bytes Must update both consistently Recommended operational pattern for important CI images

A digest does not replace vulnerability scanning or signature verification. It prevents silent content movement under the same reference.

3. Branch include versus release versus commit SHA

Reference Update behavior Trust implication Recommendation
Branch such as main Moves on every merge Consumer can change without consumer-repo diff Avoid for high-trust reusable config
Catalog release 1.4.2 Stable by maintainer convention/governance Trusts component release process Good when maintainer/release controls are trusted
Commit SHA Exact repository content Strongest Git revision identity Preferred pin for critical third-party/project component reuse
Partial semver / ~latest Automatically floats within selector Convenient but future pipelines compile different code Use only where automatic update risk is explicitly accepted

GitLab component guidance explicitly recommends a commit SHA (preferred) or trusted release version and warns against moving latest-style selectors.

4. Project SHA versus remote integrity

include:project with a full commit SHA ties resolution to Git history on the GitLab instance. include:remote is just fetched content, so include:integrity adds a content hash. These are not identical controls: Git history provides repository identity/change governance; integrity proves only that fetched bytes match the expected digest.

# Compute the SRI-like value GitLab expects for include:integrity.
python3 - <<'PY'
import base64, hashlib, pathlib
b = pathlib.Path('remote-template.yml').read_bytes()
print('sha256-' + base64.b64encode(hashlib.sha256(b).digest()).decode())
PY

5. Key-based versus keyless signing

Model Identity source Operational burden Main verification requirement
Long-lived key Possession/control of private key Key generation, HSM/vault, rotation, revocation Pin expected public key/certificate and protect signing service
Ephemeral lab key Disposable local key Minimal; not production identity Verify key fingerprint; destroy private key after lab
Sigstore keyless OIDC identity → short-lived certificate + transparency evidence No long-lived signing key, but depends on issuer/Fulcio/Rekor trust Constrain certificate identity and OIDC issuer; verify bundle/transparency evidence

GitLab.com keyless signing uses GitLab OIDC. A cryptographically valid certificate from the wrong project/ref/config path must be rejected. For Self-Managed, public Sigstore does not accept arbitrary GitLab issuers; GitLab documents a self-hosted Sigstore architecture.

6. Centralized trusted builder versus per-project build

Design Benefits Costs/risks Evidence emphasis
Per-project builder Team autonomy, simple ownership Many toolchains/policies; larger trust surface Runner identity, pinned builder image, project policy
Central trusted builder service/pool Consistent hardening, controlled signing identity, easier evidence policy Platform bottleneck; tenancy/isolation must be strong Builder pool identity, source authorization, immutable inputs
Separate build and signer Private signing material isolated from build step Requires secure digest handoff and authorization Artifact digest is the interface; signer never rebuilds

Chapter 31's runner isolation still applies. “Trusted builder” is not a label on a runner; it is a verifiable combination of controlled configuration, identity, isolation, allowed source context and evidence generation.

7. Verify at build or deploy? Usually both, for different reasons

Build-time verification protects the builder from compromised dependencies. Promotion/release verification prevents unverified outputs from entering a distribution channel. Deployment-time verification proves the exact candidate still satisfies policy and has not been substituted. The checks can reuse evidence, but each protects a different transition.

Verification gates protect transitions
            flowchart TD
            A[Pinned inputs] -->|verify inputs| B[Build]
            B --> C[Artifact digest]
            C -->|sign + provenance| D[Release candidate]
            D -->|verify identity/digest| E[Promoted artifact]
            E -->|verify again| F[Deployment]
          

8. GitLab provenance choices and tier boundaries

Capability Current availability Use
Runner artifact provenance metadata Runner feature; generated with RUNNER_GENERATE_ARTIFACTS_METADATA=true Automatically bind artifact SHA-256 to GitLab job/builder/source metadata
GitLab SLSA CI/CD components Free/Premium/Ultimate Sign Runner provenance and create/verify supply-chain attestations in reusable jobs
GitLab.com keyless Sigstore examples Free/Premium/Ultimate on GitLab.com Sign artifacts/images with OIDC-backed identity
Self-Managed keyless Sigstore Free/Premium/Ultimate with self-hosted Sigstore infrastructure Same identity pattern under organization-controlled Fulcio/Rekor/CT
Native SLSA Level 3 attestations + Attestations API Ultimate; experimental; hosted support varies as documented Platform-generated higher-assurance attestations; do not make mandatory lab depend on it

9. SLSA is a requirements framework, not a badge you get from one JSON file

A SLSA provenance-shaped statement is useful only when its generation and verification satisfy the relevant level's requirements. The local statement in Lesson 2 is deliberately SLSA-shaped training data: the build process can forge its own builder field. Runner-generated and signed provenance improves this, and hardened platform-generated attestations improve isolation further. Always state which guarantees you actually have.

10. Worked verification policy

PROMOTION POLICY ch32/v1
1. source SHA must equal the reviewed release commit
2. compiled CI config must use approved include/component refs
3. builder image and critical tools must have immutable identity
4. artifact SHA-256 must equal provenance subject SHA-256
5. signature must validate against expected project/builder identity
6. provenance builder/source/material policy must pass
7. promoted registry/package reference must resolve to the same artifact digest
8. deployment verifies the digest again; no rebuild allowed

11. Decision table

Scenario Recommended approach Tier/offering prerequisite Observable evidence
Small Free project, no public registry Commit-SHA includes + image digests + local/organization key signing Free/local Merged config refs, image digest, artifact SHA, key fingerprint/signature
GitLab.com public/releasable artifact Pinned inputs + GitLab OIDC keyless Cosign + verification bundle Free/Premium/Ultimate GitLab.com Certificate identity/issuer, bundle, artifact digest, source/job evidence
Self-Managed regulated environment Pinned inputs + protected isolated builders + self-hosted Sigstore/HSM policy Self-Managed; infrastructure required Builder identity, internal trust root, transparency/signature records
Experiment with GitLab native Level 3 attestations Use only as optional evaluation path Ultimate GitLab.com; experimental current docs Attestation record/API + documented feature status/limitations

12. Pinning needs an update process or it becomes neglect

Immutability should not mean “never update.” A healthy process discovers new releases, verifies their publisher/signature/checksum, tests them in a branch/MR, records the old and new digest/SHA, and merges the pin update through review. Automation may open the update MR; policy decides whether humans or tests approve it.

Knowledge check

When is a release tag acceptable instead of a component commit SHA?

Why is keyless signing not “no trust”?

Why can build-time verification not replace deployment-time verification?

What does include:integrity not prove?

What is the risk of never updating a digest pin?

Next lesson

Diagnostics and failure modes

Lesson 4 deliberately breaks one binding at a time and diagnoses the correct layer without masking the original evidence.

Version and compatibility note

GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.

Official references and version notes

Documentation verification date: 2026-09-12. GitLab/GitLab Runner 19.3.2 is the current patched 19.3 baseline used for version notes in this chapter; Runner tag v19.3.2 was published 2026-09-10. Current GitLab documentation states that include:integrity is available on Free/Premium/Ultimate and rejects a remote include whose Base64-encoded SHA-256 does not match; cross-project includes should use a full 40-character commit SHA when stable immutability is required; CI/CD component consumers should prefer a commit SHA or trusted release version over moving selectors; and job/service images can use name@sha256:digest. Runner can emit in-toto/SLSA provenance metadata with RUNNER_GENERATE_ARTIFACTS_METADATA=true. GitLab's SLSA CI/CD components are available across tiers for signing/verifying Runner-generated provenance, while native SLSA Level 3 attestations and the Attestations API remain Ultimate experimental capabilities. GitLab.com keyless Sigstore signing is available across tiers; Self-Managed requires self-hosted Sigstore infrastructure. The mandatory lab below uses only local OpenSSL/Python/Git and creates no registry, cloud, package, or production side effects. The design table intentionally keeps paid/experimental features optional. The core controls—Git SHAs, hashes, local signing, image digests and evidence comparison—remain free and portable.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.