Pipeline Supply-Chain Security, Dependency Pinning, Image Digests, Signing, Provenance, and Trusted Builders: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

Use the same causal order every time: preserve pipeline/job IDs and first-failure output → confirm CI_PIPELINE_SOURCE, ref and CI_COMMIT_SHA → inspect merged configuration and exact include/component refs → inspect job graph and runner/executor/image/tool identity → preserve artifact/report/provenance/signature files → independently hash the artifact → verify signer identity and provenance subject → inspect promoted registry/package/deployment digest → correct the smallest responsible layer → rerun only what must change.

2. Failure: base image tag changes

Symptom: two pipelines from the same application SHA produce different behavior. Both logs say builder:stable. The runner and source are healthy.

2026-09-10 builder:stable -> sha256:aaaaaaaa...
2026-09-12 builder:stable -> sha256:bbbbbbbb...
source_sha=0123456789abcdef... (same in both pipelines)

Diagnosis: inspect the pulled image digest or registry manifest digest. The mutable tag is the changed material. Correction: review the new builder image, then update a digest pin in a normal code review. Do not blame cache or rerun blindly.

3. Failure: remote include is compromised or simply changed

Symptom: pipeline creation fails before any jobs appear after an include:remote file changes.

include:
  - remote: https://example.invalid/ci/security.yml
    integrity: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=

If the actual content hash differs, the compilation layer rejects the include. Preserve the expected integrity value, the observed pipeline validation error and the reviewed source of the intended file. Do not remove integrity just to restore pipeline creation.

4. Failure: signature verified without expected identity

A verifier that accepts “any key whose signature is internally valid” answers only the cryptographic question. In a key-based system, pin the expected public-key fingerprint. In keyless Sigstore, constrain certificate identity and OIDC issuer. Otherwise an attacker can sign tampered bytes with their own valid identity and still pass a naive verification step.

BROKEN POLICY: signature cryptographically valid -> ALLOW
FIXED POLICY: signature valid
           AND certificate/public-key identity == expected project/builder
           AND issuer/trust root == expected
           AND subject digest == candidate digest -> ALLOW

5. Intentionally broken local example: provenance signature passes, artifact binding fails

Assume evidence/provenance.json was signed correctly, but someone replaces the artifact after signing. Verify the provenance signature first, then compare the subject digest:

cp dist/app.bundle dist/app.tampered
printf 'tamper\n' >> dist/app.tampered
openssl pkeyutl -verify -pubin -inkey evidence/builder.pub.pem -rawin   -in evidence/provenance.json -sigfile evidence/provenance.sig
python3 verify.py dist/app.tampered evidence/provenance.json   urn:devops-academy:ch32:local-openssl-builder

The first command succeeds because the provenance statement was not changed. The second fails because the candidate artifact digest no longer equals the signed statement's subject. Preserve both outcomes: they prove exactly which binding failed.

6. Failure: provenance is not bound to the artifact digest

Symptom: a provenance record lists source/build information but no candidate digest, or the consumer never compares the subject to the artifact. Impact: correct metadata can be attached to unrelated bytes. Correction: require a subject digest and enforce equality at promotion/deployment.

7. Failure: the builder itself is untrusted

A malicious or compromised builder can potentially emit an artifact and a self-consistent provenance file that both lie. This is why Chapter 31 matters. Ask who controls the provenance generation and signing identity. Prefer isolated ephemeral builders, minimal credentials, protected execution paths and platform-generated metadata for higher assurance. SLSA Level 3 focuses specifically on making provenance harder for user-controlled build steps to forge.

8. Causal layer matrix

9. Security-sensitive actions

• Never place long-lived private signing keys in repository files or ordinary job variables.
• Never log OIDC tokens, private keys, registry credentials or signature service tokens.
• Do not disable TLS or signature verification to diagnose network/signing failures.
• Do not allow untrusted fork code to run in a privileged trusted-builder pool.
• Do not use broad PATs just to fetch components or write attestations when narrower job/OIDC identities exist.
• Do not rebuild the release artifact as a troubleshooting shortcut; preserve the failed candidate and evidence.

10. Performance and availability tradeoffs

Verification adds network, cryptographic and metadata work. Control latency with cached immutable blobs, local transparency/log mirrors where supported, prebuilt pinned builder images and parallel verification of independent materials. Never “optimize” by turning off verification. Decide how the pipeline behaves when an external transparency/signing service is unavailable: fail closed for production promotion, or use a documented emergency process with evidence and later reconciliation.

11. If a trusted dependency or builder is compromised

Stop promotion first. Preserve the known-good and suspect digests, component/include SHAs, builder identity, affected pipeline/job IDs, signature bundles/keys/certificates, provenance and deployment records. Rotate/revoke affected signing identities as appropriate. Determine the affected time window, rebuild only after the builder trust boundary is restored, and create a new version/digest rather than silently replacing old evidence.

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-12. GitLab/GitLab Runner 19.3.2 is the current patched 19.3 baseline used for version notes in this chapter; Runner tag v19.3.2 was published 2026-09-10. Current GitLab documentation states that include:integrity is available on Free/Premium/Ultimate and rejects a remote include whose Base64-encoded SHA-256 does not match; cross-project includes should use a full 40-character commit SHA when stable immutability is required; CI/CD component consumers should prefer a commit SHA or trusted release version over moving selectors; and job/service images can use name@sha256:digest. Runner can emit in-toto/SLSA provenance metadata with RUNNER_GENERATE_ARTIFACTS_METADATA=true. GitLab's SLSA CI/CD components are available across tiers for signing/verifying Runner-generated provenance, while native SLSA Level 3 attestations and the Attestations API remain Ultimate experimental capabilities. GitLab.com keyless Sigstore signing is available across tiers; Self-Managed requires self-hosted Sigstore infrastructure. The mandatory lab below uses only local OpenSSL/Python/Git and creates no registry, cloud, package, or production side effects. The intentionally broken example is local and non-destructive. A failed digest/identity check is treated as evidence; the lesson never suggests weakening verification to force success.

• CI/CD YAML syntax — include, include:integrity and image digests — official reference.
• CI/CD includes — official reference.
• CI/CD components and version pinning — official reference.
• Run jobs in Docker containers — image checksums — official reference.
• GitLab Runner artifact provenance metadata — official reference.
• GitLab SLSA guidance — official reference.
• GitLab SLSA Level 3 attestations — official reference.
• GitLab Attestations API — official reference.
• GitLab Sigstore keyless signing examples — official reference.
• GitLab Self-Managed Sigstore integration — official reference.
• Sigstore Cosign installation — official reference.
• SLSA provenance specification — official reference.
• in-toto attestation framework — official reference.
• GitLab Runner tags/releases — official reference.
• GitLab 19.3.2 patch release — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.