Docker in CI/CD: Reproducible Builds, Buildx, Registry Caching, Ephemeral Runners, and Release Promotion: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first incident sequence

1. Freeze the failed run ID, source SHA, workflow definition and timestamps.
2. Capture runner/context, Docker, Buildx and builder evidence.
3. Capture build metadata, logs, cache refs and image/index digest.
4. Inspect registry resolution for every tag/alias involved.
5. Inspect attestation/signature/test records bound to the subject digest.
6. Inspect deployment/runtime digest independently.
7. Only then change the smallest relevant layer and rerun the narrowest safe scope.

2. Failure: publishing only latest

Symptom: two incident responders pull latest at different times and discuss different bytes. Cause: the pipeline discarded immutable identity. Repair: capture the digest immediately after publication, store it as a pipeline output/evidence artifact, and treat latest only as a mutable convenience alias.

3. Failure: rebuilding at deploy time

Symptom: staging passed, production failed, even though both jobs used the same Git tag. Evidence shows two different image digests. Cause: base tags, package repositories, timestamps or build tooling changed between builds. Repair: deploy/promote the staging-tested digest. If a rebuild is required, treat it as a new release candidate.

4. Failure: cache poisoning across forks

BROKEN POLICY EXAMPLE
trusted-main writes cache ref: org/app:buildcache
fork PR receives same registry token
fork PR writes same cache ref
trusted-main later imports cache from org/app:buildcache

Evidence to preserve:
- actor/fork identity
- token scope
- cache ref + timestamps
- BuildKit logs showing import/export
- resulting digest and source SHA

Repair by separating write scopes. A trusted job may optionally read a cache populated by trusted history, while untrusted jobs write nowhere or to an isolated disposable cache namespace.

5. Failure: privileged shared runners

Symptom: one repository can see or affect another repository’s Docker state or credentials. This is not a Buildx performance issue; it is a runner/daemon isolation failure. Chapter 35’s rule applies: untrusted jobs must not receive a reusable host daemon authority. Move them to ephemeral or isolated build workers.

6. Failure: secret in build arguments

Symptom: a credential appears in image history, provenance, logs, or build metadata. Preserve the leak location and rotate/revoke the credential before editing history. Repair the Dockerfile/workflow to use BuildKit secret or SSH mounts, then rebuild a new digest and rescan all evidence surfaces.

7. Failure: race-prone mutable tags

Two jobs push the same branch tag. A downstream job pulls after the second push but assumes it received the first job’s result. Repair by passing digests between jobs, not tags. Tags may be updated after digest verification, but consumers should verify the resolved digest against the expected pipeline output.

8. Failure: build success mistaken for registry success

A build can complete while push, attestation publication, promotion or downstream pull later fails. Treat each transition as separate evidence: build metadata → push response → registry digest inspection → alias inspection → runtime pull/deploy verification.

9. Intentionally broken digest check and repair

EXPECTED='sha256:1111111111111111111111111111111111111111111111111111111111111111'
ACTUAL=$(docker image inspect dca36-app:ci --format '{{.Id}}' 2>/dev/null || true)
printf 'expected=%s
actual=%s
' "$EXPECTED" "$ACTUAL"
[ "$EXPECTED" = "$ACTUAL" ] || printf 'FAIL: subject identity mismatch; do not promote.
'

The intended failure is the mismatch itself. Do not “repair” it by changing the expected value until you know which artifact the tests actually evaluated. Reconstruct the chain from source SHA and build metadata.

10. Diagnostic decision table

11. Least-destructive corrections

• Pin expected digest in downstream jobs.
• Separate cache write scopes.
• Rotate leaked credentials and rebuild once.
• Move untrusted jobs to isolated runners/builders.
• Re-run only the failed verification stage when artifact identity is unchanged.
• Do not clear all caches or prune the daemon as a generic diagnostic step.

12. Preserve the negative evidence

A failed digest comparison, missing attestation, denied registry push, or mismatched runtime digest is useful evidence. Store it with timestamps and run ID. “Fixed by rerun” without preserving the original discrepancy weakens root-cause analysis.

Knowledge check

Official references and version notes

Diagnostic baseline: 2026-09-22. Current Buildx/BuildKit make digest, metadata and attestation evidence easier to export, but pipeline correctness still depends on preserving identities between stages and on provider-specific runner/credential isolation.

• Docker Docs — Build with CI — CI patterns around Buildx/BuildKit and provider integrations.
• Docker Docs — Cache storage backends — inline, local, registry, and GitHub Actions cache backends and scope considerations.
• Docker Docs — Registry cache — external registry cache configuration and separation from the release image.
• Docker Docs — docker buildx build — metadata files, cache import/export, push, SBOM and provenance controls.
• Docker Docs — Build attestations — default provenance and SBOM/provenance persistence rules.
• Docker Docs — GitHub Actions attestations — current Docker-maintained action behavior and provenance/SBOM inputs.
• Docker Docs — GitHub Actions cache management — provider cache integration and limits.
• Docker Docs — docker buildx imagetools create — create/retag registry manifests without rebuilding.
• Docker Docs — Builders — builder instances, drivers, workers, and lifecycle.
• Docker Buildx releases — version source; v0.37.1 is current at this chapter baseline.
• Moby BuildKit releases — version source; v0.33.0 is current at this chapter baseline.
• docker/build-push-action releases — v7.4.0 current at this baseline.
• docker/setup-buildx-action releases — v4.3.0 current at this baseline.
• Docker Engine 29 release notes — Engine 29.8.1 baseline and bundled/runtime changes.