Build Cache, Cache Mounts, Bind Mounts, Secret Mounts, SSH Mounts, and Cache Import/Export: Diagnostics, Failure Modes, Security, and Performance

1. Diagnostic method: preserve cache evidence before changing cache

Cache problems are often “fixed” by deleting all cache, which destroys the evidence needed to explain the failure and may slow unrelated teams. Start with the smallest observable chain: builder/driver → Dockerfile/frontend → exact input hashes and build args → plain progress → cache source → executed/cached vertex → output digest. Change one hypothesis at a time.

2. Failure mode: stale dependency assumptions

An ordinary RUN cache key does not include “the remote package repository changed.” If a dependency-install command and its declared inputs are unchanged, BuildKit can reuse the previous result. This is correct cache behavior, not corruption. The real design question is whether your build declared the freshness/identity policy you intended.

Diagnosis: preserve the plain log showing CACHED; record the lockfile, base digest, package index/source, and expected dependency identity. Correction: update the declared dependency lock/version/base or intentionally invalidate the relevant stage. Avoid a blanket --no-cache policy as a permanent substitute for dependency identity—it throws away useful deterministic reuse without telling you what changed.

3. Failure mode: changed secret, cached secret-dependent output

Docker documents that secret contents are not included in the build-cache checksum. This protects secret values from becoming cache-key material, but it creates a subtle failure if a RUN step writes non-secret output that depends on the secret value. With fake data, demonstrate the behavior on an isolated builder:

# syntax=docker/dockerfile:1
FROM alpine:3.22
ARG CACHEBUST=0
RUN --mount=type=secret,id=demo,required=true \
    printf 'cachebust=%s\n' "$CACHEBUST" > /result.txt && \
    sha256sum /run/secrets/demo >> /result.txt
CMD ["cat", "/result.txt"]

mkdir -p secret-cache-demo
cd secret-cache-demo
printf '%s\n' 'fake-secret-A' > secret.txt

docker buildx build --progress=plain \
  --secret id=demo,src=secret.txt --load -t ch11-secret-demo:a . \
  2>&1 | tee ../evidence/diag-secret-a.txt

docker run --rm ch11-secret-demo:a \
  | tee ../evidence/diag-secret-a-result.txt

printf '%s\n' 'fake-secret-B' > secret.txt

docker buildx build --progress=plain \
  --secret id=demo,src=secret.txt --load -t ch11-secret-demo:b . \
  2>&1 | tee ../evidence/diag-secret-b.txt

docker run --rm ch11-secret-demo:b \
  | tee ../evidence/diag-secret-b-result.txt

If the RUN vertex remains cached, image b can contain the previous fake-secret hash even though the supplied secret changed. Preserve that surprising evidence. Then rebuild with --build-arg CACHEBUST=1 and verify the vertex executes. The build argument is not secret; it is an explicit invalidation signal. In production, prefer immutable remote artifact/version inputs so the cache key reflects what output you actually expect.

4. Failure mode: cache poisoning across trust domains

Suppose an untrusted pull request can change a Dockerfile and export cache to a shared reference that release jobs import. Even if BuildKit only reuses entries whose keys match, allowing an untrusted job to become a cache writer broadens the release pipeline’s input trust. The release job should not need to reason about attacker-controlled optimization metadata under privileged credentials.

Fix the policy, not just the symptom: untrusted jobs write to isolated cache namespaces or no shared cache at all; trusted jobs import from controlled sources; write credentials are scoped narrowly; cache references are not the same as release image references. Record who wrote the cache and under what source revision.

5. Failure mode: concurrent cache-mount corruption or lock contention

A tool that uses a database-like cache can misbehave when two builds write concurrently with sharing=shared. Symptoms may include corrupted metadata, intermittent checksum errors, partial package indexes, or flakiness that disappears on a quiet runner. Before blaming Docker, inspect the tool’s concurrency guarantees and compare with the mount sharing mode.

Correction options: sharing=locked for serialization, private for per-writer isolation, or project/architecture-specific cache IDs. Measure wait time and disk cost. A locked cache that serializes a large CI fleet may become a performance bottleneck even though it is correct.

6. Failure mode: credentials baked into layers or external cache

If a Dockerfile uses ARG TOKEN, ENV TOKEN, or COPY credentials, the problem is not “the cache backend leaked it.” The secret entered persistent build/image state before export. An external cache can then widen exposure. Preserve the Dockerfile/history/provenance evidence, revoke the real credential outside the course, and rebuild from clean declared inputs using secret/SSH mounts. Deleting a later file or tag does not prove earlier blobs/cache records disappeared.

For this course, never test with a real secret. Search generated image history/configuration and exported cache metadata using only the known fake marker.

7. Failure mode: “my cache disappeared” after switching builders

Internal cache belongs to a BuildKit instance. If docker buildx ls shows a different selected builder, a cache miss may be completely expected. Compare builder name, driver, BuildKit version, endpoint, and worker platform before deleting anything. If portability is needed, configure external cache export/import explicitly.

8. Failure mode: external backend unsupported by the selected driver

Current Docker documentation notes that cache backend availability depends on builder driver and, for the default docker driver, containerd image-store configuration. If a --cache-to backend fails, first confirm the selected builder and documented backend support. Do not enable insecure entitlements or reconfigure production daemon storage just to make a training cache backend work; use an isolated docker-container builder or the local backend instead.

9. Performance diagnosis: measure the expensive vertex

Wall-clock time is useful but incomplete. Capture plain progress and identify the vertices consuming time. A five-minute build may spend four minutes transferring an oversized context, downloading packages, compiling, exporting/pushing, or waiting on a locked cache. Each bottleneck has a different correction. Chapter 08 handles context size; this chapter handles reuse/mounts; later chapters handle registry and multi-platform costs.

10. Evidence-first diagnostic sequence

1. Preserve first failure: command, plain log, timestamps, builder name, source revision, output/error.
2. Confirm Docker context, Engine, Buildx, selected builder, driver, BuildKit, frontend, worker platform.
3. Hash Dockerfile, lockfiles, context inputs, and record base references/digests.
4. Identify whether the vertex was cached or executed.
5. If executed, identify bind/cache/secret/SSH mounts and their IDs/sharing.
6. If external cache is involved, record import/export type, ref/path, writer trust domain, and errors.
7. Apply one narrow correction: declared input change, sharing-mode correction, cache namespace change, explicit invalidation, or backend/driver correction.
8. Rebuild and compare output digest/test behavior. Do not stop at “it got faster.”

11. Diagnostic cleanup

docker image rm ch11-secret-demo:a ch11-secret-demo:b 2>/dev/null || true
rm -rf secret-cache-demo

Remove only diagnostic objects you created. If you used a dedicated disposable builder for a separate experiment, remove that named builder after preserving logs. Do not clear unrelated cache as a routine conclusion.

Knowledge check

Official references and version notes

• Docker build cache — current cache model, optimization guidance, invalidation, and storage backends.
• Build cache invalidation — instruction matching, COPY/ADD/bind-input checksums, RUN behavior, and the rule that secret contents do not invalidate cache.
• Optimize cache usage — instruction ordering, small contexts, build bind mounts, cache mounts, and external cache patterns.
• Cache storage backends — inline, registry, local, GHA, and availability notes for other backends.
• Local cache backend — OCI-layout local cache export/import used by the mandatory checkpoint.
• Registry cache backend — separate cache artifacts, min/max modes, and driver/containerd-image-store requirements.
• RUN --mount reference — bind, cache, tmpfs, secret, and SSH mount semantics and options.
• Build secrets — passing and consuming secret/SSH mounts without baking credentials into image layers.
• docker buildx build — --cache-from, --cache-to, --secret, --ssh, progress, metadata, and output behavior.
• Build drivers — driver capabilities that affect cache backend availability and builder isolation.
• Docker Engine 29 release notes — current Engine baseline and bundled component updates.
• Buildx releases — current Buildx release history.
• BuildKit releases — current BuildKit and built-in Dockerfile frontend release history.