Image Tags, Digests, Mutable vs Immutable References, Pull Policies, Pinning, and Reproducibility: Diagnostics, Failure Modes, Security, and Performance

1. Failure model: “wrong image” has several causal layers

When two hosts run different behavior from the same tag, do not start by deleting images. First ask which context each client addressed, what local object each container uses, whether the hosts pulled at different times, what the registry tag resolves to now, and whether a multi-platform index selected different child manifests. Each answer belongs to a different state layer.

2. Evidence-first diagnostic sequence

1. Record host, architecture, docker version, context, and current time.
2. Inspect the affected container: requested image string, local image ID, creation time, labels, and platform.
3. Inspect the local image: ID, RepoTags, RepoDigests, OS/architecture, created/config metadata.
4. Resolve the registry reference without overwriting evidence where possible.
5. For multi-platform references, identify index and child manifest digests.
6. Only then decide whether the fix is pull, pin, retag, redeploy, or release-policy correction.

3. Preserve the original identities

docker context show
docker version

docker inspect affected-container --format \
'name={{.Name}} requested={{.Config.Image}} localImageID={{.Image}} created={{.Created}}'

docker image inspect IMAGE_ID --format \
'ID={{.Id}} tags={{json .RepoTags}} digests={{json .RepoDigests}} os={{.Os}} arch={{.Architecture}} created={{.Created}}'

docker buildx imagetools inspect registry.example/team/app:stable

Do not run an always-pull command until this baseline is saved. Pulling may change the local tag mapping and make the original stale-cache condition harder to prove.

4. Intentionally broken example: two hosts, one tag, two releases

Host A can reuse its cached content while Host B pulls B. Both deployment configs still say app:stable. The fix is not “restart Docker.” Capture both image identities, verify current registry resolution, then decide whether both should pin A, both should move to approved B, or the release alias was moved incorrectly.

5. Failure matrix

6. Container name and tag are both mutable human labels

A container name can be removed and reused for a different container ID. An image tag can move to a different digest. During an incident, preserve immutable IDs and timestamps in addition to names. Human labels are for navigation; IDs and digests are for correlation.

7. Digest pinning can preserve vulnerable bytes perfectly

Reproducibility and security maintenance are separate concerns. A digest pin protects against unexpected drift, but it also prevents automatic drift toward a patched base. A mature workflow periodically detects a newer approved base, proposes the digest change, rebuilds, retests, rescans, and records a new release identity.

8. Troubleshooting shortcuts to reject

• Do not delete all local images or run broad prune commands before recording IDs/digests.
• Do not rebuild the release merely to see whether the problem disappears.
• Do not overwrite a production release tag to “put it back” without first preserving both old and current resolutions.
• Do not switch to latest or always-pull as a substitute for deciding which digest is approved.
• Do not disable registry TLS or weaken daemon trust settings to inspect an unrelated endpoint.

9. Performance: freshness has a cost, but identity comes first

Always pulling a mutable tag adds registry latency and bandwidth, while missing can reuse local bytes. Time-based Compose policies balance freshness and traffic for developer workflows. Production optimization should avoid repeated downloads by staging known digests, not by sacrificing release identity. Measure transfer sizes and pull latency separately from correctness.

10. Recovery pattern

Once evidence identifies the approved digest, create a new container or deployment from that digest. Verify the container’s requested reference and local image ID, application health, and external behavior. Preserve the incident note explaining why the mutable alias differed. Then fix the promotion/pull policy so the same ambiguity cannot recur.

Knowledge check

Official references and version notes

• docker image ls — digest display, local image IDs, and digest-addressed pull/create/run examples.
• docker image inspect — low-level local image metadata, including platform-aware inspection on capable stores.
• docker run — current --pull=missing|always|never semantics.
• Compose pull_policy — always, never, missing, build, daily, weekly, and every_<duration>.
• Docker build best practices — pinning base images by digest, controlled updates, and auditability tradeoffs.
• docker buildx imagetools inspect — registry-side manifest/index and platform inspection.
• Image digests — manifest-list/index versus per-platform image digest distinctions.
• OCI Image Index Specification — immutable index descriptors and platform selection.
• OCI Image Manifest Specification — config and layer descriptors addressed by digest.
• Docker Engine 29 release notes — Engine 29.8.1 baseline used when authoring this chapter.
• Buildx releases and BuildKit releases — current build-tool release history.