Images, Layers, Content-Addressable Storage, Manifests, Config Objects, and Image Identity: Diagnostics, Failure Modes, Security, and Performance

1. Preserve identity before “fixing” the image

Image troubleshooting is unusually vulnerable to evidence destruction. A repull can move a tag to new content. A retag can hide what name originally resolved. A prune can remove the exact blobs needed to reproduce a failure. Therefore the first action is read-only capture: context, Engine version, storage backend, tag/reference, RepoDigests, local Image ID, platform, registry index/manifest output, container image fields, and the exact error.

2. Evidence-first diagnostic sequence

1. Confirm host/platform, active context, client/server versions, and local image store behavior.
2. Record the exact input reference and registry-side top-level digest/media type.
3. Record the selected platform manifest and local config/Image ID.
4. Inspect layer descriptors/DiffIDs and pull/unpack errors if transfer failed.
5. Inspect the container object's .Config.Image and .Image if runtime behavior is involved.
6. Only then test one hypothesis with the least destructive correction.
7. Re-run the exact failed operation and compare identities before/after.

3. Intentionally broken example: using the Image ID as a registry digest

This is a safe identity error to reproduce with the Chapter 04 lab image. First ensure the versioned image exists locally and capture two different values:

SOURCE='busybox:1.37.0'
docker pull "$SOURCE"

IMAGE_ID=$(docker image inspect "$SOURCE" --format '{{.Id}}')
REPO_DIGEST=$(docker image inspect "$SOURCE" --format '{{index .RepoDigests 0}}')

printf 'Image ID:    %s\n' "$IMAGE_ID"
printf 'RepoDigest:  %s\n' "$REPO_DIGEST"

Now derive the repository name and deliberately attempt to use the config/Image ID as though it were a registry manifest digest:

REPO='busybox'
WRONG_DIGEST=${IMAGE_ID#sha256:}

docker pull "${REPO}@sha256:${WRONG_DIGEST}"

On a normal registry this should fail because the config blob digest is not the digest of a pullable image manifest/index. Preserve the exact registry error. The correction is not “try random hashes”; it is to use the repository-qualified RepoDigest captured from the registry relationship:

docker pull "$REPO_DIGEST"

The diagnosis is semantic: both values are SHA-256 digests, but they name different object types.

4. Failure mode: the tag moved

Symptom: yesterday and today both say app:stable, yet hosts run different bytes. Do not compare tag strings. Compare the digest resolved at each timestamp. If historical digest evidence is missing, the ambiguity cannot be repaired after the fact by inspecting today's tag.

Correction: record the digest during build/promotion/deploy, and pin production to an approved digest or at minimum verify that the resolved digest equals the approved one before rollout.

5. Failure mode: correct release, wrong or missing platform

Symptom: the tag resolves, but the host reports no matching manifest for its platform, or an emulated/incorrect variant is selected. Inspect the image index and host platform rather than blaming the container process.

docker info --format 'OS={{.OSType}} Arch={{.Architecture}}'
docker buildx imagetools inspect busybox:1.37.0

Check whether the index actually contains the required OS/architecture/variant descriptor. A manifest existing for amd64 does not prove arm64 support.

6. Failure mode: treating history or sizes as exact blob accounting

docker image history is useful narrative evidence, but history rows and filesystem layers are not one-to-one. Metadata-only steps can report 0B and have no rootfs layer. Docker's displayed image size is generally uncompressed logical image data, while transfer sizes in manifests are compressed blob sizes. Shared blobs also mean summing image sizes across tags overstates unique disk consumption.

Correction: use manifest descriptors for transferred blob sizes, Docker disk-accounting commands for local usage, and history for build-step narrative. State which quantity you are measuring.

7. Failure mode: deleting shared content to “fix corruption”

Deleting files under Docker's data root is not targeted image repair. Content, snapshots, metadata, containers, and images can share state. Manual deletion can transform one understandable image failure into daemon-wide corruption.

Correction: preserve daemon logs and object IDs, verify the exact affected reference, use Docker-supported remove/pull operations on a disposable scope when appropriate, and escalate persistent store corruption with a backup/recovery plan. Never teach rm -rf /var/lib/docker as troubleshooting.

8. Failure mode: assuming every Engine 29 host has the same store

A fresh Engine 29 installation normally uses the containerd image store. An upgraded older host may still use classic storage-driver state, and userns-remap changes the supported path. If one host can retain a multi-platform image locally and another cannot, inspect their store/configuration history before calling one “broken.”

9. Security: digest integrity is necessary but not sufficient

Successfully pulling by digest verifies that the fetched content matches that digest. It does not establish who authorized the digest, whether the publisher's process was trustworthy, or whether the image contains vulnerabilities. Later chapters bind SBOMs, scans, provenance, and signatures to exact digests. Do not weaken TLS or trust settings to make a digest pull “work.”

10. Performance: shared layers, transfer, and local reuse

When multiple images reference the same content-addressed layers, Docker can reuse already-present blobs. That reduces transfer and disk duplication. Performance diagnosis should therefore distinguish network transfer bytes, compressed layer sizes, uncompressed image size, snapshot usage, and total Docker data-root consumption. Optimize the measured bottleneck rather than deleting caches reflexively.

11. Repair checklist

Knowledge check

Official references and version notes

• Docker: What is an image? — image/layer fundamentals and local image inspection.
• docker image pull — content-addressable image storage, shared layers, pull digests, and pulling by digest.
• docker image inspect — local image metadata, IDs, RepoDigests, rootfs data, and platform-aware inspection.
• docker image ls — repository/tag presentation, image IDs, and digest display.
• docker image history — build-history presentation and platform selection.
• docker buildx imagetools inspect — registry-side image index/manifest inspection without assuming the local store contains every platform.
• containerd image store with Docker Engine — Engine 29 fresh-install default, snapshotters, multi-platform local storage, and upgrade/userns-remap caveats.
• Docker storage drivers — layer sharing, copy-on-write context, history examples, and classic storage-driver behavior.
• OCI Image Specification v1.1.1 — current stable OCI image-format release used as the chapter's standards baseline.
• OCI image manifest — config descriptor, ordered layer descriptors, media types, and single-image/platform semantics.
• OCI image index — higher-level descriptor set used for multi-platform publication.
• OCI image configuration — runtime configuration, rootfs DiffIDs, history, and ImageID-related content.
• OCI content descriptors — media type, digest, size, platform, and content-verification semantics.