Containerd Image Store, Snapshotters, Runtime Internals, OCI Runtime Specs, and Docker Engine Evolution: Guided Hands-On Workflow and Core Operations

1. Lab boundary and assumptions

The mandatory path works with ordinary Docker CLI access. It does not change daemon.json, does not enable embedded containerd, does not switch storage backends, and does not require a cloud registry. The image is alpine:3.22.1; record the resolved digest because the tag is only the human-readable input.

2. Preflight: prove which daemon you are about to inspect

set -eu
mkdir -p dca31-evidence

date -u +%Y-%m-%dT%H:%M:%SZ | tee dca31-evidence/time.txt
docker context show | tee dca31-evidence/context.txt
docker version | tee dca31-evidence/docker-version.txt
docker info | tee dca31-evidence/docker-info.txt

docker info --format '{{json .DriverStatus}}'   | tee dca31-evidence/driver-status.json

docker info --format 'driver={{.Driver}} root={{.DockerRootDir}} default-runtime={{.DefaultRuntime}}'   | tee dca31-evidence/engine-storage-runtime.txt

docker context show is not decoration: a remote context means filesystem paths and runtime processes belong to the daemon host, not necessarily to the shell where you typed the command. .Driver alone is insufficient for distinguishing every modern storage architecture, so preserve .DriverStatus and full docker info.

3. Inspect the remote multi-platform image before pulling

docker buildx imagetools inspect alpine:3.22.1   | tee dca31-evidence/alpine-index.txt

docker buildx imagetools inspect alpine:3.22.1 --raw   > dca31-evidence/alpine-index-raw.json

imagetools inspect reads registry-side OCI metadata. It can show an index and its platform manifests without first unpacking those layers into the local daemon’s snapshotter. This is the cleanest way to separate registry metadata from local image-store state.

4. Pull one platform and record local identity

# Record whether the tag already existed; cleanup later respects this marker.
if docker image inspect alpine:3.22.1 >/dev/null 2>&1; then
  printf 'preexisting
' > dca31-evidence/image-preflight.txt
else
  printf 'absent
' > dca31-evidence/image-preflight.txt
fi

docker pull alpine:3.22.1 | tee dca31-evidence/pull.txt

docker image inspect alpine:3.22.1   > dca31-evidence/local-image-inspect.json

docker image inspect alpine:3.22.1 \
  --format 'id={{.Id}} repoDigests={{json .RepoDigests}} os={{.Os}} arch={{.Architecture}}'   | tee dca31-evidence/local-image-summary.txt

A pull resolves the tag to a platform-specific image and stores content locally. With the containerd image store, Docker can also retain index/attestation structures that the classic local store could not represent. The local image ID, repository digest, index digest, and platform-manifest digest answer different questions; keep their labels with the evidence.

5. Create a container, but separate object creation from process start

docker create \
  --name dca31-runtime \
  --label devops-academy.lab=chapter31   alpine:3.22.1   sh -c 'echo runtime-start; sleep 120'   | tee dca31-evidence/container-id.txt

docker inspect dca31-runtime > dca31-evidence/container-created.json
docker inspect dca31-runtime \
  --format 'status={{.State.Status}} pid={{.State.Pid}} image={{.Image}} runtime={{.HostConfig.Runtime}}'   | tee dca31-evidence/container-before-start.txt

docker start dca31-runtime | tee dca31-evidence/start.txt
docker inspect dca31-runtime \
  --format 'status={{.State.Status}} pid={{.State.Pid}} image={{.Image}} runtime={{.HostConfig.Runtime}}'   | tee dca31-evidence/container-running.txt

docker create gives Docker a container object without a running task. docker start causes runtime work: an unpacked/rootfs snapshot must be usable, a task is created, the shim/runtime path executes, and the process receives a PID. This before/after pair is more educational than a single docker run.

6. Correlate process and container evidence

docker top dca31-runtime -eo pid,ppid,user,args   | tee dca31-evidence/docker-top.txt

docker logs --timestamps dca31-runtime   | tee dca31-evidence/container-logs.txt

docker inspect dca31-runtime \
  --format 'container={{.Id}} pid={{.State.Pid}} started={{.State.StartedAt}} graph={{json .GraphDriver}}'   | tee dca31-evidence/runtime-correlation.txt

The host PID is runtime evidence, not image identity. Likewise, the GraphDriver field on an individual inspect response should not be promoted into a universal statement about Engine 29 storage architecture. Use the daemon-level storage evidence from preflight.

7. Component versions: inspect what is actually installed

# These may be unavailable in Docker Desktop shells or minimal host installs.
containerd --version 2>/dev/null | tee dca31-evidence/containerd-cli-version.txt || true
runc --version 2>/dev/null | tee dca31-evidence/runc-version.txt || true

docker buildx version 2>/dev/null | tee dca31-evidence/buildx-version.txt || true
docker compose version 2>/dev/null | tee dca31-evidence/compose-version.txt || true

A containerd binary on PATH is not proof that you queried Docker’s managed containerd instance. Preserve Docker’s own version/info output as the authoritative Engine context. Component binaries are supplementary evidence.

8. Optional Linux-only: read-only inspection of Docker-managed containerd

Docker documents the managed/embedded endpoint primarily for debugging. On a default Linux Engine the endpoint is commonly /var/run/docker/containerd/containerd.sock and Docker stores its containers in the moby containerd namespace. Before using it, confirm that you are on the intended local Linux host and that your organization permits host-root diagnostics.

# OPTIONAL, LINUX-ONLY, READ-ONLY. Skip on Desktop/remote contexts.
sudo ctr --address /var/run/docker/containerd/containerd.sock   --namespace moby containers list

sudo ctr --address /var/run/docker/containerd/containerd.sock   --namespace moby tasks list

9. Snapshotter evidence without spelunking data directories

docker info --format '{{json .DriverStatus}}'   | tee dca31-evidence/driver-status-after-run.json

docker system df | tee dca31-evidence/system-df.txt

Prefer supported status/reporting APIs over traversing /var/lib/containerd. Files under that tree are implementation state. A production incident may justify read-only filesystem forensics, but normal training does not need root-level data-root access.

10. Small challenge: classify the layer before choosing the tool

Suppose the remote index lists linux/amd64 and linux/arm64, but your container fails after pull. Which evidence should you inspect first?

1. If the error says “no matching manifest,” stay at the registry/index/platform-selection layer.
2. If pull succeeds but unpack fails, inspect local storage/snapshotter evidence.
3. If the container object exists but start fails, inspect runtime/task/OCI configuration evidence.
4. If the process starts then exits, inspect application/process/log evidence.

The point is to choose the owning layer, not to run every Docker command you remember.

11. Verification and cleanup

docker inspect dca31-runtime   --format 'id={{.Id}} status={{.State.Status}} pid={{.State.Pid}} image={{.Image}}'   | tee dca31-evidence/final-container.txt

docker rm -f dca31-runtime

# Only remove the pulled tag when preflight proved it was absent before the lab.
if grep -qx absent dca31-evidence/image-preflight.txt; then
  docker image rm alpine:3.22.1 || true
fi

No daemon restart, storage-backend switch, prune, internal metadata deletion, or root filesystem cleanup is required.

12. Evidence packet

Knowledge check

Official references and version notes

Lab baseline: verified 2026-09-22 against Engine 29.8.1 documentation and containerd 2.3 concepts. The mandatory lab uses only Docker CLI/API behavior; optional ctr inspection is Linux-only and read-only.

• Docker Docs — containerd image store with Docker Engine — Engine 29 fresh-install defaults, snapshotters, disk layout, switching, and experimental migration guidance.
• Docker Docs — Storage drivers — distinction between classic graph drivers and the Engine 29 containerd image store.
• Docker Docs — Select a storage driver — current storage-backend matrix and platform notes.
• Docker Docs — Docker daemon configuration overview — /var/lib/docker versus /var/lib/containerd and data-root implications.
• Docker Docs — Run containerd in the Docker daemon — Engine 29.7+ experimental embedded-containerd mode and debugging endpoint warning.
• Docker Engine 29 release notes — Engine 29.8.1 baseline, containerd 2.3.5 static-binary packaging, BuildKit 0.33.0 and runc 1.5.1 updates.
• containerd 2.3 — Features — namespaces, images, root filesystems, snapshots, containers, tasks, and OCI runtime integration.
• containerd — Snapshotters — core snapshotter behavior and the overlayfs naming used by containerd.
• OCI Image Specification 1.1.1 — image indexes, manifests, configs, layers, and content-addressable descriptors.
• OCI Runtime Specification 1.3.0 — runtime bundle, config.json, execution environment, and lifecycle model.