Storage Drivers, overlay2, Copy-on-Write Behavior, Disk Usage, Pruning, and Storage Troubleshooting: Guided Hands-On Workflow and Core Operations

1. Lab topology and safety boundary

The mandatory lab is local/free and does not require root, daemon changes, cloud services, or direct filesystem access to Docker internals. It creates a small synthetic image, one named volume, one stopped container with a controlled writable-layer file, a second short-lived volume writer, bounded local logs, and a dedicated Buildx builder so cache ownership is isolated.

2. Preflight and collision guards

set -eu
LAB=dca32-lab
EVIDENCE="$LAB/evidence"
mkdir -p "$EVIDENCE"

date -u +%Y-%m-%dT%H:%M:%SZ | tee "$EVIDENCE/time.txt"
docker context show | tee "$EVIDENCE/context.txt"
docker version | tee "$EVIDENCE/docker-version.txt"
docker info | tee "$EVIDENCE/docker-info.txt"
docker system df | tee "$EVIDENCE/system-df-before.txt"
docker buildx ls | tee "$EVIDENCE/builders-before.txt"

for c in dca32-writable dca32-volume-writer; do
  if docker container inspect "$c" >/dev/null 2>&1; then
    echo "Refusing to overwrite preexisting container $c" >&2; exit 1
  fi
done
if docker volume inspect dca32-data >/dev/null 2>&1; then
  echo 'Refusing to overwrite preexisting volume dca32-data' >&2; exit 1
fi
if docker image inspect dca32-image:lab >/dev/null 2>&1; then
  echo 'Refusing to overwrite preexisting image dca32-image:lab' >&2; exit 1
fi
if docker buildx inspect dca32-builder >/dev/null 2>&1; then
  echo 'Refusing to overwrite preexisting builder dca32-builder' >&2; exit 1
fi

The guards turn cleanup ownership into a proven fact rather than an assumption. A name collision aborts before mutation.

3. Record predictions before creating bytes

cat > "$EVIDENCE/predictions.md" <<'EOF'
# Predictions
1. Writing a 6 MiB file inside /scratch without a mount will increase the container writable-layer size.
2. Writing a 5 MiB file into dca32-data will increase volume usage but should not appear as the container writable-layer file itself.
3. A dedicated docker-container Buildx builder can own cache independently from unrelated builders.
4. Removing only the lab container/image/volume/builder will leave unrelated Docker objects untouched.
EOF
cat "$EVIDENCE/predictions.md"

4. Create the synthetic build context

mkdir -p "$LAB/context"
cat > "$LAB/context/Dockerfile" <<'EOF'
# syntax=docker/dockerfile:1
FROM alpine:3.22.1
LABEL devops-academy.lab=chapter32
RUN dd if=/dev/zero of=/opt/build-payload.bin bs=1M count=3  && sha256sum /opt/build-payload.bin > /opt/build-payload.sha256
CMD ["sh", "-c", "cat /opt/build-payload.sha256; sleep 300"]
EOF

The 3 MiB payload is intentionally small but visible in image/build accounting. It is synthetic data, so deleting the lab later carries no business risk.

5. Create a dedicated builder and inspect zero-state cache

docker buildx create   --name dca32-builder   --driver docker-container   --use | tee "$EVIDENCE/builder-create.txt"

docker buildx inspect --bootstrap | tee "$EVIDENCE/builder-inspect.txt"
docker buildx du | tee "$EVIDENCE/buildx-du-before.txt"

A docker-container builder keeps its BuildKit state distinct from unrelated builders. This makes later cleanup bounded: removing the builder removes this training builder rather than pruning an unknown shared cache.

6. Build, load, and capture image evidence

docker buildx build   --builder dca32-builder   --load   --tag dca32-image:lab   --progress=plain   "$LAB/context" | tee "$EVIDENCE/build.txt"

docker image inspect dca32-image:lab > "$EVIDENCE/image-inspect.json"
docker image inspect dca32-image:lab   --format 'id={{.Id}} size={{.Size}} labels={{json .Config.Labels}}'   | tee "$EVIDENCE/image-summary.txt"
docker buildx du | tee "$EVIDENCE/buildx-du-after-build.txt"

--load makes the single-platform build available to the selected Docker Engine. The builder may retain build cache in addition to the loaded image; these are related but distinct lifecycles.

7. Create known writable-layer usage

docker create \
  --name dca32-writable \
  --label devops-academy.lab=chapter32 \
  --log-driver local \
  --log-opt max-size=1m \
  --log-opt max-file=2   dca32-image:lab   sh -c 'dd if=/dev/zero of=/scratch.bin bs=1M count=6; echo lab-log-line; sleep 300'

docker start dca32-writable
docker logs dca32-writable | tee "$EVIDENCE/writable-logs.txt"
docker ps -a --size --filter name=dca32-writable   | tee "$EVIDENCE/writable-size.txt"
docker inspect dca32-writable   --format 'log={{json .HostConfig.LogConfig}} mounts={{json .Mounts}}'   | tee "$EVIDENCE/writable-inspect.txt"

The file is written into the container’s copy-on-write layer because no mount covers /scratch.bin. The per-container local logging options bound log storage without changing daemon-wide defaults.

8. Put different bytes in a named volume

docker volume create   --label devops-academy.lab=chapter32   dca32-data | tee "$EVIDENCE/volume-create.txt"

docker run \
  --rm \
  --name dca32-volume-writer \
  --mount type=volume,src=dca32-data,dst=/data   alpine:3.22.1   sh -c 'dd if=/dev/zero of=/data/volume.bin bs=1M count=5; sha256sum /data/volume.bin'

docker volume inspect dca32-data > "$EVIDENCE/volume-inspect.json"
docker run \
  --rm \
  --mount type=volume,src=dca32-data,dst=/data,readonly   alpine:3.22.1   sh -c 'ls -lh /data/volume.bin; sha256sum /data/volume.bin'   | tee "$EVIDENCE/volume-verify.txt"

The second workload proves volume persistence independently of the first container’s writable layer. On some platforms docker system df -v can estimate volume usage; the checksum proves data identity regardless of how the platform reports bytes.

9. Inventory every storage class before cleanup

docker system df | tee "$EVIDENCE/system-df-populated.txt"
docker system df -v | tee "$EVIDENCE/system-df-populated-verbose.txt"
docker ps -a --size --filter label=devops-academy.lab=chapter32   | tee "$EVIDENCE/lab-containers-size.txt"
docker image ls --filter label=devops-academy.lab=chapter32   | tee "$EVIDENCE/lab-images.txt"
docker volume ls --filter label=devops-academy.lab=chapter32   | tee "$EVIDENCE/lab-volumes.txt"
docker buildx du --builder dca32-builder   | tee "$EVIDENCE/lab-build-cache.txt"

This is the “dry inventory.” No prune command has run. Compare Docker-wide totals with exact label/name lists before deciding anything is disposable.

10. Understand prune filters without using broad cleanup

Prune filters are useful in governed automation, but previewability differs across commands and negative-label filters can be especially hard to predict. This course’s mandatory path chooses exact resource deletion because lab ownership is stronger evidence than a broad filter.

11. Stop first, then remove exact resources

docker stop dca32-writable
docker inspect dca32-writable   --format 'status={{.State.Status}} sizeRw={{.SizeRw}}'   --size | tee "$EVIDENCE/writable-before-delete.txt"

docker rm dca32-writable
docker volume rm dca32-data
docker image rm dca32-image:lab
docker buildx rm dca32-builder

Each command names exactly one lab-owned object. No unrelated stopped container, image, volume, network, or build cache becomes eligible merely because it is old or unused.

12. Remeasure and preserve the evidence packet

docker system df | tee "$EVIDENCE/system-df-after.txt"
docker buildx ls | tee "$EVIDENCE/builders-after.txt"

docker ps -a --filter label=devops-academy.lab=chapter32
docker image ls --filter label=devops-academy.lab=chapter32
docker volume ls --filter label=devops-academy.lab=chapter32

Expected result: exact lab objects are gone, Docker-wide totals may fall, and all unrelated objects remain. The exact byte delta can differ because shared blobs, metadata, filesystem allocation units, compression, delayed filesystem reclamation, and builder cleanup affect physical accounting.

13. Small challenge: choose the owning layer

A developer reports that a container is only 8 KB according to docker ps --size, yet the Docker host is losing hundreds of megabytes each hour. The container emits verbose logs and mounts a named database volume. Before deleting anything, choose the next two evidence sources.

Target reasoning: inspect the container logging driver/options and Docker-wide/volume usage. Writable-layer evidence has already made one hypothesis less likely; it has not measured logging-driver files or persistent volume data.

Knowledge check

Official references and version notes

Lab baseline: all writes are synthetic and bounded. The lab intentionally uses per-container rotating local logs and a dedicated builder to avoid daemon mutation and unrelated cache deletion.

• Docker Docs — containerd image store with Docker Engine — Engine 29 fresh-install default, snapshotter reporting, switching, and migration boundaries.
• Docker Docs — Storage drivers — image layers, container writable layers, copy-on-write, and container-size accounting caveats.
• Docker Docs — OverlayFS storage driver — classic overlay2, backing-filesystem requirements, lower/upper/merged/workdir semantics, and migration cautions.
• Docker Docs — Select a storage driver — current containerd snapshotter versus classic-driver guidance and platform distinctions.
• Docker CLI — docker system df — daemon disk-usage summary and verbose object-level accounting.
• Docker CLI — docker system prune — deletion scope and supported filters; referenced here to understand risk, not as the lab cleanup path.
• Docker Buildx — buildx du — builder-cache usage, shared/private/reclaimable records, and detailed evidence.
• Docker Buildx — buildx prune — cache filters and space-budget controls.
• Docker Build — Build garbage collection — periodic BuildKit GC, default policy shape, and builder-specific configuration.
• Docker Docs — Docker daemon configuration overview — Docker data-root and the separate managed containerd root used by the Engine 29 containerd image store.
• Docker Docs — Configure logging drivers — json-file growth risk, rotation guidance, and the rotating local driver.
• Docker Engine 29 release notes — current Engine 29 behavior and storage/image-store fixes.