Checkpoint Lab — Storage Drivers, overlay2, Copy-on-Write Behavior, Disk Usage, Pruning, and Storage Troubleshooting

1. Checkpoint scenario

You will create four controlled storage sources: a synthetic image, a stopped container with a writable-layer payload, a disposable named volume, and a dedicated builder cache. A second named volume is marked protected and contains a checksum marker. You will inventory all resources, predict cleanup results, delete only the disposable set, prove the protected marker remains, and only then remove the final protected training volume as an explicit end-of-lab rollback.

2. Version/platform assumptions to record

3. Preflight and exact-name guards

set -eu
LAB=dca32-checkpoint
EVIDENCE="$LAB/evidence"
mkdir -p "$EVIDENCE/context"

date -u +%Y-%m-%dT%H:%M:%SZ | tee "$EVIDENCE/time.txt"
docker context show | tee "$EVIDENCE/context-name.txt"
docker version | tee "$EVIDENCE/docker-version.txt"
docker info | tee "$EVIDENCE/docker-info.txt"
docker buildx version 2>/dev/null | tee "$EVIDENCE/buildx-version.txt" || true
docker compose version 2>/dev/null | tee "$EVIDENCE/compose-version.txt" || true
docker system df | tee "$EVIDENCE/system-df-before.txt"
docker system df -v | tee "$EVIDENCE/system-df-before-verbose.txt"

for c in dca32-cp-writable; do
  if docker container inspect "$c" >/dev/null 2>&1; then echo "collision: $c" >&2; exit 1; fi
done
for v in dca32-cp-data dca32-cp-protected; do
  if docker volume inspect "$v" >/dev/null 2>&1; then echo "collision: $v" >&2; exit 1; fi
done
if docker image inspect dca32-cp-image:lab >/dev/null 2>&1; then echo 'image collision' >&2; exit 1; fi
if docker buildx inspect dca32-cp-builder >/dev/null 2>&1; then echo 'builder collision' >&2; exit 1; fi

If any guard fires, choose different training names. Do not delete the preexisting object merely to make the lab fit.

4. Storage architecture evidence

docker info --format 'root={{.DockerRootDir}} driver={{.Driver}} logging={{.LoggingDriver}}'   | tee "$EVIDENCE/storage-summary.txt"
docker info --format '{{json .DriverStatus}}'   | tee "$EVIDENCE/driver-status.json"
docker buildx ls | tee "$EVIDENCE/builders-before.txt"

Label the result: containerd snapshotter-backed, classic overlay2, Desktop-managed, or another supported backend. Do not force the host to match the course baseline.

5. Write predictions before creating storage

cat > "$EVIDENCE/predictions.md" <<'EOF'
# Predictions
1. The writable container will add roughly 7 MiB of private writable-layer data, visible in docker ps --size.
2. The disposable and protected volumes will persist independently of the writable container lifecycle.
3. The dedicated builder will own cache that can be removed by deleting that builder without pruning unrelated builders.
4. Removing the disposable container/image/data-volume/builder will reduce some Docker accounting, but exact reclaimed bytes may differ because of shared layers, compression, allocation units, and retained protected state.
5. The protected volume checksum will still verify after all disposable resources are removed.
EOF
cat "$EVIDENCE/predictions.md"

6. Create the checkpoint image and dedicated builder

mkdir -p "$LAB/context"
cat > "$LAB/context/Dockerfile" <<'EOF'
# syntax=docker/dockerfile:1
FROM alpine:3.22.1
LABEL devops-academy.lab=chapter32-checkpoint
RUN dd if=/dev/zero of=/opt/checkpoint-image.bin bs=1M count=4  && sha256sum /opt/checkpoint-image.bin > /opt/checkpoint-image.sha256
CMD ["sh", "-c", "cat /opt/checkpoint-image.sha256"]
EOF

docker buildx create --name dca32-cp-builder --driver docker-container --use
docker buildx inspect --bootstrap | tee "$EVIDENCE/builder-inspect.txt"
docker buildx build   --builder dca32-cp-builder   --load   -t dca32-cp-image:lab   --progress=plain   "$LAB/context" | tee "$EVIDENCE/build.txt"
docker image inspect dca32-cp-image:lab > "$EVIDENCE/image-inspect.json"
docker buildx du --builder dca32-cp-builder | tee "$EVIDENCE/build-cache.txt"

7. Create writable-layer and bounded-log usage

docker create \
  --name dca32-cp-writable \
  --label devops-academy.lab=chapter32-checkpoint \
  --log-driver local \
  --log-opt max-size=1m \
  --log-opt max-file=2   dca32-cp-image:lab   sh -c 'dd if=/dev/zero of=/writable.bin bs=1M count=7; for i in 1 2 3; do echo checkpoint-log-$i; done; sleep 300'

docker start dca32-cp-writable
docker logs dca32-cp-writable | tee "$EVIDENCE/container-logs.txt"
docker ps --size --filter name=dca32-cp-writable | tee "$EVIDENCE/container-size-running.txt"
docker stop dca32-cp-writable
docker inspect --size dca32-cp-writable > "$EVIDENCE/container-inspect-size.json"

8. Create disposable and protected volumes

docker volume create --label devops-academy.lab=chapter32-checkpoint dca32-cp-data
docker volume create --label devops-academy.protection=chapter32-checkpoint dca32-cp-protected

docker run \
  --rm \
  --mount type=volume,src=dca32-cp-data,dst=/data   alpine:3.22.1   sh -c 'dd if=/dev/zero of=/data/disposable.bin bs=1M count=5; sha256sum /data/disposable.bin'   | tee "$EVIDENCE/disposable-volume.txt"

docker run --rm   --mount type=volume,src=dca32-cp-protected,dst=/protected   alpine:3.22.1   sh -c 'printf "DO-NOT-DELETE-UNTIL-VERIFIED
" >/protected/marker.txt; sha256sum /protected/marker.txt'   | tee "$EVIDENCE/protected-before.txt"

The protection label is evidence of policy intent; the stronger safety mechanism is that cleanup commands below never name the protected volume until after verification.

9. Inventory populated state and estimate reclaimability

docker system df | tee "$EVIDENCE/system-df-populated.txt"
docker system df -v | tee "$EVIDENCE/system-df-populated-verbose.txt"
docker ps -a --size --filter label=devops-academy.lab=chapter32-checkpoint   | tee "$EVIDENCE/lab-containers.txt"
docker image ls --filter label=devops-academy.lab=chapter32-checkpoint   | tee "$EVIDENCE/lab-images.txt"
docker volume ls --filter label=devops-academy.lab=chapter32-checkpoint   | tee "$EVIDENCE/disposable-volumes.txt"
docker volume ls --filter label=devops-academy.protection=chapter32-checkpoint   | tee "$EVIDENCE/protected-volumes.txt"
docker buildx du --builder dca32-cp-builder   | tee "$EVIDENCE/builder-du-populated.txt"

Write your estimated reclaimable categories into $EVIDENCE/reclaim-plan.md. Do not promise exact physical bytes; state which objects are expected to disappear and why they are safe to remove.

10. Remove only disposable objects

docker rm dca32-cp-writable
docker volume rm dca32-cp-data
docker image rm dca32-cp-image:lab
docker buildx rm dca32-cp-builder

docker system df | tee "$EVIDENCE/system-df-after-disposable.txt"
docker system df -v | tee "$EVIDENCE/system-df-after-disposable-verbose.txt"

No command names the protected volume. No system/image/container/volume/build-cache prune is used.

11. Prove protected data survived

docker volume inspect dca32-cp-protected > "$EVIDENCE/protected-inspect.json"
docker run \
  --rm \
  --mount type=volume,src=dca32-cp-protected,dst=/protected,readonly   alpine:3.22.1   sh -c 'cat /protected/marker.txt; sha256sum /protected/marker.txt'   | tee "$EVIDENCE/protected-after.txt"

diff -u "$EVIDENCE/protected-before.txt" "$EVIDENCE/protected-after.txt" || true

The container IDs differ and line formatting may vary, but the marker content/checksum must remain stable. That is the checkpoint’s proof that targeted cleanup preserved protected persistent data.

12. Compare predictions with evidence

Prediction 1 — writable size:
  Evidence:
  Result: confirmed / different because ...

Prediction 2 — volume independence:
  Evidence:
  Result:

Prediction 3 — dedicated builder ownership:
  Evidence:
  Result:

Prediction 4 — Docker accounting delta:
  Evidence:
  Result:

Prediction 5 — protected checksum:
  Evidence:
  Result:

An unexpected delta is not automatically a failure. Shared layers, base-image retention, BuildKit storage ownership, compressed logs, filesystem allocation, and asynchronous VM reclamation can all make physical-byte change differ from a simple arithmetic estimate.

13. Evidence packet manifest

14. Final rollback of the remaining protected training volume

# Only after protected-after.txt proves the retention test succeeded:
docker volume rm dca32-cp-protected

docker system df | tee "$EVIDENCE/system-df-final.txt"

The protected volume was “protected during cleanup,” not permanent production data. Its final exact deletion returns the disposable lab to zero persistent resources while preserving the evidence directory.

15. Production storage operating model and Chapter 33 bridge

Chapter 32 adds a storage runbook to the secure Docker operating model: identify the active Engine storage architecture; monitor both bytes and inodes; measure image/content, writable-layer, volume, cache, and log state separately; protect persistent data through explicit ownership/backup; budget BuildKit cache and logs; and delete only resources whose lifecycle is understood.

Knowledge check

Official references and version notes

• Docker Docs — containerd image store with Docker Engine — Engine 29 fresh-install default, snapshotter reporting, switching, and migration boundaries.
• Docker Docs — Storage drivers — image layers, container writable layers, copy-on-write, and container-size accounting caveats.
• Docker Docs — OverlayFS storage driver — classic overlay2, backing-filesystem requirements, lower/upper/merged/workdir semantics, and migration cautions.
• Docker Docs — Select a storage driver — current containerd snapshotter versus classic-driver guidance and platform distinctions.
• Docker CLI — docker system df — daemon disk-usage summary and verbose object-level accounting.
• Docker CLI — docker system prune — deletion scope and supported filters; referenced here to understand risk, not as the lab cleanup path.
• Docker Buildx — buildx du — builder-cache usage, shared/private/reclaimable records, and detailed evidence.
• Docker Buildx — buildx prune — cache filters and space-budget controls.
• Docker Build — Build garbage collection — periodic BuildKit GC, default policy shape, and builder-specific configuration.
• Docker Docs — Docker daemon configuration overview — Docker data-root and the separate managed containerd root used by the Engine 29 containerd image store.
• Docker Docs — Configure logging drivers — json-file growth risk, rotation guidance, and the rotating local driver.
• Docker Engine 29 release notes — current Engine 29 behavior and storage/image-store fixes.