Checkpoint Lab — Docker Performance Engineering: Build Speed, Image Size, Runtime Overhead, Network, Storage, and Resource Profiling

1. Scenario and acceptance contract

You inherit a small Docker build whose incremental feedback is unexpectedly slow. The runtime payload itself is tiny. Your checkpoint is to prove whether Dockerfile cache invalidation is the dominant bottleneck, improve only that dimension, and show that the conclusion survives repeated incremental builds.

2. Preflight and assumptions packet

set -eu
printf 'timestamp=%s
' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
printf 'context=%s
' "$(docker context show)"
docker version
docker info --format 'Server={{.ServerVersion}} Driver={{.Driver}} Cgroup={{.CgroupVersion}} CPUs={{.NCPU}} Memory={{.MemTotal}}'
docker compose version 2>/dev/null || true
docker buildx version

docker pull alpine:3.22
BASE_REF="$(docker image inspect alpine:3.22 --format '{{index .RepoDigests 0}}')"
printf 'base=%s
' "$BASE_REF" | tee ch40cp-base.txt

Also record whether this is native Linux or Docker Desktop, whether the Docker data/build cache lives on local or remote storage, and whether the target platform is native or emulated. Those facts bound the validity of the result.

3. Predict before changing anything

Prediction 1: after app-only edits, the baseline Dockerfile will re-execute the expensive dependency-simulation vertex because COPY . . precedes it.
Prediction 2: after reorganizing COPY boundaries, the optimized Dockerfile will keep that dependency vertex cached while still rebuilding the app-dependent vertex.
Invariant: base digest, synthetic workload, builder, target platform, runtime files, and benchmark procedure remain the same.

4. Create the exact source fixture

rm -rf ch40-checkpoint
mkdir ch40-checkpoint
cd ch40-checkpoint
printf 'dependency-set-v1
' > dependency.lock
printf 'app-v1
' > app.txt
printf '*.log
results/
' > .dockerignore
mkdir results

cat > Dockerfile.baseline <<'EOF'
# syntax=docker/dockerfile:1
ARG BASE
FROM ${BASE} AS build
WORKDIR /src
COPY . .
RUN i=0; while [ "$i" -lt 4500 ]; do       printf '%s:%s
' "$(cat dependency.lock)" "$i" | sha256sum >/dev/null;       i=$((i+1));     done  && mkdir -p /out && cp dependency.lock app.txt /out/
FROM ${BASE}
WORKDIR /app
COPY --from=build /out/ ./
CMD ["sh","-c","cat dependency.lock app.txt"]
EOF

cat > Dockerfile.optimized <<'EOF'
# syntax=docker/dockerfile:1
ARG BASE
FROM ${BASE} AS build
WORKDIR /src
COPY dependency.lock .
RUN i=0; while [ "$i" -lt 4500 ]; do       printf '%s:%s
' "$(cat dependency.lock)" "$i" | sha256sum >/dev/null;       i=$((i+1));     done
COPY app.txt .
RUN mkdir -p /out && cp dependency.lock app.txt /out/
FROM ${BASE}
WORKDIR /app
COPY --from=build /out/ ./
CMD ["sh","-c","cat dependency.lock app.txt"]
EOF

5. Create and bootstrap the dedicated builder

BUILDER=ch40cp-builder
if docker buildx inspect "$BUILDER" >/dev/null 2>&1; then
  echo "Checkpoint builder already exists; clean the prior checkpoint first." >&2
  exit 1
fi
docker buildx create --name "$BUILDER" --driver docker-container --use
docker buildx inspect "$BUILDER" --bootstrap | tee results/builder.txt
docker buildx du --builder "$BUILDER" | tee results/cache-before.txt

6. Warm both build graphs

BASE_REF="$(cat ../ch40cp-base.txt 2>/dev/null | sed 's/^base=//' || true)"
if [ -z "$BASE_REF" ]; then BASE_REF="$(docker image inspect alpine:3.22 --format '{{index .RepoDigests 0}}')"; fi

docker buildx build \
  --builder ch40cp-builder \
  --progress=plain \
  --load \
  --build-arg BASE="$BASE_REF" -f Dockerfile.baseline -t ch40cp:baseline .   2>&1 | tee results/baseline-warmup.log

docker buildx build \
  --builder ch40cp-builder \
  --progress=plain \
  --load \
  --build-arg BASE="$BASE_REF" -f Dockerfile.optimized -t ch40cp:optimized .   2>&1 | tee results/optimized-warmup.log

The warm-up is not part of the incremental comparison. Its job is to establish both caches under the same builder.

7. Baseline incremental benchmark: three app-only edits

for n in 2 3 4; do
  printf 'app-v%s
' "$n" > app.txt
  START=$SECONDS
  docker buildx build --builder ch40cp-builder --progress=plain --load     --build-arg BASE="$BASE_REF" -f Dockerfile.baseline -t ch40cp:baseline .     >"results/baseline-$n.log" 2>&1
  ELAPSED=$((SECONDS-START))
  printf 'baseline run=%s seconds=%s
' "$n" "$ELAPSED" | tee -a results/times.txt
  grep -E 'COPY|RUN i=0|CACHED|DONE' "results/baseline-$n.log" | tail -20
 done

Preserve every run, not only the fastest. Confirm whether the expensive RUN actually executed after each app-only change. Timing without vertex evidence is insufficient.

8. Apply exactly one Docker-specific change

The change is Dockerfile input ordering: dependency metadata is copied before volatile application source. No base image, loop count, builder, platform, context ignore rules, or runtime payload changes. This is the controlled optimization.

9. Optimized incremental benchmark: same three app-only edits

for n in 5 6 7; do
  printf 'app-v%s
' "$n" > app.txt
  START=$SECONDS
  docker buildx build --builder ch40cp-builder --progress=plain --load     --build-arg BASE="$BASE_REF" -f Dockerfile.optimized -t ch40cp:optimized .     >"results/optimized-$n.log" 2>&1
  ELAPSED=$((SECONDS-START))
  printf 'optimized run=%s seconds=%s
' "$n" "$ELAPSED" | tee -a results/times.txt
  grep -E 'COPY|RUN i=0|CACHED|DONE' "results/optimized-$n.log" | tail -20
 done

docker buildx du --builder ch40cp-builder | tee results/cache-after.txt

The success criterion is primarily causal: the expensive dependency vertex remains cached after app-only changes. Wall time should usually improve, but a noisy host can obscure a small time delta; the cache evidence remains decisive for the Dockerfile behavior.

10. Verify final images and runtime behavior

docker image inspect ch40cp:baseline ch40cp:optimized   --format 'tags={{join .RepoTags ","}} id={{.Id}} size={{.Size}}' | tee results/images.txt

docker run --rm --name ch40cp-baseline-run ch40cp:baseline | tee results/baseline-runtime.txt
docker run --rm --name ch40cp-optimized-run ch40cp:optimized | tee results/optimized-runtime.txt

The tags may point to images built from different app versions because the incremental sequences differ; that is acceptable for this build-cache checkpoint. The required invariant is that both constructions preserve the same dependency/application file contract and exit successfully. In a production experiment, use the same source revision for final artifact comparison.

11. Optional runtime snapshot—do not confuse it with build evidence

docker run -d --name ch40cp-runtime --memory=64m --cpus=0.5 ch40cp:optimized sh -c 'sleep 30'
docker inspect ch40cp-runtime --format 'Image={{.Image}} NanoCpus={{.HostConfig.NanoCpus}} Memory={{.HostConfig.Memory}}'
docker stats --no-stream ch40cp-runtime | tee results/runtime-stats.txt
docker rm -f ch40cp-runtime

This proves the runtime envelope and stats tooling work, but it is not the optimization target. Keep build and runtime conclusions separate.

12. Evidence packet and report

Performance question: Why are app-only incremental builds slow?
Population: <native Linux / Desktop / remote; host/VM notes>
Base digest: <from ch40cp-base.txt>
Builder: <results/builder.txt>
Baseline runs: <results/times.txt baseline rows>
Optimized runs: <results/times.txt optimized rows>
Causal evidence: <baseline expensive RUN executed; optimized expensive RUN CACHED>
Cache usage: <cache-before.txt / cache-after.txt>
Image identity/size: <images.txt>
Runtime sanity: <runtime output>
One change: COPY dependency.lock before app.txt
Security/reliability changes: none intended
Timer limitation: Bash SECONDS, one-second resolution
Variance limitation: small synthetic workload; host background load uncontrolled
External-validity limit: repeat on production-like builder/storage before capacity decisions
Conclusion: <supported / not supported by evidence>

13. Verification checklist

14. Bounded cleanup

docker rm -f ch40cp-runtime 2>/dev/null || true
docker image rm ch40cp:baseline ch40cp:optimized 2>/dev/null || true
docker buildx rm ch40cp-builder
cd ..
rm -rf ch40-checkpoint
rm -f ch40cp-base.txt

The benchmark logs under ch40-checkpoint/results are deleted by this cleanup. Copy the evidence packet elsewhere first if you intend to retain it. No broad Docker cleanup is required.

15. What Chapter 40 adds to the production operating model

You can now turn “Docker feels slow” into a measured, layer-specific hypothesis; separate cold, warm, and incremental builds; distinguish compressed image transfer from runtime footprint; interpret Docker stats alongside cgroup/host evidence; profile storage/network paths without overgeneralizing; and accept an optimization only after the same benchmark improves under a controlled change. Chapter 41 turns this evidence discipline into full Docker troubleshooting across daemon failures, networking, DNS, storage, permissions, builds, and container incidents.

Knowledge check

Official references and version notes

• Docker Docs — Optimize cache usage in builds — cache ordering, small contexts, bind/cache mounts, and external cache.
• Docker Docs — Cache storage backends — local/registry/inline/GHA cache boundaries and current driver requirements.
• Docker Docs — Registry cache — cache mode and compression options.
• Docker Docs — Exporters overview — output compression and size-versus-compute tradeoffs.
• Docker Docs — Image and registry exporters — gzip/estargz/zstd, compression levels, OCI media types, and timestamp rewriting.
• Docker CLI — docker buildx du — builder cache disk-usage evidence.
• Docker CLI — docker stats — CPU, memory, network, block I/O, PIDs, and Linux memory-cache presentation semantics.
• Docker Docs — Runtime metrics — cgroup-level CPU, memory, and block-I/O evidence.
• Docker Docs — Resource constraints — CPU/memory controls and measurement-before-limits guidance.
• Docker Docs — Storage drivers — writable-layer behavior and Engine 29 containerd image-store note.
• Docker Docs — Select a storage backend — containerd snapshotters versus classic overlay2 context.
• Docker Docs — OverlayFS/overlay2 — copy-on-write behavior, prerequisites, and performance considerations.
• Docker Docs — Volumes — Docker-managed persistent storage and host-coupling boundary.
• Docker Docs — Bind mounts — daemon-host path coupling and Desktop file-sharing implications.
• Docker Engine 29 release notes — current Engine baseline.
• Buildx releases and BuildKit releases — current builder versions.