Multi-Stage Builds, Minimal Runtime Images, Distroless Patterns, Debug Stages, and Build Separation: Diagnostics, Failure Modes, Security, and Performance

1. Preserve the image that failed before “making it bigger”

A minimal-runtime failure often tempts engineers to add a shell, package manager, compiler, or root access immediately. That destroys the very evidence needed to distinguish a missing runtime dependency from a build problem. Record the exact image ID/digest, Dockerfile/source revision, target, build metadata, container exit/error text, configured user/entrypoint, and any external dependency failure first.

docker image inspect devops-academy-ch10:runtime \
  > evidence/runtime-inspect-before.json
docker image history --no-trunc devops-academy-ch10:runtime \
  > evidence/runtime-history-before.txt

2. Evidence-first diagnostic sequence

1. Confirm host/platform, active Docker context, Engine/CLI, Buildx, selected builder, target platform, and frontend assumptions.
2. Confirm source/Dockerfile/base identities and the exact target/output that produced the image.
3. Inspect image config: user, entrypoint/CMD, environment, workdir, architecture, size, and labels.
4. Preserve the original create/start/run error, exit code, and events before rebuilding.
5. Ask whether the executable itself exists and whether its interpreter/loader/libraries/data files exist.
6. Use the dedicated debug target or offline extraction to inspect artifacts; do not hot-patch the release container.
7. Change one declared dependency or configuration choice, rebuild to a new image identity, and verify the smallest failing scope.

3. Intentionally broken example: a script in scratch

This Dockerfile copies a script whose shebang requires /bin/sh, but the scratch filesystem contains no shell:

# syntax=docker/dockerfile:1
FROM busybox:1.37.0 AS build
RUN printf '#!/bin/sh\necho hello\n' > /start.sh \
    && chmod 755 /start.sh

FROM scratch
COPY --from=build /start.sh /start.sh
ENTRYPOINT ["/start.sh"]

Build it under a disposable tag, then preserve the expected start failure:

docker buildx build --load \
  -t devops-academy-ch10:broken-script \
  -f Dockerfile.broken .

set +e
docker run --rm devops-academy-ch10:broken-script \
  > evidence/broken.stdout 2> evidence/broken.stderr
printf 'exit=%s\n' "$?" > evidence/broken-exit.txt
set -e

The file exists, but its interpreter does not. The repair is not “Docker needs privileged mode” or “disable seccomp.” Either use an exec-able static binary, or choose/copy a runtime that deliberately supplies the interpreter. The cause is the runtime filesystem contract.

4. Missing dynamic loader can look like a missing executable

A dynamically linked ELF binary records an interpreter such as a glibc or musl loader. If you copy only the binary into scratch, startup can fail even though /app is present. Diagnose linkage in a build/debug environment with tools such as file, readelf, or ldd where appropriate, and record the output. Then either produce a truly static artifact or choose/copy the exact runtime libraries and loader.

5. TLS failure: certificate data is a runtime dependency

An application can start perfectly in scratch yet fail outbound HTTPS with an unknown-authority error because no CA roots exist. Preserve the application error and endpoint identity first. Then decide whether to copy a reviewed CA bundle into the runtime stage or choose a base whose documented contract includes CA certificates, such as an appropriate current Distroless family.

Do not “fix” certificate failures by disabling TLS verification. That changes the trust model rather than supplying the missing dependency.

6. Timezone, locale, and name-service assumptions are easy to miss

Programs that convert named time zones may need timezone data. Some language/runtime features expect locale files or /etc/nsswitch.conf. User-name lookup may expect /etc/passwd. DNS resolution itself is provided through container runtime configuration and resolver files, but library behavior can still vary with libc/runtime choices. Record the exact failing operation and required file/library instead of assuming “scratch cannot network.”

7. Non-root failure: inspect ownership before escalating privilege

If UID 65532 cannot read an executable or write a required directory, inspect mode/ownership and mounts. The Dockerfile may need COPY --chown, an explicitly prepared writable directory, or a correctly owned volume. Switching the final stage to root changes the security boundary and can mask the packaging defect.

docker image inspect devops-academy-ch10:runtime \
  --format 'User={{.Config.User}} Entrypoint={{json .Config.Entrypoint}}'

docker create --name devops-academy-ch10-perm devops-academy-ch10:runtime
docker cp devops-academy-ch10-perm:/app ./evidence/app-for-permission-check
ls -ln ./evidence/app-for-permission-check
docker rm devops-academy-ch10-perm

8. Toolchain leakage: prove what crossed instead of guessing

If the release unexpectedly contains source, package managers, or compilers, inspect the Dockerfile's copy boundary and image history. For a shell-equipped debug/comparison target, list expected paths. For a shell-less release, create a stopped container and use docker cp only for known paths, or export to an isolated analysis directory if your process permits. Do not run untrusted tools from the image on the host.

Common causes include FROM build AS runtime when a fresh runtime base was intended, broad COPY --from=build / /, or copying a workspace rather than a build artifact directory.

9. Performance: measure transfer and rebuild cost, not only compressed size

Smaller runtimes generally transfer faster and consume less registry/disk bandwidth, but startup time also depends on decompression, page faults, application initialization, storage driver, host cache, and network. Multi-stage builds can also improve build clarity without changing runtime performance at all. Measure image sizes and startup/build timings separately and avoid claiming causality from one metric.

10. Immutable repair pattern

After identifying a missing dependency, change the Dockerfile or artifact build, rebuild to a new image identity, rerun the failing test, and compare evidence. Do not docker exec a hot fix into the production-like container and then treat that modified writable layer as the solution. Chapter 06 established why runtime mutation is not image engineering.

11. Diagnostic challenge

A scratch image returns an HTTPS certificate error but runs as UID 65532 and has the correct binary hash. Which layer is already exonerated, which evidence should you preserve, and why are “run as root” and “disable TLS verification” both invalid first fixes?

Knowledge check

Official references and version notes

• Docker multi-stage builds — named stages, COPY --from, stage reuse, and stopping at a specific target.
• Dockerfile reference — FROM, stage naming, COPY --from, USER, ENTRYPOINT, and current frontend semantics.
• Base images — choosing base images and creating minimal images with the reserved scratch base.
• Docker build best practices — small trusted bases, rebuilding, pinning, decoupling applications, and non-root considerations.
• docker buildx build — --target, --metadata-file, progress, output, and build-result evidence.
• docker image inspect — size and runtime configuration evidence.
• docker image history — image-history evidence and its limits.
• GoogleContainerTools/distroless — current Distroless image families, Debian 13 tags, nonroot/debug variants, no-shell behavior, and signature guidance.
• Distroless base image contents — static/base/base-nossl runtime contents such as CA certificates, tzdata, glibc, and libssl.
• Distroless support policy — current Debian-family support timelines.
• Docker Engine 29 release notes — current Engine baseline and bundled component updates.
• Buildx releases — current Buildx release history.
• BuildKit releases — current BuildKit and built-in Dockerfile frontend history.