Troubleshooting Docker: Daemon Failures, Networking, DNS, Storage, Permissions, Builds, and Container Incidents: Guided Hands-On Workflow and Core Operations

1. Lab contract and preflight

This lab uses a recorded Alpine image digest, exact ch41-* names, one user-defined bridge, one small bind directory, one 4 MiB tmpfs, and one tiny build context. It does not restart Docker, alter daemon JSON, change host firewall/DNS policy, broaden permissions, or prune shared resources.

set -eu
printf 'timestamp=%s
' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
printf 'context=%s
' "$(docker context show)"
docker version
docker info --format 'Server={{.ServerVersion}} Driver={{.Driver}} Cgroup={{.CgroupVersion}}'
docker buildx version

docker pull alpine:3.22
BASE_REF="$(docker image inspect alpine:3.22 --format '{{index .RepoDigests 0}}')"
printf 'base=%s
' "$BASE_REF"

2. Incident A — client/context failure

Before debugging the daemon, prove that the client can select the requested context. This deliberately references a context that should not exist; no daemon object is mutated.

docker --context ch41-context-does-not-exist ps 2>&1 | tee ch41-context-error.log || true
printf 'actual_context=%s
' "$(docker context show)"
docker context inspect "$(docker context show)" > ch41-context-inspect.json

Interpretation: a “context not found” result is client/context state. The smallest correction is to use the intended named context explicitly or repair context configuration—not restart containers.

3. Incident B — controlled crash loop

The process exits with a synthetic application code and a bounded on-failure:3 restart policy. Preserve state after the retries instead of deleting the container immediately.

docker rm -f ch41-crash 2>/dev/null || true
docker run -d \
  --name ch41-crash \
  --label devops.academy.lab=ch41 \
  --restart on-failure:3   "$BASE_REF" sh -c 'echo "synthetic startup failure" >&2; exit 17'
sleep 5

docker ps -a --filter name=ch41-crash --no-trunc
docker inspect ch41-crash --format 'Id={{.Id}} Status={{.State.Status}} Exit={{.State.ExitCode}} RestartCount={{.RestartCount}} Image={{.Image}}'
docker logs --timestamps ch41-crash | tee ch41-crash.log
END="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
docker events --since 10m --until "$END" --filter container=ch41-crash > ch41-crash.events || true

The daemon has successfully executed the policy; the application process is the failing layer. Repair the process contract only after saving the original ID/log/event packet.

OLD_ID="$(docker inspect ch41-crash --format '{{.Id}}')"
docker rm ch41-crash
docker run -d --name ch41-crash --label devops.academy.lab=ch41 "$BASE_REF" sleep 300
NEW_ID="$(docker inspect ch41-crash --format '{{.Id}}')"
printf 'old=%s
new=%s
' "$OLD_ID" "$NEW_ID"
docker inspect ch41-crash --format 'Status={{.State.Status}} Exit={{.State.ExitCode}}' 

4. Incident C — user-defined-network DNS typo

Create a scoped network and a sleeping peer. The first query uses the wrong service name; the second repeats the same resolver check with the correct name.

docker network rm ch41-net 2>/dev/null || true
docker network create --label devops.academy.lab=ch41 ch41-net
docker rm -f ch41-api 2>/dev/null || true
docker run -d --name ch41-api --network ch41-net --label devops.academy.lab=ch41 "$BASE_REF" sleep 300

docker run --rm --network ch41-net "$BASE_REF" nslookup ch41-api-typo 2>&1 | tee ch41-dns-fail.log || true
docker network inspect ch41-net > ch41-network.json
docker inspect ch41-api --format '{{json .NetworkSettings.Networks}}'

docker run --rm --network ch41-net "$BASE_REF" nslookup ch41-api | tee ch41-dns-fixed.log

The fix changes the queried identity, not host DNS. On a user-defined bridge, Docker’s embedded DNS supplies container-name resolution. If external names fail instead, inspect the container resolver/upstream path as a different branch.

5. Incident D — deterministic read-only bind failure

This incident avoids host-specific UID/GID behavior by making the mount contract explicitly read-only. The error should be interpreted as mount configuration, not “Docker permissions are broken.”

rm -rf ch41-host
mkdir ch41-host
printf 'seed
' > ch41-host/input.txt

docker run \
  --rm \
  --name ch41-ro \
  --mount type=bind,source="$PWD/ch41-host",target=/work,readonly   "$BASE_REF" sh -c 'echo attempt > /work/output.txt'   2>&1 | tee ch41-bind-fail.log || true

docker create --name ch41-ro-inspect   --mount type=bind,source="$PWD/ch41-host",target=/work,readonly   "$BASE_REF" true >/dev/null
docker inspect ch41-ro-inspect --format '{{json .Mounts}}' | tee ch41-bind-inspect.json
docker rm ch41-ro-inspect

docker run \
  --rm \
  --name ch41-rw \
  --mount type=bind,source="$PWD/ch41-host",target=/work   "$BASE_REF" sh -c 'echo fixed > /work/output.txt && cat /work/output.txt'
cat ch41-host/output.txt

The correction is narrow: only this disposable mount becomes writable because the workload contract requires a write. In a real incident, also verify host ownership, SELinux/AppArmor policy, user namespaces, and the runtime user before changing permissions.

6. Incident E — bounded “disk full” using tmpfs

Never fill the host filesystem just to practice ENOSPC. A small tmpfs gives a local, self-cleaning capacity boundary.

docker run \
  --rm \
  --name ch41-enospc \
  --mount type=tmpfs,destination=/scratch,tmpfs-size=4m   "$BASE_REF" sh -c 'df -h /scratch; dd if=/dev/zero of=/scratch/blob bs=1M count=8; df -h /scratch'   2>&1 | tee ch41-enospc.log || true

If the failure is “No space left on device,” the evidence points to that mount capacity. Do not respond by pruning unrelated images or volumes. In a real host-pressure incident, separately inspect bytes, inodes, Docker-wide usage, build cache, container writable size, volumes, and logs.

7. Incident F — BuildKit input failure

The Dockerfile asks for a source file that does not exist in the context. Preserve plain BuildKit progress before creating the missing file.

rm -rf ch41-build
mkdir ch41-build
cat > ch41-build/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM alpine:3.22
WORKDIR /app
COPY required.txt ./
CMD ["cat","/app/required.txt"]
EOF

docker buildx build --progress=plain --load -t ch41-build:broken ch41-build   > ch41-build-fail.log 2>&1 || true
cat ch41-build-fail.log

printf 'present
' > ch41-build/required.txt
docker buildx build --progress=plain --load -t ch41-build:fixed ch41-build   2>&1 | tee ch41-build-fixed.log
docker run --rm ch41-build:fixed

The failed vertex belongs to build input/context state. Restarting the Docker daemon would not create required.txt.

8. Optional daemon/system evidence — read only first

If a lab command instead fails because the daemon is unreachable, stop the scenario and capture daemon evidence appropriate to the platform. On systemd Linux, journalctl -xu docker.service is the documented starting point. Docker Desktop and Windows containers use different log locations; do not assume a Linux path.

# Native Linux/systemd example (read-only):
journalctl -xu docker.service --since '-15 min' --no-pager 2>/dev/null | tail -200 || true

9. Challenge — identify the owning layer before the command

10. Cleanup only exact lab state

docker rm -f ch41-crash ch41-api 2>/dev/null || true
docker network rm ch41-net 2>/dev/null || true
docker image rm ch41-build:broken ch41-build:fixed 2>/dev/null || true
rm -rf ch41-host ch41-build
# Keep or explicitly remove the evidence files you created:
# ch41-*.log, ch41-*.json, ch41-*.events

No Docker-wide cleanup is required. Evidence files are left to you deliberately; incident preservation is a choice, not an accidental side effect.

Knowledge check

Official references and version notes

• Docker Docs — Troubleshoot the Docker daemon — daemon startup/configuration and networking/DNS troubleshooting.
• Docker Docs — Read the daemon logs — Linux journal/syslog locations, Docker Desktop VM logs, Windows Event Log, debug logging, and non-destructive SIGUSR1 stack traces.
• Docker Engine 29 release notes — current 29.8.1 fixes, security changes, regressions, and documented compatibility notes.
• docker context show and docker context inspect — prove the daemon endpoint before mutation.
• docker inspect — low-level Docker object state, IDs, mounts, network settings, health, and runtime configuration.
• docker logs — bounded container stdout/stderr evidence.
• docker events — daemon object event timeline and filters.
• docker buildx build — --progress=plain and raw build progress for reproducible failure traces.
• Docker Docs — Optimize build cache — input/cache boundaries relevant to failed and unexpectedly expensive builds.
• Docker Docs — Networking overview — container network namespaces, default versus user-defined networks, and embedded DNS at 127.0.0.11.
• Docker Docs — Bridge network driver — automatic name resolution on user-defined bridges and network isolation.
• Docker Docs — Storage — writable-layer versus persistent mount boundaries.
• Docker Docs — Volumes — Docker-managed persistent data and lifecycle evidence.
• Docker Docs — Bind mounts — daemon-host source paths, read/write authority, and host coupling.
• Docker Docs — Storage drivers — copy-on-write writable-layer behavior and performance implications.
• Docker Docs — Running containers — process state, exit behavior, health checks, resource and mount options.
• Docker Docs — Resource constraints — distinguish host pressure from configured CPU/memory/PID limits.