Daemon Configuration, daemon.json, systemd, Proxies, Registry Mirrors, Live Restore, and Host Integration: Guided Hands-On Workflow and Core Operations

1. Two-track lab design

Track A (mandatory everywhere) is read-only/offline: inventory the current Engine, validate temporary configuration files, generate a systemd drop-in model, and run a local mirror service without pointing the production daemon at it. It does not restart or reload Docker.

Track B (optional but fully runnable) is for a fresh disposable Linux VM you control. It applies one harmless reloadable daemon label, optionally points the disposable daemon at the local mirror, and demonstrates live restore. Do not use Track B on a workstation hosting important containers.

2. Capture environment and component assumptions

set -eu
LAB=dca33-lab
EVIDENCE="$LAB/evidence"
mkdir -p "$EVIDENCE"

date -u +%Y-%m-%dT%H:%M:%SZ | tee "$EVIDENCE/time.txt"
docker context show | tee "$EVIDENCE/context.txt"
docker version | tee "$EVIDENCE/docker-version.txt"
docker info | tee "$EVIDENCE/docker-info.txt"
docker compose version 2>&1 | tee "$EVIDENCE/compose-version.txt" || true
docker buildx version 2>&1 | tee "$EVIDENCE/buildx-version.txt" || true
containerd --version 2>&1 | tee "$EVIDENCE/containerd-version.txt" || true
runc --version 2>&1 | tee "$EVIDENCE/runc-version.txt" || true

Record actual installed versions rather than assuming that Engine packages, Docker Desktop, static binaries, and distro packaging bundle the same component builds.

3. Read-only service ownership on native Linux

# Run only if the selected Docker daemon is this Linux host.
systemctl show docker --property=FragmentPath,DropInPaths,MainPID,ActiveEnterTimestamp   | tee "$EVIDENCE/systemd-owner.txt"
ps -eo pid,args | grep '[d]ockerd' | tee "$EVIDENCE/dockerd-command.txt"

# Inspect privately; do NOT tee raw output if it may contain credentials:
# sudo systemctl cat docker
# sudo cat /etc/docker/daemon.json

If docker context show points elsewhere, these systemd commands are client-host evidence and must not be mislabelled as remote-daemon evidence.

4. Offline validation: one good file and one broken file

cat > "$LAB/good-daemon.json" <<'EOF'
{
  "labels": ["devops-academy.chapter=33-lab"]
}
EOF

cat > "$LAB/bad-daemon.json" <<'EOF'
{
  "labels": ["devops-academy.chapter=33-lab"],
  "definitely-not-a-dockerd-option": true
}
EOF

sudo dockerd --validate --config-file="$LAB/good-daemon.json"   | tee "$EVIDENCE/validate-good.txt"

set +e
sudo dockerd --validate --config-file="$LAB/bad-daemon.json"   >"$EVIDENCE/validate-bad.txt" 2>&1
BAD_RC=$?
set -e
printf 'bad validation rc=%s
' "$BAD_RC" | tee -a "$EVIDENCE/validate-bad.txt"
test "$BAD_RC" -ne 0

The failed candidate remains as evidence. The repair is not “restart and see”; it is to correct the unsupported directive, revalidate, and only then consider rollout.

5. Model a systemd proxy drop-in with fake values

mkdir -p "$LAB/systemd-model"
cat > "$LAB/systemd-model/http-proxy.conf" <<'EOF'
[Service]
Environment="HTTP_PROXY=http://proxy.example.test:3128"
Environment="HTTPS_PROXY=http://proxy.example.test:3128"
Environment="NO_PROXY=localhost,127.0.0.1,.example.test"
EOF
cat "$LAB/systemd-model/http-proxy.conf"

This is a model file only. It is not installed under /etc/systemd/system. Real proxy URLs can contain credentials; never commit or publish those values casually.

6. Compare daemon proxy and container/build proxy

7. Start a local pull-through-cache simulation

# Guard against name collision.
if docker container inspect dca33-mirror >/dev/null 2>&1; then
  echo 'Refusing to reuse preexisting dca33-mirror' >&2; exit 1
fi

docker pull registry:3.1.1
docker image inspect registry:3.1.1 > "$EVIDENCE/registry-image-inspect.json"

docker run -d \
  --name dca33-mirror \
  --label devops-academy.lab=chapter33   -p 127.0.0.1:5001:5000   -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io   registry:3.1.1   | tee "$EVIDENCE/mirror-container-id.txt"

curl -fsS http://127.0.0.1:5001/v2/ | tee "$EVIDENCE/mirror-v2.txt"
docker logs dca33-mirror > "$EVIDENCE/mirror-logs.txt" 2>&1

This proves that a local Distribution service is reachable and configured in proxy mode. It does not prove the Docker daemon is using it as a mirror. Daemon use requires a registry-mirrors policy change, which Track A intentionally does not apply.

8. Validate a mirror configuration without applying it

cat > "$LAB/mirror-daemon.json" <<'EOF'
{
  "registry-mirrors": ["http://127.0.0.1:5001"],
  "labels": ["devops-academy.chapter=33-lab"]
}
EOF
sudo dockerd --validate --config-file="$LAB/mirror-daemon.json"   | tee "$EVIDENCE/validate-mirror.txt"

Loopback is used only for the local disposable simulation. Production mirrors should use authenticated/trusted TLS appropriate to your environment. The lesson does not add a broad insecure-registry rule.

9. Track B: preflight the disposable Linux VM before daemon mutation

# OPTIONAL: disposable Linux VM only.
test "${DCA33_DISPOSABLE_VM:-}" = YES || {
  echo 'Set DCA33_DISPOSABLE_VM=YES only inside the disposable VM.' >&2
  exit 1
}

docker ps -a --format '{{.Names}}' | tee "$EVIDENCE/vm-containers-before.txt"

# This teaching track expects no preexisting daemon.json.
if sudo test -e /etc/docker/daemon.json; then
  echo 'Existing daemon.json detected. Use Track A instead of overwriting it.' >&2
  exit 1
fi

systemctl show docker --property=MainPID,ActiveEnterTimestamp   | tee "$EVIDENCE/daemon-before.txt"

Refusing to overwrite an existing configuration is deliberate. A real configuration-management rollout would merge versioned policy; this small lab uses a fresh VM to keep rollback exact.

10. Track B: apply one harmless reloadable daemon label

cat > /tmp/dca33-daemon.json <<'EOF'
{
  "labels": ["devops-academy.chapter=33-checkpoint"]
}
EOF
sudo dockerd --validate --config-file=/tmp/dca33-daemon.json   | tee "$EVIDENCE/vm-validate.txt"
sudo install -o root -g root -m 0644 /tmp/dca33-daemon.json /etc/docker/daemon.json
sudo systemctl reload docker

systemctl show docker --property=MainPID,ActiveEnterTimestamp   | tee "$EVIDENCE/daemon-after-reload.txt"
docker info | tee "$EVIDENCE/docker-info-after-reload.txt"

Daemon labels are currently reloadable. The expected observation is that the daemon PID/start timestamp stays stable while the effective label appears in docker info.

11. Track B: optional mirror-use verification

# OPTIONAL in the disposable VM: replace daemon.json with the validated mirror+label model.
sudo install -o root -g root -m 0644 "$LAB/mirror-daemon.json" /etc/docker/daemon.json
sudo systemctl reload docker

docker pull busybox:1.37.0
docker image inspect busybox:1.37.0 > "$EVIDENCE/busybox-inspect.json"
docker logs dca33-mirror > "$EVIDENCE/mirror-logs-after-pull.txt" 2>&1

Evidence is the combination of daemon configuration, successful pull, and mirror logs showing the request path. A successful pull alone cannot prove which upstream path was used.

12. Track B: live-restore semantics, only in the disposable VM

# Replace config with live-restore + lab label; validate first.
cat > /tmp/dca33-live.json <<'EOF'
{
  "live-restore": true,
  "labels": ["devops-academy.chapter=33-checkpoint"]
}
EOF
sudo dockerd --validate --config-file=/tmp/dca33-live.json
sudo install -o root -g root -m 0644 /tmp/dca33-live.json /etc/docker/daemon.json
sudo systemctl reload docker

if docker container inspect dca33-live >/dev/null 2>&1; then
  echo 'Refusing to reuse preexisting dca33-live' >&2; exit 1
fi

docker run -d --name dca33-live   --label devops-academy.lab=chapter33   -p 127.0.0.1:18080:80   nginx:1.29.1-alpine
curl -fsS http://127.0.0.1:18080/ > "$EVIDENCE/http-before-daemon-stop.html"

docker inspect -f 'id={{.Id}} pid={{.State.Pid}} restart={{.RestartCount}}' dca33-live   | tee "$EVIDENCE/live-before.txt"

sudo systemctl stop docker
curl -fsS http://127.0.0.1:18080/ > "$EVIDENCE/http-while-daemon-down.html"
sudo systemctl start docker

docker inspect -f 'id={{.Id}} pid={{.State.Pid}} restart={{.RestartCount}}' dca33-live   | tee "$EVIDENCE/live-after.txt"

This is intentionally disruptive to the disposable VM daemon and therefore never belongs on a shared host as a training shortcut. The HTTP request is external evidence that the container process kept serving while dockerd was unavailable.

13. Cleanup and rollback

docker rm -f dca33-live 2>/dev/null || true
docker rm -f dca33-mirror 2>/dev/null || true

# Disposable VM Track B only: return to the original “no daemon.json” state.
if [ "${DCA33_DISPOSABLE_VM:-}" = YES ]; then
  sudo rm -f /etc/docker/daemon.json
  sudo systemctl reload docker || sudo systemctl restart docker
  systemctl show docker --property=MainPID,ActiveEnterTimestamp     | tee "$EVIDENCE/daemon-after-rollback.txt"
fi

The fallback restart is allowed only in the disposable VM because removing a configuration key that is not reloadable may require restart. On any non-disposable host, use the organization’s normal change/rollback procedure instead.

14. Small challenge: choose the layer before the command

Your organization reports: “containers can reach the corporate proxy, but docker pull cannot.” Before changing anything, identify the owner and evidence. The correct hypothesis is daemon egress, not container environment. Capture daemon proxy policy, endpoint identity, and daemon logs; do not add proxy variables to the application as a random retry.

Knowledge check

Official references and version notes

Lab baseline date: 2026-09-22. The optional mirror simulation uses Docker Official Image registry:3.1.1, the current stable v3 line at verification time; the lab records the pulled image inspection/digest so evidence remains immutable even if tags later move.

• Docker Docs — Docker daemon configuration overview — preferred JSON configuration, default paths, flag/JSON conflicts, and Docker Desktop distinction.
• Docker CLI reference — dockerd — configuration keys, --validate, proxy flags, registry trust, reloadable options, and multi-daemon cautions.
• Docker Docs — Daemon proxy configuration — daemon-side HTTP/HTTPS/NO_PROXY behavior and systemd environment drop-ins.
• Docker Docs — Docker CLI proxy configuration — container/build proxy injection and why it is distinct from daemon egress.
• Docker Docs — Mirror the Docker Hub library — pull-through cache configuration, daemon registry-mirrors, and credential/privacy warnings.
• Docker Docs — Live restore — standalone-container continuity, reload, patch-upgrade scope, configuration-change limitations, FIFO log behavior, and Swarm boundary.
• Docker Docs — Read the daemon logs — platform-specific daemon-log locations and incident evidence.
• Docker Docs — Linux post-installation — systemd service enablement and host-integration context.
• Docker Engine 29 release notes — current Engine 29 behavior, validation changes, fixes, and component updates.
• Docker Official Image — registry — current local OCI Distribution image tags used for the optional mirror simulation.