Logs, Logging Drivers, Rotation, stdout/stderr Contracts, Events, stats, and Container Observability: Guided Hands-On Workflow and Core Operations

1. Preflight and lab identity

Record the daemon and image identity first. The examples use alpine:3.22; record the resolved repository digest because the human-readable tag is not the immutable evidence.

mkdir -p dkr23-evidence

docker context show | tee dkr23-evidence/context.txt
docker version | tee dkr23-evidence/docker-version.txt
docker info | tee dkr23-evidence/docker-info.txt
docker info --format 'default_logging_driver={{.LoggingDriver}}' | tee dkr23-evidence/default-driver.txt
docker compose version | tee dkr23-evidence/compose-version.txt 2>&1 || true
docker buildx version | tee dkr23-evidence/buildx-version.txt 2>&1 || true

docker pull alpine:3.22
docker image inspect alpine:3.22 --format '{{json .RepoDigests}}' | tee dkr23-evidence/image-digests.txt

2. Emit structured stdout and stderr

Create one short-lived producer with a stable label. JSON Lines are convenient for machines, but Docker does not require JSON. We emit separate stdout and stderr records and intentionally avoid credentials.

docker rm -f dkr23-streams 2>/dev/null || true

docker run --name dkr23-streams   --label academy=docker-ch23   --label component=stream-demo   alpine:3.22 sh -c '
    i=1
    while [ "$i" -le 5 ]; do
      printf "{\"level\":\"info\",\"seq\":%s,\"msg\":\"heartbeat\"}\n" "$i"
      if [ "$i" -eq 3 ]; then printf "{\"level\":\"warn\",\"seq\":3,\"msg\":\"synthetic warning\"}\n" >&2; fi
      sleep 1
      i=$((i+1))
    done
  '

docker inspect dkr23-streams --format 'id={{.Id}} image={{.Image}} exit={{.State.ExitCode}} log={{json .HostConfig.LogConfig}}'   | tee dkr23-evidence/streams-inspect.txt

3. Read only the log window you need

docker logs can add RFC3339Nano timestamps with --timestamps. --tail, --since, and --until keep an incident query bounded. Avoid dumping an entire high-volume container during diagnosis.

docker logs --timestamps --tail 20 dkr23-streams 2>&1   | tee dkr23-evidence/streams-logs.txt

docker logs --timestamps --since 10m dkr23-streams 2>&1   | tee dkr23-evidence/streams-last-10m.txt

4. Use a rotating driver per container

The mandatory lab uses local only for this disposable container. A small rotation threshold makes the policy observable without generating gigabytes of output. Inspect the configuration; do not browse Docker’s internal driver files.

docker rm -f dkr23-rotate 2>/dev/null || true

docker run -d --name dkr23-rotate   --label academy=docker-ch23   --log-driver local   --log-opt max-size=100k   --log-opt max-file=2   alpine:3.22 sh -c '
    i=1
    while [ "$i" -le 5000 ]; do
      printf "seq=%05d payload=abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz\n" "$i"
      i=$((i+1))
    done
    sleep 30
  '

sleep 2
docker inspect dkr23-rotate --format '{{json .HostConfig.LogConfig}}'   | tee dkr23-evidence/rotate-logconfig.json
docker logs --tail 20 --timestamps dkr23-rotate   | tee dkr23-evidence/rotate-tail.txt

5. Capture a bounded Engine-event timeline

Create a long-running app, record a start timestamp, sample stats, then stop and restart it. After the actions, query only this label and time window. This avoids a noisy live terminal and produces an evidence file you can compare to logs.

docker rm -f dkr23-app 2>/dev/null || true
START_UTC=$(date -u +%Y-%m-%dT%H:%M:%SZ)

docker run -d \
  --name dkr23-app \
  --label academy=docker-ch23 \
  --label component=observed-app \
  --log-driver local \
  --log-opt max-size=1m \
  --log-opt max-file=3   alpine:3.22 sh -c '
    trap "echo event=term_received ts=$(date -u +%Y-%m-%dT%H:%M:%SZ); exit 0" TERM
    i=0
    while :; do i=$((i+1)); echo "event=heartbeat seq=$i"; sleep 1; done
  '

sleep 3
docker stats --no-stream --format '{{json .}}' dkr23-app   | tee dkr23-evidence/app-stats.json
docker stop -t 5 dkr23-app
docker start dkr23-app
sleep 2
END_UTC=$(date -u +%Y-%m-%dT%H:%M:%SZ)

docker events \
  --since "$START_UTC" \
  --until "$END_UTC" \
  --filter type=container \
  --filter label=academy=docker-ch23 \
  --format '{{json .}}'   | tee dkr23-evidence/app-events.jsonl

docker logs --timestamps --since "$START_UTC" dkr23-app 2>&1   | tee dkr23-evidence/app-logs.txt

6. Inspect the active driver and current state

Do not infer driver choice from the daemon default because a container may override it. Inspect the exact object you are diagnosing.

docker inspect dkr23-app \
  --format 'id={{.Id}} image={{.Image}} started={{.State.StartedAt}} status={{.State.Status}} restart_count={{.RestartCount}} log={{json .HostConfig.LogConfig}} labels={{json .Config.Labels}}'   | tee dkr23-evidence/app-inspect.txt

7. Daemon logs: read-only, platform-aware evidence

If you are on a systemd Linux host and have permission, capture a small recent daemon window. On Docker Desktop, use the documented Desktop log location instead. Failure to read daemon logs is a permissions/platform fact—not a reason to broaden privileges.

# Optional on Linux/systemd; read-only and bounded.
journalctl -u docker.service --since '10 minutes ago' --no-pager   | tail -n 200 > dkr23-evidence/daemon-recent.txt 2>&1 || true

8. Challenge: choose the failing layer

Suppose docker logs shows heartbeats until 12:00:10, docker events records die at 12:00:11, and a point-in-time stats sample at 12:00:09 shows memory near its limit. Which evidence should you inspect next?

Reasoned answer: first inspect .State.ExitCode/.State.OOMKilled and resource configuration, then cgroup/resource evidence if available. Do not start by changing the logging driver; the available evidence already points toward process/resource termination.

9. Exact cleanup

Remove only the named lab containers. Keep the evidence directory. Do not use a system-wide prune.

docker rm -f dkr23-streams dkr23-rotate dkr23-app 2>/dev/null || true
docker ps -a --filter label=academy=docker-ch23

Knowledge check

Official references and version notes

• Docker Docs — Configure logging drivers — daemon/container driver selection, delivery modes, labels/tags, and current default-driver behavior.
• Docker Docs — Local file logging driver — default rotation/compression behavior and supported options.
• Docker Docs — JSON file logging driver — JSON framing and explicit rotation options.
• Docker Docs — Dual logging — how docker logs can remain available with remote drivers and when it does not.
• Docker CLI — docker container logs — timestamps, --since, --until, follow, and tail behavior.
• Docker CLI — docker system events — event scope, filters, JSON Lines formatting, and bounded event history.
• Docker CLI — docker container stats — CPU, memory, network, block I/O, PIDs, and Linux cache-reporting notes.
• Docker Docs — Read daemon logs — platform-specific locations and systemd/desktop guidance.
• Docker Engine 29 release notes — current Engine baseline and logging-related fixes/features.