Logs, Logging Drivers, Rotation, stdout/stderr Contracts, Events, stats, and Container Observability: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Record context, Engine/CLI versions, host/Desktop boundary, and time.
2. Record container ID/name, exact image identity, state, exit code/OOM flag, and logging driver/options.
3. Capture a bounded application-log window and Engine events before history rolls over.
4. Capture a stats sample if the container is still running; preserve resource configuration if it has exited.
5. Inspect daemon logs for driver, filesystem, transport, or Engine errors.
6. For remote logging, prove external sink receipt independently without exposing credentials.
7. Apply one least-destructive correction and rerun the smallest reproduction.

docker context show
docker version
docker inspect dkr23-app --format '{{json .State}} {{json .HostConfig.LogConfig}}'
docker logs --timestamps --since 10m --tail 500 dkr23-app 2>&1
docker events --since 10m --filter container=dkr23-app --format '{{json .}}'
docker stats --no-stream --format '{{json .}}' dkr23-app 2>/dev/null || true

2. Failure: unbounded json-file growth

Symptom: daemon-host disk pressure rises while application behavior is otherwise normal. Cause: high-volume json-file containers with no rotation can accumulate data indefinitely until external capacity limits intervene.

Preserve: exact container driver/options, host disk usage trend, daemon warnings, container output rate, and retention requirements. Correction: for future/recreated containers, choose local or configure explicit json-file rotation. Do not delete Docker-managed log files by hand.

3. Failure: critical logs exist only inside the writable layer

If the application writes only /app/logs/app.log and the container is replaced, operational evidence can disappear with its writable layer. The correction is not “docker cp the file every incident.” Establish stdout/stderr or an explicit persistent/collector contract, then rebuild/redeploy.

4. Failure: a fake credential leaks into logs

Use a synthetic value to understand the response without exposing a real secret. Once a real secret reaches logs, rotation does not guarantee immediate removal from every retained local/remote copy. Treat it as a credential incident according to the relevant secret-rotation and retention procedures.

docker rm -f dkr23-badlog 2>/dev/null || true
docker run --name dkr23-badlog --label academy=docker-ch23 alpine:3.22   sh -c 'echo "ERROR demo_auth=FAKE-ONLY-NOT-A-CREDENTIAL" >&2; exit 23' || true

docker logs --timestamps dkr23-badlog 2>&1 | tee dkr23-evidence/badlog-original.txt
docker inspect dkr23-badlog --format 'exit={{.State.ExitCode}} image={{.Image}} log={{json .HostConfig.LogConfig}}'

5. Failure: docker logs and the remote sink disagree

With a remote driver, docker logs can be served by dual logging rather than by reading back from the remote system. Therefore these states are possible: local recent logs exist but remote delivery failed; remote logs exist but local cache was disabled/rotated; both are missing because delivery/cache failed; or the application emitted nothing.

Inspect the per-container driver/options, daemon logs, network/TLS/provider errors, dual-cache configuration, and remote receipt. Do not equate a successful docker logs command with successful external transport.

6. Failure: changing the daemon default does not fix existing containers

Docker documentation explicitly states that daemon logging changes apply to newly created containers. Existing containers retain the logging configuration they were created with. A restart of the existing container does not replace its HostConfig.LogConfig.

# Read-only proof for two containers created at different times.
docker inspect OLD NEW --format '{{.Name}} {{json .HostConfig.LogConfig}}'

7. Failure: tuning without disk or latency evidence

Increasing buffers, retention, or remote retries can move pressure elsewhere. Larger local retention consumes disk; non-blocking buffers consume memory and can still drop; blocking delivery can stall the application; aggressive remote retry can amplify incidents. Measure output rate, disk headroom, sink latency/error rate, and application tolerance first.

8. Causal map: do not troubleshoot the wrong layer

Knowledge check

Official references and version notes

• Docker Docs — Configure logging drivers — daemon/container driver selection, delivery modes, labels/tags, and current default-driver behavior.
• Docker Docs — Local file logging driver — default rotation/compression behavior and supported options.
• Docker Docs — JSON file logging driver — JSON framing and explicit rotation options.
• Docker Docs — Dual logging — how docker logs can remain available with remote drivers and when it does not.
• Docker CLI — docker container logs — timestamps, --since, --until, follow, and tail behavior.
• Docker CLI — docker system events — event scope, filters, JSON Lines formatting, and bounded event history.
• Docker CLI — docker container stats — CPU, memory, network, block I/O, PIDs, and Linux cache-reporting notes.
• Docker Docs — Read daemon logs — platform-specific locations and systemd/desktop guidance.
• Docker Engine 29 release notes — current Engine baseline and logging-related fixes/features.