Logs, Logging Drivers, Rotation, stdout/stderr Contracts, Events, stats, and Container Observability: Configuration, Design Choices, and Tradeoffs

1. Start with the operating contract

Before choosing a driver, answer four questions: How much output can the application produce? Must the application ever block on logging? How long must evidence survive locally and remotely? What is the acceptable data-loss mode if the sink or disk fails? Those answers drive the configuration.

2. local versus json-file

The daemon default remains json-file for compatibility. Without max-size/max-file, it does not rotate and can consume substantial disk. Docker recommends the local driver for general use because it rotates and compresses by default. The current documented local defaults are five files with a 20 MB maximum each, before compression—about 100 MB per container.

3. Blocking versus non-blocking delivery

Docker’s default log-delivery mode is blocking: application output is delivered directly to the driver. If the driver or destination blocks, application writes can block. Non-blocking mode inserts a per-container buffer; current Docker documentation gives a default max-buffer-size of 1 MB. When that buffer is full, new messages are dropped.

This is not a performance toggle with no downside. Blocking favors log preservation but can couple application progress to the logging path. Non-blocking favors application availability but explicitly accepts message loss under sustained backpressure. Document which outcome is safer for the workload.

# Example policy only; run on a disposable container if testing.
docker run --rm   --log-driver local   --log-opt mode=non-blocking   --log-opt max-buffer-size=4m   alpine:3.22 echo 'synthetic telemetry'

4. Remote drivers and the dual-logging boundary

Modern Docker Engine automatically enables a local dual-logging cache when a configured remote logging driver cannot itself read logs. This normally keeps recent output available through docker logs. If that cache is disabled, docker logs may fail for a remote driver that cannot read back data. The external sink remains the authoritative proof of remote delivery.

Current defaults for the dual-log cache mirror local: up to five 20 MB files before compression. Network problems to a remote driver can also prevent cache writes in some failure paths; do not assume “remote failed but local always has everything.”

5. Application file versus stdout contract

A log file written only into the container writable layer has poor lifecycle semantics: replacement can discard it, a stopped/dead container can make retrieval awkward, and backup/rotation become application-specific. Prefer stdout/stderr for operational events unless the application has a strong file-based contract that is deliberately mounted, rotated, and collected.

Do not bind-mount arbitrary host log directories merely to imitate a VM. That increases host coupling and permissions complexity. If a required file log exists, treat its volume/mount and collection path as explicit persistent/external state.

6. Labels and tags for correlation—not secrets

Some logging drivers can add selected container labels or environment values to log records. Engine 29.5 also added custom attribute support to the local driver. This can improve correlation for fields such as service, environment, release, or project. Never use secret-bearing environment variables as log metadata.

docker run --rm   --label app=payments-demo   --label release=2026-09-lab   --log-driver local   --log-opt labels=app,release   alpine:3.22 echo 'event=ready'

7. Decide who owns retention

Daemon-side rotation protects host disk and gives short troubleshooting history. External retention supports longer search, compliance, and cross-host correlation. These are complementary. Define maximum local bytes/container, expected volume/day, external retention duration, and failure behavior when either tier is unavailable.

Retention is also a privacy/security control: shorter retention can reduce exposure, while longer retention can be mandatory for audit. The correct answer is workload-specific.

8. Platform and daemon-log differences

On Linux with systemd, daemon logs are commonly read through journald. Docker Desktop runs Engine components inside its VM and documents platform-specific init.log locations. Windows containers use Windows Event Log. A runbook that hard-codes /var/log/docker.log is not portable evidence.

9. Worked decision table

Knowledge check

Official references and version notes

• Docker Docs — Configure logging drivers — daemon/container driver selection, delivery modes, labels/tags, and current default-driver behavior.
• Docker Docs — Local file logging driver — default rotation/compression behavior and supported options.
• Docker Docs — JSON file logging driver — JSON framing and explicit rotation options.
• Docker Docs — Dual logging — how docker logs can remain available with remote drivers and when it does not.
• Docker CLI — docker container logs — timestamps, --since, --until, follow, and tail behavior.
• Docker CLI — docker system events — event scope, filters, JSON Lines formatting, and bounded event history.
• Docker CLI — docker container stats — CPU, memory, network, block I/O, PIDs, and Linux cache-reporting notes.
• Docker Docs — Read daemon logs — platform-specific locations and systemd/desktop guidance.
• Docker Engine 29 release notes — current Engine baseline and logging-related fixes/features.