Logs, Logging Drivers, Rotation, stdout/stderr Contracts, Events, stats, and Container Observability: Concepts, Architecture, and Mental Model

1. The practical problem: one incident, several evidence planes

A container can be unhealthy while its application log is perfectly readable. A container can also emit useful stdout while a remote logging sink is unavailable, or consume memory rapidly without emitting any application error. If all of those are called “the logs,” diagnosis becomes guesswork.

The operational goal is therefore to identify the producer, transport/storage path, retention policy, and identity/timestamp for each signal. Chapter 22 measured resource controls; this chapter turns those runtime observations into a coherent incident timeline.

2. Mental model: process output, Engine events, and metrics are parallel signals

The application writes bytes to stdout and stderr. Docker connects those streams to the selected logging driver. A file-based driver stores records on the daemon host; a remote driver transports them to another system. Separately, the Engine emits object lifecycle events, and the runtime exposes resource counters that docker stats samples. The daemon itself has its own logs. Correlation happens only when you preserve shared identity and time.

  flowchart TD
    A[Application process] -->|stdout / stderr| B[Container stdio]
    B --> C[Docker logging driver]
    C --> D[Local driver-owned storage]
    C --> E[Remote logging sink]
    F[Docker Engine lifecycle] --> G[Engine event stream]
    H[Runtime / cgroups / network counters] --> I[docker stats / API]
    J[dockerd / containerd / Desktop VM] --> K[Daemon logs]
    D --> L[Correlated incident timeline]
    E --> L
    G --> L
    I --> L
    K --> L
            

3. The stdout/stderr contract

A container-friendly application normally emits operational logs to stdout and stderr and treats files inside the container writable layer as ephemeral runtime state. Docker captures stdout/stderr and passes it to the active logging driver. The driver is an Engine concern; the application should not need to know whether the operator selected local, json-file, journald, or a remote service.

stdout versus stderr is useful context, not a severity schema. Many applications write all structured logs to stdout; others reserve stderr for errors. If severity matters, include an explicit structured field such as "level":"error".

# Read-only baseline before creating anything
docker context show
docker version
docker info --format 'default_logging_driver={{.LoggingDriver}}'
docker ps --format 'table {{.ID}}	{{.Image}}	{{.Names}}	{{.Status}}'

4. Logging drivers: storage/transport policy belongs to the container configuration

5. Identity and time are part of the observation

A log line without container/image identity is ambiguous after replacement. A container name is convenient but reusable. Preserve the container ID, image reference and immutable repository digest where available, creation/start timestamps, logging driver/options, labels, and project/service labels for Compose workloads.

# Replace NAME with an existing disposable container if you have one
docker inspect NAME --format 'container={{.Id}} image_id={{.Image}} created={{.Created}} log={{json .HostConfig.LogConfig}} labels={{json .Config.Labels}}'

6. Engine events answer “what changed?”

docker events is a real-time Engine event stream, not a durable audit database. Current Docker documentation states that only the last 256 events are returned from the daemon history. Use narrow time ranges and filters during an incident, and export durable audit/monitoring data elsewhere when retention matters.

Useful container events include create, start, die, stop, restart, oom, health_status, and exec_*. Multiple filters on different keys are ANDed; repeating the same filter is ORed.

docker events   --since '10m'   --filter type=container   --filter event=die   --format '{{json .}}'

7. stats answers “what was the runtime doing at this sample?”

docker stats reports a live stream for running containers, or one sample with --no-stream. It includes CPU, memory, network I/O, block I/O, and PIDs on Linux. The CLI and Engine API are not identical representations: on Linux the CLI subtracts cache from displayed memory usage, while the API exposes raw usage plus cache fields so clients can calculate their own view.

The PIDs field includes processes and kernel threads. A large PIDs count with few visible processes may indicate a highly threaded application rather than a process leak.

docker stats --no-stream --format '{{json .}}' NAME

8. Daemon logs are a different failure domain

If the logging driver cannot write, the Engine cannot connect to a remote sink, or container lifecycle operations fail before the application runs, daemon logs may contain the relevant evidence. Location is platform-specific: Linux commonly uses journalctl -xu docker.service; Docker Desktop stores VM service logs under its platform-specific Desktop log path; Windows containers use Windows Event Log.

Do not enable debug logging casually on a shared daemon. First capture normal daemon evidence and reproduce the smallest safe scope.

9. DevOps connection: observability must be independently verifiable

A reproducible operating model records exact image identity, runtime configuration, logging policy, event time, metric sample, and external delivery proof. A dashboard screenshot alone is not enough if you cannot tie it to a container ID and image digest. Conversely, raw logs alone are not enough if you cannot tell whether the process was OOM-killed, restarted, or disconnected from its sink.

Knowledge check

Official references and version notes

• Docker Docs — Configure logging drivers — daemon/container driver selection, delivery modes, labels/tags, and current default-driver behavior.
• Docker Docs — Local file logging driver — default rotation/compression behavior and supported options.
• Docker Docs — JSON file logging driver — JSON framing and explicit rotation options.
• Docker Docs — Dual logging — how docker logs can remain available with remote drivers and when it does not.
• Docker CLI — docker container logs — timestamps, --since, --until, follow, and tail behavior.
• Docker CLI — docker system events — event scope, filters, JSON Lines formatting, and bounded event history.
• Docker CLI — docker container stats — CPU, memory, network, block I/O, PIDs, and Linux cache-reporting notes.
• Docker Docs — Read daemon logs — platform-specific locations and systemd/desktop guidance.
• Docker Engine 29 release notes — current Engine baseline and logging-related fixes/features.