CPU, Memory, PIDs, Devices, ulimits, cgroups v2, Reservations, and Container Resource Governance: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Preserve first-failure evidence: container ID, image digest, timestamps, logs, exit code, OOMKilled, and current HostConfig.
2. Confirm host/platform and docker version/info/context; on Desktop, record VM resource settings.
3. Confirm cgroup mode and daemon/runtime state.
4. Confirm exact image/build identity; do not assume a new application release uses the same resources.
5. Inspect process/health and runtime metrics.
6. Inspect cgroup files and rlimits for the suspected controller.
7. Only then inspect unrelated storage/network/external dependencies if the evidence points there.
8. Apply the least destructive correction and rerun the smallest safe scope.

2. Failure mode: “the app crashed” but the memory cgroup killed it

A memory failure can surface as an abrupt exit, commonly code 137 when a process receives SIGKILL. Do not infer OOM from 137 alone; inspect .State.OOMKilled, logs, memory.events, the configured memory/swap values, and observed working set. A deployment wrapper or explicit kill -9 can also produce a SIGKILL-style exit.

docker inspect APP --format 'image={{.Image}} exit={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}} error={{json .State.Error}}'
docker inspect APP --format 'memory={{.HostConfig.Memory}} reservation={{.HostConfig.MemoryReservation}} memorySwap={{.HostConfig.MemorySwap}}'
docker stats --no-stream APP
docker logs --timestamps APP > first-failure.log 2>&1
# If still running and cgroup v2:
docker exec APP sh -c 'test -f /sys/fs/cgroup/memory.events && cat /sys/fs/cgroup/memory.events || true'

3. Failure mode: CPU throttling misread as an application regression

First distinguish high demand from quota throttling and host contention. Inspect cpu.max and, where available, cpu.stat counters such as throttled periods. A low CPU percentage may be perfectly consistent with a very low quota. Conversely, high CPU percentage is not automatically a problem if no latency/error SLO is violated.

docker inspect APP \
  --format 'NanoCpus={{.HostConfig.NanoCpus}} quota={{.HostConfig.CpuQuota}} period={{.HostConfig.CpuPeriod}} cpuset={{.HostConfig.CpusetCpus}} shares={{.HostConfig.CpuShares}}'
docker exec APP sh -c 'test -f /sys/fs/cgroup/cpu.max && { cat /sys/fs/cgroup/cpu.max; cat /sys/fs/cgroup/cpu.stat; } || true'

4. Intentionally broken example: safe PID exhaustion

This disposable example proves that process-creation failure belongs to the PIDs controller, not to DNS or image corruption. It attempts a fixed number of short-lived children under a low limit, preserves the logs, then removes only that container.

docker rm -f dkr22-broken-pids 2>/dev/null || true
docker run --name dkr22-broken-pids --label academy=docker-ch22   --pids-limit 8 alpine:3.22 sh -c '
    i=1
    while [ "$i" -le 14 ]; do
      sleep 5 &
      rc=$?
      echo "spawn=$i rc=$rc"
      i=$((i+1))
    done
    wait
  ' || true

docker inspect dkr22-broken-pids --format 'exit={{.State.ExitCode}} pids={{.HostConfig.PidsLimit}}'
docker logs dkr22-broken-pids 2>&1 | tee dkr22-pids-first-failure.log
docker rm dkr22-broken-pids

5. Failure mode: open-file exhaustion

Applications can report “too many open files” even when memory and PIDs look healthy. Preserve the container's Ulimits configuration and read the process limit. Engine 29's current runtime behavior makes explicit nofile verification important because defaults can follow systemd rather than older Docker-era assumptions.

docker inspect APP --format '{{json .HostConfig.Ulimits}}'
docker exec APP sh -c 'ulimit -n; cat /proc/1/limits | grep -i "open files" || true'

6. Failure mode: one device is missing, so someone suggests privileged mode

Reject the shortcut. Inspect the device requirement, daemon-host device presence, HostConfig device mapping, runtime/plugin configuration, ownership, and LSM policy. A device denial is a security/identity boundary; turning off the boundary hides the cause and grants unrelated authority.

7. Common causal mistakes

8. Performance: limits should reveal contention, not create hidden queues

Too-tight CPU quota can stretch latency; too-tight memory can turn normal cache behavior into reclaim/OOM churn; too-low PIDs or nofile limits can create burst failures. Treat the limit and its workload metrics as one experiment. A successful benchmark on an idle developer host does not prove production headroom under contention.

9. Least-destructive correction patterns

• Application leak: fix/revert the release; keep the protective memory ceiling while validating.
• Legitimate larger working set: raise the envelope only after measured evidence and host capacity review.
• CPU contention: tune quota/weight or placement based on latency and throughput evidence.
• PID/thread growth: fix unbounded concurrency, then set a ceiling with headroom.
• Open files: set an explicit tested nofile limit, not a host-wide unlimited value.
• Device requirement: use the narrow runtime/device mechanism; never blanket privileged mode.

Knowledge check

Official references and version notes

• Docker Docs — Resource constraints — CPU and memory hard/soft controls, swap semantics, and OOM guidance.
• Docker Docs — Runtime metrics — docker stats, cgroup discovery, and cgroups v2 behavior.
• Docker CLI — docker container run — CPU, memory, PID, ulimit, device, and cgroup options.
• Docker CLI — docker container stats — CPU, memory, PIDs, I/O, and one-shot JSON/template output.
• Compose Deploy Specification — resources — limits, reservations, PIDs, and device reservations.
• Docker Desktop — Resources — CPU/memory/swap capacity of the Docker Desktop Linux VM.
• Docker Engine 29 release notes — current Engine baseline and recent runtime/ulimit changes.
• Linux kernel — Control Group v2 — controller semantics for cpu.*, memory.*, and pids.*.