Health Checks, Restart Policies, Dependency Health, Graceful Shutdown, Signals, and Self-Healing Patterns: Configuration, Design Choices, and Tradeoffs

1. Exec versus shell healthcheck form

Exec form runs the probe directly and avoids shell parsing. Shell form is appropriate when you deliberately need operators such as &&, variable expansion, or multiple checks. The shell form also introduces the shell itself as a dependency and changes signal/quoting behavior.

2. Choose probe depth to match the failure you need to detect

PID existence is extremely cheap but often too shallow: Docker already knows whether PID 1 is alive. A local protocol check proves more, such as “the HTTP handler returns a valid response.” A deep dependency check can prove end-to-end behavior but may mark many containers unhealthy because one shared dependency is down, causing synchronized recovery storms.

3. Tune timing from measured startup and failure budgets

Probe timing should be derived from expected startup duration, request latency, transient-error tolerance, and acceptable failure-detection time. An aggressive one-second interval across hundreds of containers can create real load; a five-minute interval can make failure detection useless.

4. Restart policy is a process-lifecycle choice

Use on-failure when non-zero exits represent recoverable failures and a bounded retry count can prevent endless loops. always can be appropriate for a service expected to run whenever Docker runs. unless-stopped preserves an operator’s intentional stop across daemon restarts. None of these policies fixes a process that is alive but unhealthy.

5. Application retries versus Compose dependency gating

Compose service_healthy is useful for initial startup ordering. Applications still need bounded retries/backoff, connection re-establishment, and sensible timeouts because dependencies can restart later. The two mechanisms are complementary: gating prevents an avoidable cold-start race; application resilience handles the long-running system.

6. Docker health versus external load-balancer health

A local probe should usually be fast, local, and cheap. An external health check can validate the route users actually take, including DNS, host networking, reverse proxy, TLS, and possibly a broader dependency chain. Use both only when each answers a different operational question.

7. Decision table

8. Keep adjacent state domains distinct

• Host/client/context: which Docker daemon receives the command.
• Image: inherited or declared healthcheck, STOPSIGNAL, probe tooling.
• Container runtime: current PID, exit code, restart count, stop timeout.
• Health: probe command, timing, status, failing streak, probe output.
• Dependency: Compose startup conditions and application retries.
• External availability: published path, DNS, firewall, proxy, TLS, upstreams.

Knowledge check

Official references and version notes

• Dockerfile reference — HEALTHCHECK — health states, timing, exit codes, output retention, and health-status events.
• Dockerfile reference — STOPSIGNAL — image-defined graceful-stop signal.
• Docker CLI — docker container stop — stop signal and timeout/escalation behavior.
• Docker Docs — Restart policies — on-failure, always, unless-stopped, successful-start threshold, and manual-stop behavior.
• Docker CLI — restart policies — backoff and retry semantics.
• Docker Compose — startup/shutdown order — service_started, service_healthy, and service_completed_successfully.
• Compose services — healthcheck — Compose healthcheck fields and override behavior.
• Compose services — depends_on — dependency conditions and explicit Compose restart propagation.
• Docker Engine 29 release notes — current Engine baseline and healthcheck fixes.