Production Container Patterns, Immutable Delivery, Configuration, Statelessness, Sidecars, and Operational Contracts: Configuration, Design Choices, and Tradeoffs
Choose stateless versus stateful boundaries, companion-service patterns, immutable replacement, configuration mechanisms, single-host Compose scope, rollback, and evidence requirements from explicit operational tradeoffs.
Learning objectives
- Choose between stateless and stateful runtime patterns from data lifecycle and recovery requirements.
- Decide when a companion service is justified versus a library, host service, or orchestrator-native facility.
- Compare immutable replacement with in-place mutation and define rollback around known image/config/data identities.
- Place single-host Compose in its correct production scope and identify when scheduling/high-availability needs exceed it.
- Create a decision record with prerequisites, affected state, observability, rollback, and trust boundaries.
1. Design starts with failure and replacement
A production pattern is useful only if it predicts what happens when a process exits, a host reboots, an image is replaced, a volume fills, a credential rotates, or a dependency is unavailable. Design choices should therefore be expressed as state and recovery decisions, not as “best-practice” labels.
2. Stateless app versus stateful service
| Question | Stateless application | Stateful service |
|---|---|---|
| Irreplaceable local data? | No | Yes or externally coordinated |
| Replacement expectation | new instance can start from image + config | must reattach/restore correct data state |
| Scaling | instances usually interchangeable | coordination/consistency may constrain scaling |
| Backup focus | external dependencies/config | volume/database/application-consistent state |
| Rollback risk | image/config compatibility | plus schema/data compatibility |
3. Immutable replacement versus in-place mutation
| Choice | Benefits | Risks / prerequisites |
|---|---|---|
| Digest replacement | reproducible, auditable, rollbackable artifact identity | requires build/publish pipeline and externalized state/config |
| Exec patch | fast local experiment only | drift, no durable provenance, lost on replacement, hard rollback |
| Package update inside running container | none for immutable delivery | changes release bytes outside build evidence |
4. Companion service decision
| Need | Prefer companion when… | Prefer another pattern when… |
|---|---|---|
| Telemetry | agent must observe this workload/network with narrow access | host/daemon collector already covers requirement |
| Proxy | per-app protocol boundary/config is truly lifecycle-coupled | shared ingress/service mesh owns routing |
| File transform | separate process needs a clearly scoped shared volume | same process/library call is simpler and safer |
| Secrets refresh | provider requires local file rendering with narrow identity | app can use provider SDK directly without extra process |
5. Companion contract checklist
- Exact image digest and runtime user.
- Only required network memberships.
- Only required read-only/read-write mounts.
- Independent health/log/resource limits.
- No Docker socket or host namespace unless the architecture explicitly requires privileged infrastructure authority.
- Defined behavior when the primary app is unavailable.
- Upgrade/rollback compatibility rules between app and companion.
6. Configuration mechanism and rollback
| Mechanism | Good fit | Rollback evidence |
|---|---|---|
| Versioned Compose config/file | non-secret small runtime config on a single host | config checksum/revision + prior file |
| Compose secret file | local/synthetic secret injection | secret version/rotation record; never value in logs |
| Environment variable | simple non-secret scalar | rendered config with secret values excluded |
| External provider | production credentials/rotation/policy | provider version/identity/audit evidence |
7. Resource strategy: measure, then bound
Start with representative load and observe CPU, memory, PIDs, I/O and latency. Apply limits that protect the host while allowing expected peaks. A hard memory limit can intentionally turn uncontrolled host pressure into a bounded container OOM, but the application still needs recovery and capacity planning. CPU quotas can introduce throttling; a “slow app” may actually be a resource-governance symptom.
8. Health strategy: cheap, local, and meaningful
Use a probe that confirms the application can perform its minimum local service, not a deep dependency transaction that overloads every downstream system. A deep external synthetic check belongs at another observability layer. Avoid probing only “PID exists,” but also avoid making every healthcheck call the entire dependency graph.
9. Compose single-host versus orchestrator boundary
| Requirement | Single-host Compose | Orchestrator / platform |
|---|---|---|
| Declarative multi-container app on one host | strong fit | also possible |
| Replace/restart on same Engine | supported | supported |
| Automatic reschedule after host loss | not provided | core scheduler capability |
| Rolling replicas across hosts | not Compose up semantics |
service/orchestrator capability |
| Cluster secret/config distribution | local mechanisms only | platform-specific distributed mechanisms |
| Placement/anti-affinity/quorum | outside Compose single-host scope | scheduler/stateful platform concern |
10. Release and rollback contract
A rollback record should name the previous image digest, configuration revision, secret compatibility requirement, and persistent-data/schema compatibility. “Retag latest back” is not enough because it hides which bytes and state transition are being reversed. If an image rollback cannot read the current data schema, the release contract must include a forward-fix or data migration plan.
11. Decision table: worked production scenario
| Decision | Selected approach | Prerequisites | Observable proof |
|---|---|---|---|
| Artifact | registry digest | trusted registry + build evidence | running .Config.Image + repo digest |
| State | named volume for synthetic durable file | backup/retention owner | volume ID + checksum/backup |
| Runtime user | UID/GID 10001, no added caps | owned data path | inspect + id |
| Filesystem | read-only root + tmpfs + data volume | app writes only declared paths | mounts + failed undeclared write test |
| Recovery | unless-stopped on single host | Engine available; app tolerates restart | restart count + events + state continuity |
| Companion | network-only health observer | no secret/data mount required | inspect grants + observer logs |
| Orchestration | Compose on one host | host availability accepted | documented limitation; no HA claim |
12. Cost and developer-experience tradeoffs
Production hardening can increase local friction if every environment must exactly reproduce production. Keep the contract portable but allow development-specific conveniences in Chapter 37’s dev definition. The important boundary is that release bytes and production runtime controls are explicit, reviewed, and independently verifiable.
13. Platform prerequisites to record
Record before approving the design:
- Engine/CLI/Compose versions and OS/architecture
- image/index digest and platform
- cgroup/resource-control support
- logging driver availability and disk budget
- volume driver and backup semantics
- local secret/config implementation versus external provider
- host exposure/firewall/load-balancer boundary
- whether single-host downtime is acceptable
- recovery objective (RTO/RPO) and state compatibility constraints
Knowledge check
When should a workload be called stateless?
When any individual instance can be replaced without losing irreplaceable business state because durable state lives outside its disposable container layer.
What is a key danger of rolling back only the image digest?
The older image may be incompatible with current configuration, secret expectations, or persistent-data/schema state.
When is a companion container a poor choice?
When the function is better provided by a library, host/platform service, or shared infrastructure and the companion would require broad mounts/secrets/privileges.
What production capability does single-host Compose not provide?
Automatic rescheduling across hosts after node loss and other cluster scheduler semantics.
Why should resource limits be tied to evidence?
Too-low limits cause throttling/OOM; absent limits risk host exhaustion. Representative measurements make the envelope defensible.
Official references and version notes
Design baseline: 2026-09-22. Compose is treated as a single-host production model. The chapter does not imply that service-level health, restart policy, or companion containers replace multi-host orchestration, load balancing, backup systems, or external secret management.
- Docker Docs — Use Compose in production — single-host production use, production-specific overrides, and service recreation.
- Docker Docs — Why use Compose? — current single-host deployment boundary and application-model use cases.
- Docker Docs — Compose services reference — healthcheck, restart, read-only rootfs, resource limits, logging, configs, secrets, stop signal and grace period.
- Docker Docs — Compose Deploy Specification — resource limits/reservations and deployment-oriented service controls.
- Docker Docs — Start containers automatically — restart-policy semantics and the successful-start threshold.
- Docker Docs — Resource constraints — CPU/memory governance and OOM implications.
- Docker Docs — Volumes — persistent data lifecycle independent of containers.
- Docker Docs — Storage — writable-layer versus volume/tmpfs durability boundaries.
- Docker Docs — Manage secrets securely in Compose — file-mounted secret access and environment-variable risk.
- Docker Docs — Compose secrets reference — top-level secret sources and service grants.
- Docker Docs — Configure logging drivers — default json-file behavior and recommendation for bounded local logging.
- Docker Docs — Local logging driver — automatic rotation, max-size/max-file, and daemon-owned log files.
- Docker Engine 29 release notes — current Engine-era baseline; 29.8.1 released 2026-09-15.
- Docker Compose releases — current Compose 5.5.1 baseline used for version-sensitive examples.
- Docker Buildx releases — current Buildx 0.37.1 baseline.
- BuildKit releases — current BuildKit 0.33.0 baseline.
- Docker Official Image — registry — current Distribution Registry 3.1.1 local-registry image used in the optional digest-pinning lab path.
Keep the academy open
Support free, practical DevOps education.
Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0Send only Ethereum/ERC-20 compatible assets to this
address.