Production Container Patterns, Immutable Delivery, Configuration, Statelessness, Sidecars, and Operational Contracts: Guided Hands-On Workflow and Core Operations

1. Lab scope and assumptions

This lab is local, disposable, and synthetic. It uses Distribution Registry 3.1.1 bound only to 127.0.0.1:50038, a fake secret, and an application published only on loopback at 127.0.0.1:18038. It does not modify daemon configuration and does not expose a remote Docker endpoint. If your environment cannot run a localhost registry, use the simulation note in section 14 and still record local image IDs.

2. Preflight and evidence directory

set -eu
LAB=dca38-lab
mkdir -p "$LAB"/{src,config,secrets,evidence}
date -u +%Y-%m-%dT%H:%M:%SZ | tee "$LAB/evidence/time-start.txt"
docker context show | tee "$LAB/evidence/context.txt"
docker version | tee "$LAB/evidence/docker-version.txt"
docker compose version | tee "$LAB/evidence/compose-version.txt"
docker buildx version | tee "$LAB/evidence/buildx-version.txt"
docker info --format 'logging={{.LoggingDriver}} driver={{.Driver}} cgroup={{.CgroupVersion}}' | tee "$LAB/evidence/docker-info.txt"

3. Synthetic application with explicit signal/state behavior

# dca38-lab/src/app.py
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
import hashlib, json, os, signal

DATA = Path('/data/state.json')
CONFIG = Path('/etc/dca38/app.conf')
SECRET = Path('/run/secrets/app_token')
VERSION = os.getenv('APP_VERSION', 'v1')

state = {'starts': 0}
if DATA.exists():
    state = json.loads(DATA.read_text())
state['starts'] += 1
DATA.write_text(json.dumps(state) + '
')
config_text = CONFIG.read_text().strip()
secret_bytes = SECRET.read_bytes()
secret_fingerprint = hashlib.sha256(secret_bytes).hexdigest()[:12]
print(f'start version={VERSION} starts={state["starts"]} config={config_text} secret_sha256_12={secret_fingerprint}', flush=True)

class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        print('request ' + (fmt % args), flush=True)
    def do_GET(self):
        if self.path == '/health':
            body = b'ok
'; code = 200
        elif self.path == '/state':
            body = (json.dumps({'version': VERSION, **state}) + '
').encode(); code = 200
        else:
            body = f'version={VERSION} message={config_text}
'.encode(); code = 200
        self.send_response(code); self.end_headers(); self.wfile.write(body)

server = HTTPServer(('0.0.0.0', 8080), Handler)
def stop(signum, frame):
    print(f'shutdown signal={signum}', flush=True)
    raise SystemExit(0)
signal.signal(signal.SIGTERM, stop)
server.serve_forever()

# dca38-lab/src/observer.py
import time, urllib.request
while True:
    try:
        body = urllib.request.urlopen('http://app:8080/health', timeout=2).read().decode().strip()
        print(f'observer health={body}', flush=True)
    except Exception as exc:
        print(f'observer health_error={type(exc).__name__}', flush=True)
    time.sleep(10)

4. Production image: non-root and narrow writable path

# dca38-lab/Dockerfile
# syntax=docker/dockerfile:1
FROM python:3.13-alpine
ARG APP_UID=10001
ARG APP_GID=10001
RUN addgroup -g ${APP_GID} app  && adduser -D -u ${APP_UID} -G app app  && mkdir -p /app /data  && chown -R app:app /app /data
WORKDIR /app
COPY --chown=app:app src/app.py src/observer.py /app/
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
USER app
EXPOSE 8080
CMD ["python", "/app/app.py"]

The image defines the intended runtime user and a writable /data directory. The Compose contract will make the root filesystem read-only and mount a named volume exactly at /data; Python bytecode writes are disabled so application code remains immutable.

5. External configuration and fake secret

# dca38-lab/config/app.conf
message=production-contract

# dca38-lab/secrets/app_token.txt
FAKE-DCA38-TOKEN-ROTATE-ME

The fake secret is intentionally non-sensitive. In local Compose, file-backed secrets are convenient runtime mounts, not a claim of encrypted-at-rest enterprise secret management. Production providers and authorization/rotation belong in the actual platform contract.

6. Start a loopback-only local registry

docker pull registry:3.1.1
docker image inspect registry:3.1.1 --format 'id={{.Id}} repoDigests={{json .RepoDigests}}' | tee dca38-lab/evidence/registry-image.txt
docker run -d --name dca38-registry   --label academy.lab=dca38   -p 127.0.0.1:50038:5000   registry:3.1.1 | tee dca38-lab/evidence/registry-container-id.txt
docker port dca38-registry | tee dca38-lab/evidence/registry-port.txt

The registry is reachable only from the local host loopback interface. This is a learning registry, not a template for an unauthenticated production registry.

7. Build, push, and capture the v1 release digest

docker build   --label academy.lab=dca38   --label academy.release=v1   -t localhost:50038/dca38/app:v1   dca38-lab | tee dca38-lab/evidence/build-v1.log
docker push localhost:50038/dca38/app:v1 | tee dca38-lab/evidence/push-v1.log
APP_V1=$(docker image inspect localhost:50038/dca38/app:v1 --format '{{index .RepoDigests 0}}')
printf '%s
' "$APP_V1" | tee dca38-lab/evidence/app-v1-ref.txt
printf 'APP_IMAGE=%s
OBSERVER_IMAGE=%s
' "$APP_V1" "$APP_V1" > dca38-lab/.env.prod

8. Define the production-like Compose contract

# dca38-lab/compose.production.yaml
services:
  app:
    image: ${APP_IMAGE:?set APP_IMAGE to digest-pinned reference}
    environment:
      APP_VERSION: v1
    user: "10001:10001"
    read_only: true
    cap_drop: ["ALL"]
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=16m
    configs:
      - source: app_config
        target: /etc/dca38/app.conf
    secrets:
      - app_token
    volumes:
      - dca38-data:/data
    networks: [front]
    ports:
      - "127.0.0.1:18038:8080"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=1).read()"]
      interval: 5s
      timeout: 2s
      retries: 5
      start_period: 5s
    restart: unless-stopped
    stop_grace_period: 10s
    cpus: 0.50
    mem_limit: 128m
    pids_limit: 64
    logging:
      driver: local
      options:
        max-size: "10m"
        max-file: "3"
    labels:
      academy.lab: dca38
      academy.contract: production

  observer:
    image: ${OBSERVER_IMAGE:?set OBSERVER_IMAGE to digest-pinned reference}
    command: ["python", "/app/observer.py"]
    user: "10001:10001"
    read_only: true
    cap_drop: ["ALL"]
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=8m
    networks: [front]
    depends_on:
      app:
        condition: service_healthy
    restart: unless-stopped
    cpus: 0.20
    mem_limit: 64m
    pids_limit: 32
    logging:
      driver: local
      options:
        max-size: "5m"
        max-file: "2"
    labels:
      academy.lab: dca38
      academy.contract: companion

configs:
  app_config:
    file: ./config/app.conf
secrets:
  app_token:
    file: ./secrets/app_token.txt
volumes:
  dca38-data:
    name: dca38-data
    labels:
      academy.lab: dca38
networks:
  front:
    name: dca38-front
    labels:
      academy.lab: dca38

9. Render before execution and predict state changes

Prediction 1: app and observer will use the recorded v1 digest reference, not an unqualified mutable release tag.
Prediction 2: app rootfs is read-only; only /data and /tmp are writable, and only /data survives replacement.
Prediction 3: app is the only service granted the fake secret and persistent volume.
Prediction 4: app publishes only 127.0.0.1:18038; observer has no host port.
Prediction 5: replacing app changes the app container ID while dca38-data remains the same volume.

docker compose   --env-file dca38-lab/.env.prod   -p dca38-prod   -f dca38-lab/compose.production.yaml   config | tee dca38-lab/evidence/compose.rendered.yaml

10. Start and inspect the contract

docker compose --env-file dca38-lab/.env.prod -p dca38-prod -f dca38-lab/compose.production.yaml up -d
APP_CID=$(docker compose --env-file dca38-lab/.env.prod -p dca38-prod -f dca38-lab/compose.production.yaml ps -q app)
OBS_CID=$(docker compose --env-file dca38-lab/.env.prod -p dca38-prod -f dca38-lab/compose.production.yaml ps -q observer)
printf '%s
' "$APP_CID" | tee dca38-lab/evidence/app-v1-container-id.txt
printf '%s
' "$OBS_CID" | tee dca38-lab/evidence/observer-container-id.txt
docker inspect "$APP_CID" \
  --format 'configuredImage={{.Config.Image}} imageId={{.Image}} user={{.Config.User}} readonly={{.HostConfig.ReadonlyRootfs}} restart={{.HostConfig.RestartPolicy.Name}} memory={{.HostConfig.Memory}} nanoCpus={{.HostConfig.NanoCpus}} pids={{.HostConfig.PidsLimit}} log={{json .HostConfig.LogConfig}} mounts={{json .Mounts}}' | tee dca38-lab/evidence/app-v1-inspect.txt
docker volume inspect dca38-data | tee dca38-lab/evidence/data-volume.json
docker network inspect dca38-front | tee dca38-lab/evidence/front-network.json

11. Verify health, external request, state, logs, and stats

docker inspect "$APP_CID" --format 'health={{.State.Health.Status}} restartCount={{.RestartCount}}' | tee dca38-lab/evidence/app-v1-health.txt
curl -fsS http://127.0.0.1:18038/health | tee dca38-lab/evidence/external-health.txt
curl -fsS http://127.0.0.1:18038/state | tee dca38-lab/evidence/state-v1.txt
docker logs --timestamps --tail 30 "$APP_CID" | tee dca38-lab/evidence/app-v1-logs.txt
docker logs --timestamps --tail 20 "$OBS_CID" | tee dca38-lab/evidence/observer-logs.txt
docker stats --no-stream "$APP_CID" "$OBS_CID" | tee dca38-lab/evidence/stats.txt

The external loopback request is deliberately separate from Docker health status. The observer’s success is also separate evidence: it tests service-name reachability from another container, not host publication.

12. Build v2 and replace only the application by digest

docker build   --build-arg APP_UID=10001 --build-arg APP_GID=10001   --label academy.lab=dca38 --label academy.release=v2   -t localhost:50038/dca38/app:v2   dca38-lab | tee dca38-lab/evidence/build-v2.log
docker push localhost:50038/dca38/app:v2 | tee dca38-lab/evidence/push-v2.log
APP_V2=$(docker image inspect localhost:50038/dca38/app:v2 --format '{{index .RepoDigests 0}}')
printf '%s
' "$APP_V2" | tee dca38-lab/evidence/app-v2-ref.txt
printf 'APP_IMAGE=%s
OBSERVER_IMAGE=%s
' "$APP_V2" "$APP_V1" > dca38-lab/.env.prod
# The APP_VERSION setting is updated in the next command only for demonstration.
APP_VERSION=v2 docker compose --env-file dca38-lab/.env.prod -p dca38-prod -f dca38-lab/compose.production.yaml up -d --no-deps app
NEW_CID=$(docker compose --env-file dca38-lab/.env.prod -p dca38-prod -f dca38-lab/compose.production.yaml ps -q app)
printf '%s
' "$NEW_CID" | tee dca38-lab/evidence/app-v2-container-id.txt
curl -fsS http://127.0.0.1:18038/state | tee dca38-lab/evidence/state-after-replacement.txt

Important: the sample Compose file hard-codes APP_VERSION: v1 to make configuration identity visible. In a real exercise, update that versioned config field to v2 before rendering/redeploying, or remove it and derive version from build metadata. Do not use an environment override as hidden production state.

13. Prove durable state and backup evidence

NEW_CID=$(docker compose --env-file dca38-lab/.env.prod -p dca38-prod -f dca38-lab/compose.production.yaml ps -q app)
docker cp "$NEW_CID:/data/state.json" dca38-lab/evidence/state-backup.json
sha256sum dca38-lab/evidence/state-backup.json | tee dca38-lab/evidence/state-backup.sha256
docker inspect "$NEW_CID" \
  --format 'configuredImage={{.Config.Image}} imageId={{.Image}} health={{.State.Health.Status}}' | tee dca38-lab/evidence/app-v2-inspect.txt

The copied synthetic state is a small evidence artifact, not a general database-backup procedure. Real databases require application-consistent backup/restore methods from Chapter 20.

14. Local-registry fallback

If your environment cannot use the loopback registry, build dca38/app:v1 and v2 locally, record each immutable local image ID, and set APP_IMAGE=sha256:<image-id> in the disposable Compose model if your local Engine accepts that reference. Record this as a simulation: a production registry digest, remote distribution, and pull verification were not exercised.

15. Bounded cleanup

docker compose --env-file dca38-lab/.env.prod -p dca38-prod -f dca38-lab/compose.production.yaml down
docker volume rm dca38-data 2>/dev/null || true
docker network rm dca38-front 2>/dev/null || true
docker image rm localhost:50038/dca38/app:v1 localhost:50038/dca38/app:v2 2>/dev/null || true
docker rm -f dca38-registry 2>/dev/null || true
printf 'Retain dca38-lab/evidence for review.
'

Every cleanup target is named by this lab. No global Docker cleanup command is required.

Knowledge check

Official references and version notes

Lab baseline: 2026-09-22. Distribution Registry 3.1.1 is the current pinned local-registry image used here. Record the observed digest after pull. The mandatory learning path is local and free; no cloud registry or paid service is required.

• Docker Docs — Use Compose in production — single-host production use, production-specific overrides, and service recreation.
• Docker Docs — Why use Compose? — current single-host deployment boundary and application-model use cases.
• Docker Docs — Compose services reference — healthcheck, restart, read-only rootfs, resource limits, logging, configs, secrets, stop signal and grace period.
• Docker Docs — Compose Deploy Specification — resource limits/reservations and deployment-oriented service controls.
• Docker Docs — Start containers automatically — restart-policy semantics and the successful-start threshold.
• Docker Docs — Resource constraints — CPU/memory governance and OOM implications.
• Docker Docs — Volumes — persistent data lifecycle independent of containers.
• Docker Docs — Storage — writable-layer versus volume/tmpfs durability boundaries.
• Docker Docs — Manage secrets securely in Compose — file-mounted secret access and environment-variable risk.
• Docker Docs — Compose secrets reference — top-level secret sources and service grants.
• Docker Docs — Configure logging drivers — default json-file behavior and recommendation for bounded local logging.
• Docker Docs — Local logging driver — automatic rotation, max-size/max-file, and daemon-owned log files.
• Docker Engine 29 release notes — current Engine-era baseline; 29.8.1 released 2026-09-15.
• Docker Compose releases — current Compose 5.5.1 baseline used for version-sensitive examples.
• Docker Buildx releases — current Buildx 0.37.1 baseline.
• BuildKit releases — current BuildKit 0.33.0 baseline.
• Docker Official Image — registry — current Distribution Registry 3.1.1 local-registry image used in the optional digest-pinning lab path.