Production Capstone: Build, Secure, Publish, Operate, Observe, and Recover a Complete Dockerized Application: Security, Governance, and Reliability Validation

1. Review the architecture as trust boundaries

The build client can influence source, build arguments, registry output, attestations, and cache; the Docker daemon controls runtime containers/networks/volumes; the registry stores image and attestation objects; the runtime secret source is a host file; and the application owns the SQLite data contract. A review must not collapse these into one “Docker security” box.

2. Supply-chain validation

3. Runtime least-privilege review

APP_REF="$SUBJECT" docker compose -p ch42cap config > evidence/review-compose.yaml
APP_ID="$(APP_REF="$SUBJECT" docker compose -p ch42cap ps -q app)"
docker inspect "$APP_ID" --format '{{json .Config.User}} {{json .HostConfig.CapDrop}} {{json .HostConfig.SecurityOpt}} {{.HostConfig.ReadonlyRootfs}}'
docker inspect "$APP_ID" --format '{{json .Mounts}}'
docker port "$APP_ID"
docker network inspect ch42cap_frontend > evidence/review-frontend.json
docker network inspect ch42cap_ops > evidence/review-ops.json

4. Secret and configuration governance

Check three independent questions: is the value absent from Git and the image; is it absent from docker inspect environment/config output; and is access limited to the app service? Then state the limitation: local Compose reads the fake secret from the host filesystem. Production should use an external secret system or orchestrator-backed secret mechanism appropriate to the environment, with rotation/audit controls beyond this lab.

git ls-files | grep -E 'secret|token' && exit 1 || true
docker image inspect "$SUBJECT" --format '{{json .Config.Env}}' > evidence/image-env.json
docker inspect "$APP_ID" --format '{{json .Config.Env}}' > evidence/container-env.json
grep -R 'ch42-fake-runtime-token-v1' evidence Dockerfile app.py compose.yaml config 2>/dev/null && echo 'review any hit carefully' || true

5. Network and publication governance

The app joins two networks because they serve different purposes. frontend enables the loopback publish path; ops is internal and connects the observer. Review actual endpoints rather than trusting YAML. A production deployment would normally terminate TLS and authentication at a trusted ingress/load balancer or application endpoint, with firewall/network policy outside this single-host lab.

6. Persistent-data and backup design review

The named volume is the state boundary. The app container is disposable; the SQLite database is not. Backup must therefore identify the volume, backup timestamp, source subject/config, application consistency method, archive digest, and restore target. The mandatory lab uses a controlled app stop before tar backup so SQLite is closed cleanly; production databases should use database-native backup/replication semantics appropriate to their engine.

7. Health, restart, shutdown, and availability

Review docker compose ps, health history, app logs, and stop/recreate evidence. Never equate “Up” with end-to-end availability.

8. Resource and logging governance

docker inspect "$APP_ID" \
  --format 'Memory={{.HostConfig.Memory}} NanoCPUs={{.HostConfig.NanoCpus}} PidsLimit={{.HostConfig.PidsLimit}} Log={{json .HostConfig.LogConfig}}' | tee evidence/review-resources.txt
docker stats --no-stream "$APP_ID" | tee evidence/review-stats.txt

The configured envelope is not automatically the correct production envelope. It is a bounded lab policy. A production value needs workload measurements, headroom, host capacity, and SLO context. Likewise, local log rotation limits host growth but does not replace centralized retention, search, alerting, or tamper-resistant audit storage.

9. Rollback design

Rollback means choosing an earlier verified digest, not rebuilding an older tag. Preserve the previous subject and its signature/SBOM/provenance/scan evidence. Change APP_REF to the prior digest, render normalized Compose config, recreate only intended services, and verify both runtime digest and application/data behavior. Database schema compatibility must be reviewed separately: artifact rollback cannot magically reverse incompatible state migrations.

10. Compose versus orchestrator boundary

11. Accepted limitations register

12. Operational readiness checklist

[ ] Active Docker context/endpoint documented
[ ] Engine/API/Compose/Buildx/BuildKit/scanner/signer versions recorded
[ ] Git source SHA clean and immutable
[ ] Base image digest recorded
[ ] Image index digest and expected platforms verified
[ ] SBOM/provenance extracted from subject digest
[ ] Vulnerability scan timestamp/version recorded and triaged
[ ] Signature verifies against expected identity/key
[ ] Promotion alias digest equals release subject
[ ] Normalized Compose config archived
[ ] Runtime user/capabilities/read-only/security options verified
[ ] Secret absent from Git/image/env evidence
[ ] Network endpoints and host publication verified
[ ] Named volume identified and writable owner verified
[ ] Health, restart, shutdown and resource limits tested
[ ] Log rotation configured and logs retrievable
[ ] Backup archive checksum captured
[ ] Restore into a new volume tested
[ ] Rollback digest retained and procedure documented
[ ] Incident entry commands preserve first-failure evidence
[ ] Cleanup list contains only exact ch42 identities
[ ] Accepted limitations have owner + safer production alternative

13. Version-sensitive revalidation

Before using this capstone later, re-check Engine release notes, Buildx/BuildKit attestation behavior, Compose schema fields, scanner database/tool versions, and Cosign CLI behavior. The evidence packet should make version drift obvious. Avoid “latest” as an operational requirement; use human-readable release identity plus captured actual version/digest.

Knowledge check

Official references and version notes

• Docker Engine 29 release notes — current Engine 29.8.1 behavior, fixes, known issues, and component changes.
• Docker Docs — Multi-platform builds — image indexes, native/emulated/cross-compilation strategies, and platform verification.
• Docker Docs — Build attestations — BuildKit SBOM and provenance metadata and image-store/registry requirements.
• Docker Docs — SBOM attestations — SPDX SBOM creation and inspection with Buildx imagetools.
• Docker Docs — Provenance attestations — SLSA provenance modes and inspection.
• docker buildx imagetools inspect — digest, platform, SBOM, and provenance inspection.
• Docker Docs — Use Compose in production — single-host production guidance and production-specific overrides.
• Docker Docs — Use secrets in Compose — runtime secret files and scope; local Compose secrets are not a substitute for an external encrypted secret manager.
• Docker Docs — Volumes — persistent data lifecycle, backup patterns, and container replacement semantics.
• Docker Docs — Resource constraints — CPU, memory, and runtime governance.
• Docker Docs — Configure logging drivers — bounded log retention and driver tradeoffs.
• Docker Docs — Seccomp security profiles — default syscall filtering and safer customization guidance.
• Docker Docs — Rootless mode — threat reduction and operational limits.
• dockerd reference — local registry trust behavior; 127.0.0.0/8 local registries are treated as insecure for testing, but production registries should use trusted TLS.
• Trivy releases — free local vulnerability scanner version identity used in the capstone.
• Cosign releases — signature tooling version identity used in the capstone.
• Open Container Initiative — image/runtime/distribution specifications underlying Docker artifact and runtime interoperability.