Production Capstone: Build, Secure, Publish, Operate, Observe, and Recover a Complete Dockerized Application: Final Operational Review and Handoff

1. Final evidence directory contract

find evidence -maxdepth 3 -type f -print | sort > evidence/final-file-list.txt
sha256sum evidence/backups/state.tgz > evidence/final-backup-checksum.txt
printf 'final_review_utc=%s
' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" > evidence/final-review.txt
printf 'source_sha=%s
subject=%s
release_tag=%s
' "$SOURCE_SHA" "$SUBJECT" "$RELEASE_TAG" >> evidence/final-review.txt
printf 'context=%s
' "$(docker context show)" >> evidence/final-review.txt

2. Independent clean-shell verification

Open a new shell in the project directory. Do not rely on SUBJECT, SOURCE_SHA, or other exported variables from previous lessons. Reconstruct identity from evidence first.

set -eu
SUBJECT="$(sed -n 's/^subject=//p' evidence/subject.txt)"
SOURCE_SHA="$(sed -n 's/^source_sha=//p' evidence/source.txt)"
RELEASE_DIGEST="$(awk '/^Digest:/ {print $2; exit}' evidence/release-imagetools.txt)"
SUBJECT_DIGEST="${SUBJECT##*@}"

test "$(git rev-parse HEAD)" = "$SOURCE_SHA"
test -z "$(git status --porcelain)"
test "$RELEASE_DIGEST" = "$SUBJECT_DIGEST"

docker buildx imagetools inspect "$SUBJECT" > /tmp/ch42-verify-imagetools.txt
grep -q 'linux/amd64' /tmp/ch42-verify-imagetools.txt
grep -q 'linux/arm64' /tmp/ch42-verify-imagetools.txt

cosign verify --key cosign.pub --allow-insecure-registry "$SUBJECT" > /tmp/ch42-signature.json
APP_ID="$(APP_REF="$SUBJECT" docker compose -p ch42cap ps -q app)"
test -n "$APP_ID"
docker inspect "$APP_ID" --format '{{.Image}} {{.Config.User}} {{.HostConfig.ReadonlyRootfs}}'
curl -fsS http://127.0.0.1:18080/health
sha256sum -c evidence/backups/state.tgz.sha256
printf 'independent_verification=PASS
' | tee evidence/independent-verification.txt

This checklist proves identity and selected controls from a clean shell. It deliberately does not infer “secure production” from one pass; it verifies the evidence claims this capstone actually makes.

3. Compare original and recovered state

cat evidence/backups/original-after-backup.json
cat evidence/incidents/recovered-state.json
cat evidence/incidents/recovered-runtime.txt
cat evidence/incidents/recovery-time.txt

Record whether the counter matches the backup point and whether any post-backup write is absent as expected. That observation closes the RPO/RTO exercise.

4. Final operational review questions

5. Upgrade and patch process

1. Read Engine/Compose/Buildx/BuildKit/scanner/signer release notes and advisories.
2. Capture current production digests, versions, daemon/context, backups, and rollback subject.
3. Update reviewed base/tool references in a branch; never mutate the running container in place.
4. Build a new immutable candidate from an exact source revision.
5. Test, scan, generate/inspect attestations, sign, and review exceptions.
6. Promote the same digest; deploy by digest in a staging/controlled environment.
7. Verify health, data compatibility, logs/resources, and external request path.
8. Roll forward or roll back by verified digest; handle database migration rollback separately.
9. Archive evidence and update the accepted-risk/readiness record.

6. Retention and backup policy template

Release evidence retention: <organization policy>
Image digests retained: current + rollback window
SBOM/provenance/signature/scan: retain with release record
Application logs: bounded local buffer + central retention in production
Backups: frequency based on RPO; immutable/off-host copy in production
Restore drills: scheduled and measured; archive checksum + application validation required
Secret rotation: owner + cadence + revocation evidence
Vulnerability exceptions: owner + reason + expiry + compensating control
Incident evidence: preserve IDs/digests/timestamps/logs before destructive recovery

7. Incident entry points

docker context show
docker version
docker info
APP_REF="<release@sha256:...>" docker compose -p ch42cap ps
docker inspect <exact-container-id>
docker logs --timestamps --since 15m <exact-container-id>
docker events --since 15m --filter container=<exact-container-id>
docker network inspect <exact-network>
docker volume inspect <exact-volume>
docker stats --no-stream <exact-container-id>
docker buildx imagetools inspect <release@sha256:...>

These are evidence entry points, not a substitute for a runbook. The next action depends on which layer owns the symptom.

8. Final cleanup preflight

Before deleting anything, inventory only expected lab identities and make sure the final evidence packet is copied wherever you intend to retain it.

docker ps -a --filter label=devops.academy.lab=ch42
docker ps -a --filter name=ch42
docker volume ls --filter name=ch42
docker network ls --filter name=ch42
docker buildx ls | grep ch42 || true
ls -lh evidence/backups/state.tgz evidence/backups/state.tgz.sha256 cosign.pub

9. Bounded cleanup — recovery project first

SUBJECT="$(sed -n 's/^subject=//p' evidence/subject.txt)"
STATE_VOLUME=ch42_state_restore APP_REF="$SUBJECT"   docker compose -p ch42restore -f compose.yaml -f compose.recovery.yaml down

docker volume rm ch42_state_restore 2>/dev/null || true

10. Bounded cleanup — primary runtime and persistent lab data

APP_REF="$SUBJECT" docker compose -p ch42cap down
# Explicitly remove only the capstone volume after backup/restore evidence has been accepted.
docker volume rm ch42cap_state 2>/dev/null || true

docker rm -f ch42-registry 2>/dev/null || true
docker volume rm ch42_registry_data 2>/dev/null || true
docker buildx rm ch42-builder 2>/dev/null || true

docker compose down intentionally does not use --volumes; persistent data is removed in a separate, named step after the recovery review. No Docker-wide prune is needed.

11. Bounded cleanup — temporary credentials and local objects

rm -f cosign.key
# Keep cosign.pub and evidence if retaining the review packet.
docker image rm "$SUBJECT" 2>/dev/null || true
# Verify that the intended lab resources are gone; do not delete anything unexpected.
docker ps -a --filter name=ch42
docker volume ls --filter name=ch42
docker network ls --filter name=ch42
docker buildx ls | grep ch42 || true

12. Production handoff

13. What the complete Docker course now gives you

You can reason from Docker client/context and daemon architecture through images, BuildKit, Compose, networking, storage, resource governance, observability, security, rootless/user namespaces, secrets, vulnerability management, SBOM/provenance/signatures, containerd internals, daemon/API automation, CI isolation, release promotion, production contracts, Swarm estates, performance, and incident response. The capstone connects those topics through one requirement-to-evidence chain.

The next platform courses should not erase these Docker fundamentals. Kubernetes, Helm, and OpenShift add scheduling, reconciliation, policy, and cluster abstractions, but immutable image identity, least privilege, persistent-data semantics, supply-chain evidence, observability, and evidence-first troubleshooting remain the underlying operating discipline.

14. Course completion criterion

The Docker course is complete when you can hand the packet to another engineer and they can independently verify the release subject, source/build lineage, security and runtime controls, state and backup/restore behavior, observed incident evidence, accepted limitations, and cleanup boundary without relying on undocumented facts in your head or mutable-only tags.

Knowledge check

Official references and version notes

• Docker Engine 29 release notes — current Engine 29.8.1 behavior, fixes, known issues, and component changes.
• Docker Docs — Multi-platform builds — image indexes, native/emulated/cross-compilation strategies, and platform verification.
• Docker Docs — Build attestations — BuildKit SBOM and provenance metadata and image-store/registry requirements.
• Docker Docs — SBOM attestations — SPDX SBOM creation and inspection with Buildx imagetools.
• Docker Docs — Provenance attestations — SLSA provenance modes and inspection.
• docker buildx imagetools inspect — digest, platform, SBOM, and provenance inspection.
• Docker Docs — Use Compose in production — single-host production guidance and production-specific overrides.
• Docker Docs — Use secrets in Compose — runtime secret files and scope; local Compose secrets are not a substitute for an external encrypted secret manager.
• Docker Docs — Volumes — persistent data lifecycle, backup patterns, and container replacement semantics.
• Docker Docs — Resource constraints — CPU, memory, and runtime governance.
• Docker Docs — Configure logging drivers — bounded log retention and driver tradeoffs.
• Docker Docs — Seccomp security profiles — default syscall filtering and safer customization guidance.
• Docker Docs — Rootless mode — threat reduction and operational limits.
• dockerd reference — local registry trust behavior; 127.0.0.0/8 local registries are treated as insecure for testing, but production registries should use trusted TLS.
• Trivy releases — free local vulnerability scanner version identity used in the capstone.
• Cosign releases — signature tooling version identity used in the capstone.
• Open Container Initiative — image/runtime/distribution specifications underlying Docker artifact and runtime interoperability.