Production Capstone: Build, Secure, Publish, Operate, Observe, and Recover a Complete Dockerized Application: Implementation and Automation Build-Out

1. Create the capstone workspace

mkdir ch42-capstone
cd ch42-capstone
mkdir -p config secrets evidence/backups
printf 'ch42-fake-runtime-token-v1
' > secrets/api_token.txt
chmod 600 secrets/api_token.txt
printf 'secrets/
evidence/
cosign.key
' > .gitignore
printf 'APP_GREETING=Hello-from-Chapter-42
' > config/app.env
printf 'seed
' > data-sentinel

The fake token is intentionally not committed. The committed configuration is non-secret. Do not replace the fake token with a production credential.

2. Application: health, state, config, secret, graceful shutdown

cat > app.py <<'PYAPP'
import json, os, signal, sqlite3, threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

DATA = Path('/data')
DB = DATA / 'state.db'
UNHEALTHY = DATA / 'unhealthy'
CONFIG = Path('/run/config/app.env')
SECRET = Path('/run/secrets/api_token')

def read_config():
    values = {}
    if CONFIG.exists():
        for line in CONFIG.read_text().splitlines():
            if '=' in line and not line.lstrip().startswith('#'):
                k, v = line.split('=', 1); values[k] = v
    return values

def token():
    return SECRET.read_text().strip() if SECRET.exists() else ''

def db():
    c = sqlite3.connect(DB)
    c.execute('create table if not exists kv (k text primary key, v integer not null)')
    c.execute("insert or ignore into kv values ('counter', 0)")
    c.commit(); return c

class H(BaseHTTPRequestHandler):
    def send_json(self, code, obj):
        body = json.dumps(obj, sort_keys=True).encode()
        self.send_response(code)
        self.send_header('content-type', 'application/json')
        self.send_header('content-length', str(len(body)))
        self.end_headers(); self.wfile.write(body)
    def log_message(self, fmt, *args):
        print(json.dumps({'remote': self.client_address[0], 'message': fmt % args}), flush=True)
    def do_GET(self):
        if self.path == '/health':
            if UNHEALTHY.exists(): return self.send_json(503, {'status':'unhealthy'})
            try:
                with db() as c: c.execute("select v from kv where k='counter'").fetchone()
                return self.send_json(200, {'status':'healthy'})
            except Exception as e:
                return self.send_json(503, {'status':'unhealthy','error':type(e).__name__})
        if self.path == '/':
            cfg = read_config()
            with db() as c: value = c.execute("select v from kv where k='counter'").fetchone()[0]
            return self.send_json(200, {'greeting':cfg.get('APP_GREETING','hello'), 'counter':value})
        return self.send_json(404, {'error':'not found'})
    def do_POST(self):
        if self.path != '/increment': return self.send_json(404, {'error':'not found'})
        if self.headers.get('X-API-Token','') != token(): return self.send_json(403, {'error':'forbidden'})
        with db() as c:
            c.execute("update kv set v=v+1 where k='counter'")
            value = c.execute("select v from kv where k='counter'").fetchone()[0]
        return self.send_json(200, {'counter':value})

server = ThreadingHTTPServer(('0.0.0.0', 8080), H)
def stop(*_):
    threading.Thread(target=server.shutdown, daemon=True).start()
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
print(json.dumps({'event':'ready','uid':os.getuid(),'gid':os.getgid()}), flush=True)
server.serve_forever(); server.server_close()
print(json.dumps({'event':'stopped'}), flush=True)
PYAPP

3. Dockerfile: no target-platform execution, numeric least privilege

cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1.27
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
WORKDIR /app
COPY --chown=10001:10001 app.py /app/app.py
COPY --chown=10001:10001 data-sentinel /data/.seed
USER 10001:10001
EXPOSE 8080
STOPSIGNAL SIGTERM
ENTRYPOINT ["python", "/app/app.py"]
EOF

There is no RUN instruction. Numeric --chown avoids user-name resolution at build time, and the seeded /data directory gives a new named volume a writable UID/GID 10001 ownership during Docker’s initial copy-up. The runtime root filesystem can therefore remain read-only while the named volume is writable.

4. Compose production-like runtime contract

cat > compose.yaml <<'EOF'
name: ch42cap
services:
  app:
    image: ${APP_REF:?set APP_REF to immutable registry digest}
    user: "10001:10001"
    read_only: true
    cap_drop: [ALL]
    security_opt: ["no-new-privileges:true"]
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=16m
    configs:
      - source: app_config
        target: /run/config/app.env
    secrets:
      - source: api_token
        target: api_token
    volumes:
      - type: volume
        source: state
        target: /data
    networks: [frontend, ops]
    ports:
      - "127.0.0.1:18080:8080"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=2).read()"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 5s
      start_interval: 2s
    restart: unless-stopped
    stop_grace_period: 15s
    cpus: 0.75
    mem_limit: 256m
    pids_limit: 128
    logging:
      driver: local
      options:
        max-size: "5m"
        max-file: "3"
  observer:
    image: ${APP_REF:?set APP_REF to immutable registry digest}
    user: "10001:10001"
    read_only: true
    cap_drop: [ALL]
    security_opt: ["no-new-privileges:true"]
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=8m
    networks: [ops]
    depends_on:
      app:
        condition: service_healthy
    entrypoint: ["python", "-c"]
    command:
      - "import time,urllib.request; [(print(urllib.request.urlopen('http://app:8080/health',timeout=2).read().decode(),flush=True),time.sleep(10)) for _ in iter(int,1)]"
    restart: unless-stopped
    logging:
      driver: local
      options:
        max-size: "2m"
        max-file: "2"
configs:
  app_config:
    file: ./config/app.env
secrets:
  api_token:
    file: ./secrets/api_token.txt
volumes:
  state:
    name: ${STATE_VOLUME:-ch42cap_state}
networks:
  frontend: {}
  ops:
    internal: true
EOF

The companion service has no host port and no secret. The internal ops network prevents external egress through that network; the app also joins frontend for loopback publishing. This is a teaching contract, not a substitute for production ingress TLS or cluster network policy.

5. Commit exact source revision

git init
git config user.name "DevOps Academy Lab"
git config user.email "lab@example.invalid"
git add app.py Dockerfile compose.yaml config/app.env data-sentinel .gitignore
git commit -m "capstone source"
SOURCE_SHA="$(git rev-parse HEAD)"
SOURCE_DATE_EPOCH="$(git show -s --format=%ct HEAD)"
printf 'source_sha=%s
source_date_epoch=%s
' "$SOURCE_SHA" "$SOURCE_DATE_EPOCH" | tee evidence/source.txt
test -z "$(git status --porcelain)"

6. Resolve external image identities before use

PYTHON_TAG=python:3.13-alpine
REGISTRY_TAG=registry:2
ALPINE_TAG=alpine:3.22

docker buildx imagetools inspect "$PYTHON_TAG" > evidence/python-base-imagetools.txt
PYTHON_DIGEST="$(awk '/^Digest:/ {print $2; exit}' evidence/python-base-imagetools.txt)"
BASE_REF="$PYTHON_TAG@$PYTHON_DIGEST"

docker pull "$REGISTRY_TAG"
REGISTRY_REF="$(docker image inspect "$REGISTRY_TAG" --format '{{index .RepoDigests 0}}')"
docker pull "$ALPINE_TAG"
ALPINE_REF="$(docker image inspect "$ALPINE_TAG" --format '{{index .RepoDigests 0}}')"
printf 'base=%s
registry=%s
alpine=%s
' "$BASE_REF" "$REGISTRY_REF" "$ALPINE_REF" | tee evidence/external-images.txt

Never copy example digests from course text. Resolve the registry’s current immutable subject, store it, and then use that exact reference in the run.

7. Start a loopback-only disposable registry

docker rm -f ch42-registry 2>/dev/null || true
docker volume inspect ch42_registry_data >/dev/null 2>&1 || docker volume create ch42_registry_data >/dev/null
docker run -d --name ch42-registry   --label devops.academy.lab=ch42   -p 127.0.0.1:5000:5000   -v ch42_registry_data:/var/lib/registry   "$REGISTRY_REF"
curl -fsS http://127.0.0.1:5000/v2/ | tee evidence/registry-v2.txt

8. Create a builder with registry trust scoped to the builder

cat > buildkitd.toml <<'EOF'
[registry."127.0.0.1:5000"]
  http = true
  insecure = true
EOF

docker buildx rm ch42-builder 2>/dev/null || true
docker buildx create --name ch42-builder --driver docker-container   --buildkitd-config ./buildkitd.toml --use
docker buildx inspect --bootstrap | tee evidence/builder.txt
docker buildx version | tee evidence/buildx-version.txt

The insecure registry setting belongs only to this disposable BuildKit daemon. The host Docker daemon is not globally reconfigured.

9. Build once and publish the candidate with attestations

REPO=127.0.0.1:5000/ch42/app
BUILD_TAG="$REPO:build-$SOURCE_SHA"

docker buildx build   --builder ch42-builder   --platform linux/amd64,linux/arm64   --build-arg BASE_IMAGE="$BASE_REF"   --build-arg SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH"   --label "org.opencontainers.image.revision=$SOURCE_SHA"   --label "org.opencontainers.image.source=devops-academy-ch42"   --provenance=mode=max,version=v1   --sbom=true   --metadata-file evidence/build-metadata.json   --progress=plain   --tag "$BUILD_TAG"   --push . 2>&1 | tee evidence/build.log

docker buildx imagetools inspect "$BUILD_TAG" | tee evidence/imagetools.txt
DIGEST="$(awk '/^Digest:/ {print $2; exit}' evidence/imagetools.txt)"
SUBJECT="$REPO@$DIGEST"
printf 'subject=%s
' "$SUBJECT" | tee evidence/subject.txt

The internal registry push is the single build output. Subsequent testing, scanning, signing, promotion, deployment, rollback, and restore all refer to SUBJECT; no deployment rebuild is allowed.

10. Extract SBOM and provenance from the same subject

docker buildx imagetools inspect "$SUBJECT"   --format '{{json .SBOM}}' > evidence/sbom.json
docker buildx imagetools inspect "$SUBJECT"   --format '{{json .Provenance}}' > evidence/provenance.json
python - <<'PY'
import json
for p in ['evidence/sbom.json','evidence/provenance.json']:
    x=json.load(open(p)); print(p, 'top_keys=', sorted(x)[:10])
PY

11. Test the published candidate by digest

docker pull "$SUBJECT"
docker rm -f ch42-candidate-test 2>/dev/null || true
docker run -d \
  --name ch42-candidate-test \
  --label devops.academy.lab=ch42 \
  --user 10001:10001 \
  --read-only \
  --cap-drop ALL \
  --security-opt no-new-privileges:true \
  --tmpfs /tmp:rw,noexec,nosuid,size=8m \
  --tmpfs /data:rw,nosuid,size=32m   -p 127.0.0.1:18079:8080 "$SUBJECT"
for i in 1 2 3 4 5; do curl -fsS http://127.0.0.1:18079/health && break || sleep 1; done
curl -fsS http://127.0.0.1:18079/ | tee evidence/candidate-response.json
docker rm -f ch42-candidate-test >/dev/null

This tests the current host platform manifest from the already-published index. The capstone verifies the second platform’s manifest and attestations structurally; execute both platforms only when authorized native/emulated workers are available.

12. Scan and sign the immutable subject

trivy --version | tee evidence/trivy-version.txt
trivy image --insecure --format json --output evidence/scan-amd64.json   --platform linux/amd64 "$SUBJECT"
trivy image --insecure --format json --output evidence/scan-arm64.json   --platform linux/arm64 "$SUBJECT"

export COSIGN_PASSWORD='ch42-lab-only'
cosign generate-key-pair >/dev/null
cosign sign --yes --key cosign.key --allow-insecure-registry "$SUBJECT"
cosign verify --key cosign.pub --allow-insecure-registry "$SUBJECT"   > evidence/signature-verify.json
unset COSIGN_PASSWORD

13. Promote the same digest, not a rebuild

RELEASE_TAG="$REPO:release-1"
docker buildx imagetools create --tag "$RELEASE_TAG" "$SUBJECT"
docker buildx imagetools inspect "$RELEASE_TAG" | tee evidence/release-imagetools.txt
RELEASE_DIGEST="$(awk '/^Digest:/ {print $2; exit}' evidence/release-imagetools.txt)"
test "$RELEASE_DIGEST" = "$DIGEST"
printf 'promotion_digest_equal=true
' | tee evidence/promotion.txt

If a specific local tool build refuses plain-HTTP imagetools operations, secure the local registry with TLS or perform the alias through the local Distribution API. Do not “solve” it by making a non-loopback registry globally insecure.

14. Normalize and deploy by digest

APP_REF="$SUBJECT" docker compose -p ch42cap config > evidence/compose.normalized.yaml
APP_REF="$SUBJECT" docker compose -p ch42cap up -d
APP_ID="$(APP_REF="$SUBJECT" docker compose -p ch42cap ps -q app)"
OBS_ID="$(APP_REF="$SUBJECT" docker compose -p ch42cap ps -q observer)"
printf 'app_id=%s
observer_id=%s
' "$APP_ID" "$OBS_ID" | tee evidence/runtime-ids.txt
APP_REF="$SUBJECT" docker compose -p ch42cap ps | tee evidence/compose-ps.txt
docker inspect "$APP_ID" > evidence/app-inspect.json
docker volume inspect ch42cap_state > evidence/state-volume.json
docker network inspect ch42cap_frontend > evidence/frontend-network.json
docker network inspect ch42cap_ops > evidence/ops-network.json

15. Verify runtime contract and persistent state

curl -fsS http://127.0.0.1:18080/ | tee evidence/app-before.json
curl -fsS -X POST -H 'X-API-Token: ch42-fake-runtime-token-v1'   http://127.0.0.1:18080/increment | tee evidence/increment-1.json
curl -fsS -X POST -H 'X-API-Token: ch42-fake-runtime-token-v1'   http://127.0.0.1:18080/increment | tee evidence/increment-2.json
curl -fsS http://127.0.0.1:18080/ | tee evidence/app-state.json

docker inspect "$APP_ID" \
  --format 'User={{.Config.User}} ReadonlyRootfs={{.HostConfig.ReadonlyRootfs}} CapDrop={{json .HostConfig.CapDrop}} SecurityOpt={{json .HostConfig.SecurityOpt}} Image={{.Image}}'
docker stats --no-stream "$APP_ID" | tee evidence/stats.txt
APP_REF="$SUBJECT" docker compose -p ch42cap logs --no-color --timestamps --tail=100 > evidence/runtime.log

# Replace only the app container; volume must survive.
APP_REF="$SUBJECT" docker compose -p ch42cap up -d --force-recreate --no-deps app
curl -fsS http://127.0.0.1:18080/ | tee evidence/app-after-replace.json

The counter after replacement is persistent-data evidence. The container ID should change while the subject digest and named volume remain the intended identities.

16. Implementation evidence checkpoint

Knowledge check

Official references and version notes

• Docker Engine 29 release notes — current Engine 29.8.1 behavior, fixes, known issues, and component changes.
• Docker Docs — Multi-platform builds — image indexes, native/emulated/cross-compilation strategies, and platform verification.
• Docker Docs — Build attestations — BuildKit SBOM and provenance metadata and image-store/registry requirements.
• Docker Docs — SBOM attestations — SPDX SBOM creation and inspection with Buildx imagetools.
• Docker Docs — Provenance attestations — SLSA provenance modes and inspection.
• docker buildx imagetools inspect — digest, platform, SBOM, and provenance inspection.
• Docker Docs — Use Compose in production — single-host production guidance and production-specific overrides.
• Docker Docs — Use secrets in Compose — runtime secret files and scope; local Compose secrets are not a substitute for an external encrypted secret manager.
• Docker Docs — Volumes — persistent data lifecycle, backup patterns, and container replacement semantics.
• Docker Docs — Resource constraints — CPU, memory, and runtime governance.
• Docker Docs — Configure logging drivers — bounded log retention and driver tradeoffs.
• Docker Docs — Seccomp security profiles — default syscall filtering and safer customization guidance.
• Docker Docs — Rootless mode — threat reduction and operational limits.
• dockerd reference — local registry trust behavior; 127.0.0.0/8 local registries are treated as insecure for testing, but production registries should use trusted TLS.
• Trivy releases — free local vulnerability scanner version identity used in the capstone.
• Cosign releases — signature tooling version identity used in the capstone.
• Open Container Initiative — image/runtime/distribution specifications underlying Docker artifact and runtime interoperability.