Checkpoint Lab — Container Filesystems, Writable Layers, docker cp, exec Workflows, Processes, Signals, and PID 1

1. Checkpoint mission and success criteria

Your goal is to prove—not merely state—that runtime filesystem changes and process behavior are separate from declared image state. You will create one digest-identified container, mutate it deliberately, record the delta, copy evidence, observe PID 1 and signal handling, rebuild the intended change into a new image, replace the original container, and prove the manual drift is gone.

2. Preflight and assumptions record

mkdir -p ch06-checkpoint/evidence
cd ch06-checkpoint

date -u +%Y-%m-%dT%H:%M:%SZ | tee evidence/00-started-at.txt
docker context show | tee evidence/01-context.txt
docker version | tee evidence/02-version.txt
docker info | tee evidence/03-info.txt
docker buildx version | tee evidence/04-buildx.txt || true
docker compose version | tee evidence/05-compose.txt || true

SOURCE='busybox:1.37.0'
docker pull "$SOURCE"
PINNED=$(docker image inspect "$SOURCE" --format '{{index .RepoDigests 0}}')
printf 'source=%s\npinned=%s\n' "$SOURCE" "$PINNED" | tee evidence/06-source-image.txt

Record any platform limitation: Windows container mode, remote context, Desktop VM boundary, unavailable Buildx plugin, or restricted daemon access. Do not fabricate unsupported observations.

3. Predict state changes before execution

Write evidence/07-predictions.txt with at least these predictions in your own words:

• The original container ID will remain stable while it runs/stops, but its writable layer will gain files after copy/exec.
• The image digest/ID will not change when runtime files are added.
• The manually added files will disappear when the original container is removed and a replacement is created from a separately built image that does not declare them.
• The new declared file will be present in the replacement because it is part of the new image.
• A graceful TERM-aware primary process should log its handler and exit before timeout escalation.

The checkpoint is incomplete if you only perform commands without comparing actual evidence to predictions.

4. Create the original container and capture immutable/runtime identity

ORIGINAL='devops-academy-ch06-checkpoint-original'

docker run -d \
  --name "$ORIGINAL" \
  --label devops-academy.lab=ch06-checkpoint \
  "$PINNED" \
  sh -c 'trap "echo checkpoint-received-TERM; exit 0" TERM; mkdir -p /academy; echo original-runtime > /academy/runtime.txt; while :; do sleep 1; done'

docker inspect "$ORIGINAL" \
  --format 'ID={{.Id}} Image={{.Image}} Status={{.State.Status}} Pid={{.State.Pid}} Started={{.State.StartedAt}}' | tee evidence/08-original-inspect.txt
docker inspect "$ORIGINAL" --format '{{json .Mounts}}' | tee evidence/09-original-mounts.json
docker top "$ORIGINAL" | tee evidence/10-original-top.txt
docker diff "$ORIGINAL" | tee evidence/11-original-diff.txt

Confirm that no unexpected host-sensitive bind mount or Docker socket is present. If it is, stop: you are not in the intended disposable lab.

5. Create controlled runtime drift

printf 'manual-copy-only\n' > manual.txt
docker cp manual.txt "$ORIGINAL":/academy/manual.txt

docker exec "$ORIGINAL" sh -c 'printf "exec-hotfix-only\n" > /academy/hotfix.txt; echo exec-pid=$$'

docker diff "$ORIGINAL" | tee evidence/12-diff-after-mutation.txt
docker exec "$ORIGINAL" sh -c 'sha256sum /academy/runtime.txt /academy/manual.txt /academy/hotfix.txt' | tee evidence/13-runtime-hashes.txt

Classify each changed path. The runtime and copied/hotfix files exist in this container, but none of them changes $PINNED.

6. Preserve non-secret evidence before shutdown

docker cp "$ORIGINAL":/academy/manual.txt evidence/manual-from-container.txt
docker cp "$ORIGINAL":/academy/hotfix.txt evidence/hotfix-from-container.txt
sha256sum manual.txt evidence/manual-from-container.txt | tee evidence/14-copy-proof.txt

docker exec "$ORIGINAL" sh -c 'echo PID1=$(cat /proc/1/stat | cut -d" " -f1); tr "\\0" " " < /proc/1/cmdline; echo; ps' | tee evidence/15-process.txt

The evidence packet contains only synthetic text. If a real incident file can contain credentials or regulated data, use your organization's secure evidence process instead of blindly copying it to a workstation.

7. Demonstrate signal and exit behavior

docker stop --timeout 5 "$ORIGINAL"
docker logs "$ORIGINAL" | tee evidence/16-original-logs.txt
docker inspect "$ORIGINAL" \
  --format 'Status={{.State.Status}} Exit={{.State.ExitCode}} Started={{.State.StartedAt}} Finished={{.State.FinishedAt}}' | tee evidence/17-original-exit.txt

docker diff "$ORIGINAL" | tee evidence/18-stopped-diff.txt

Expected evidence includes the TERM-handler log. If the container required SIGKILL, preserve that result as a diagnostic failure instead of editing the evidence to match the lesson.

8. Encode the intended change as image state

Assume only the policy intent “every replacement needs /academy/declared.txt” should survive. The copied/hotfix files were diagnostic experiments and must not be promoted.

printf 'declared-release-state\n' > declared.txt
cat > Dockerfile <<'EOF'
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
COPY declared.txt /academy/declared.txt
CMD ["sh", "-c", "trap 'echo replacement-received-TERM; exit 0' TERM; while :; do sleep 1; done"]
EOF

docker build \
  --label devops-academy.lab=ch06-checkpoint \
  --build-arg BASE_IMAGE="$PINNED" \
  -t devops-academy-ch06-checkpoint:declared . | tee evidence/19-build.txt

docker image inspect devops-academy-ch06-checkpoint:declared --format 'ImageID={{.Id}} Created={{.Created}}' | tee evidence/20-new-image.txt

Record the local image ID. If you were publishing a release, later chapters would add registry digest, SBOM/provenance/signature, scan, and promotion evidence. This checkpoint intentionally remains local and free.

9. Remove the mutated object only after evidence capture, then create a replacement

docker rm "$ORIGINAL"

REPLACEMENT='devops-academy-ch06-checkpoint-replacement'
docker run -d \
  --name "$REPLACEMENT" \
  --label devops-academy.lab=ch06-checkpoint \
  devops-academy-ch06-checkpoint:declared

docker inspect "$REPLACEMENT" --format 'ID={{.Id}} Image={{.Image}} Status={{.State.Status}} Pid={{.State.Pid}}' | tee evidence/21-replacement-inspect.txt

docker exec "$REPLACEMENT" sh -c '
  printf "declared="; cat /academy/declared.txt;
  for f in runtime.txt manual.txt hotfix.txt; do
    if [ -e "/academy/$f" ]; then echo "$f=unexpected-present"; else echo "$f=absent"; fi;
  done
' | tee evidence/22-replacement-files.txt

docker diff "$REPLACEMENT" | tee evidence/23-replacement-diff.txt

The replacement must have a different container ID. Its image identity is the newly built local image, the declared file is present, and the original runtime drift is absent. That is the observable difference between “fixed one container” and “changed the deployable artifact.”

10. Optional init evidence

If your platform supports Linux containers, run a short optional comparison using --init and save process output as evidence/24-init.txt. Mark it “not observed” if unavailable rather than treating optional evidence as mandatory.

docker run --rm --init "$PINNED" sh -c 'ps; echo PID1:; tr "\\0" " " < /proc/1/cmdline; echo' \
  | tee evidence/24-init.txt

11. Review the evidence packet

Your evidence directory should answer these questions independently:

• Which context/daemon/version ran the lab?
• Which immutable base digest was used?
• What was the original container ID and primary PID?
• Were any mounts present?
• Which paths changed and when?
• Which files were copied and what were their hashes?
• Which process was PID 1 and what signal evidence was observed?
• What was the original exit state?
• What new image identity was built?
• Did the replacement contain only declared intended state?

12. Guarded cleanup and rollback

docker ps -a --filter label=devops-academy.lab=ch06-checkpoint --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}'

docker stop --timeout 5 "$REPLACEMENT"
docker rm "$REPLACEMENT"
docker image rm devops-academy-ch06-checkpoint:declared

docker ps -a --filter label=devops-academy.lab=ch06-checkpoint
date -u +%Y-%m-%dT%H:%M:%SZ | tee evidence/25-finished-at.txt

Keep the evidence directory. Do not use docker system prune, delete Docker's data root, or remove unrelated images/volumes. If cleanup fails, inspect the exact dependency instead of escalating destructively.

13. Operational review and Chapter 07 handoff

Chapter 06 added two operational invariants to the course. First, runtime filesystem mutation is not equivalent to image state. Second, process launch form determines PID 1 and signal behavior. You can now prove both using docker diff, copy evidence, process inspection, logs, exit state, and replacement testing.

Chapter 07 moves the durable side of that model into Dockerfile fundamentals: FROM, RUN, COPY, ADD, WORKDIR, ENV, ARG, USER, CMD, and ENTRYPOINT. The checkpoint's tiny Dockerfile is therefore a bridge, not a substitute for the next chapter.

Knowledge check

Official references and version notes

• Docker storage overview — explains the ephemeral per-container writable layer and why persistent data belongs in explicit mounts rather than the container layer.
• Storage drivers and writable layers — copy-on-write concepts, writable-layer behavior, and the Engine 29 distinction between the containerd image store and classic storage-driver examples.
• docker container diff — reports added, changed, and deleted paths in the container filesystem relative to its initial image-backed state.
• docker container cp — current copy semantics, stopped-container support, destination ownership, archive mode, symlink handling, and path rules.
• docker container exec — starts a new command only while the container's primary PID 1 is running and does not make image changes durable.
• docker container run — current runtime options including --init, stop signals/timeouts, mounts, TTY behavior, and security controls.
• docker container stop — graceful signal delivery and timeout-to-SIGKILL escalation.
• docker container attach — documents special PID 1 signal behavior and terminal signal-proxy implications.
• JSONArgsRecommended build check — why exec-form CMD/ENTRYPOINT avoids an unintended shell parent and improves signal handling.
• Docker build best practices — entrypoint scripts should normally exec the final application so it becomes PID 1 and receives signals directly.
• Bind mounts — mounting over an existing container path obscures image/writable-layer content until the container is recreated without that mount.
• Docker Engine 29 release notes — Engine 29.8.1 is the current patch baseline at verification time; recent 29.x releases also include multiple docker cp security and compatibility fixes.