Checkpoint Lab — JENKINS_HOME, Filesystem Layout, Controller Configuration, System Settings, Tools, and Global Properties

1. Scenario and success criteria

You are handed a disposable Jenkins controller and told, “We have a volume, so disaster recovery is covered.” Your task is to prove or disprove that claim. You will create synthetic configuration, inventory the home directory, create a stopped-controller snapshot, recreate the controller from separated backup volumes, and verify exactly what returns.

2. Preflight and disposable resource identity

docker --version
docker image inspect jenkins/jenkins:2.568.3-jdk21 --format '{{index .RepoDigests 0}}' || \
  docker pull jenkins/jenkins:2.568.3-jdk21

# Required names for this checkpoint:
# source controller:  jenkins-ch03-controller
# source home:        jenkins-ch03-home
# config backup:      jenkins-ch03-config-backup
# secret backup:      jenkins-ch03-secret-backup
# restore home:       jenkins-ch03-restored-home
# restored controller:jenkins-ch03-restored

If the Lesson 2 source controller still exists, reuse it. Otherwise recreate it on 127.0.0.1:18083 using the official image and named volume, complete local setup, set the System Message to DevOps Academy — Chapter 03 disposable controller, and add global environment variable ACADEMY_CHAPTER=03.

3. Predict the state changes before acting

Write these predictions into jenkins-ch03-checkpoint/predictions.txt before taking the snapshot:

1. The System Message and global property will be serialized under JENKINS_HOME and survive restart/restore.
2. $JENKINS_HOME/ch03-durable-marker.txt will return after restore because it is in the backed-up home.
3. /tmp/jenkins-ch03-ephemeral.txt will not return because it belonged to the old container layer.
4. The restored controller needs the matching protected secret-key material to interpret encrypted controller data correctly.
5. External systems and agent workspaces are not recreated by this controller-home restore.

4. Capture source-controller evidence

mkdir -p jenkins-ch03-checkpoint

docker exec jenkins-ch03-controller sh -lc '
  printf "home=%s\n" "$JENKINS_HOME"
  id
  java -version 2>&1 | head -4
  stat -c "owner=%U:%G mode=%a path=%n" "$JENKINS_HOME"
  stat -c "%y %s %n" "$JENKINS_HOME/config.xml"
  grep -n -E "systemMessage|ACADEMY_CHAPTER" "$JENKINS_HOME/config.xml" || true
  printf "top-level-size-map:\n"
  du -sh "$JENKINS_HOME"/* 2>/dev/null | sort -h | tail -25
' | tee jenkins-ch03-checkpoint/source-state.txt

docker inspect --format='image={{.Config.Image}} mounts={{json .Mounts}}' \
  jenkins-ch03-controller > jenkins-ch03-checkpoint/source-container.txt

In the UI, record Jenkins core version and installed plugin names/versions. Do not copy plugin configuration or secret values into the evidence packet.

5. Create one durable and one ephemeral marker

docker exec jenkins-ch03-controller sh -lc '
  printf "durable-marker chapter03\n" > "$JENKINS_HOME/ch03-durable-marker.txt"
  printf "ephemeral-marker chapter03\n" > /tmp/jenkins-ch03-ephemeral.txt
  ls -l "$JENKINS_HOME/ch03-durable-marker.txt" /tmp/jenkins-ch03-ephemeral.txt
'

These synthetic markers make the recovery boundary visible. They are not substitutes for normal Jenkins configuration or artifact publication.

6. Create a consistent snapshot with separated key material

docker stop jenkins-ch03-controller

docker volume create jenkins-ch03-config-backup
docker volume create jenkins-ch03-secret-backup

# Replace any previous checkpoint archives inside the disposable backup volumes.
docker run --rm --entrypoint /bin/sh \
  -v jenkins-ch03-home:/source:ro \
  -v jenkins-ch03-config-backup:/backup \
  jenkins/jenkins:2.568.3-jdk21 -c '
    rm -f /backup/jenkins-home-nonsecrets.tgz &&
    cd /source &&
    tar --exclude=./secrets -czf /backup/jenkins-home-nonsecrets.tgz . &&
    sha256sum /backup/jenkins-home-nonsecrets.tgz
  ' | tee jenkins-ch03-checkpoint/config-backup-sha256.txt

docker run --rm --entrypoint /bin/sh \
  -v jenkins-ch03-home:/source:ro \
  -v jenkins-ch03-secret-backup:/backup \
  jenkins/jenkins:2.568.3-jdk21 -c '
    rm -f /backup/jenkins-secrets.tgz &&
    cd /source &&
    tar -czf /backup/jenkins-secrets.tgz secrets &&
    chmod 600 /backup/jenkins-secrets.tgz &&
    stat -c "%a %s %n" /backup/jenkins-secrets.tgz
  ' > jenkins-ch03-checkpoint/secret-backup-metadata.txt

7. Restore into a fresh volume

docker rm jenkins-ch03-controller

docker volume rm jenkins-ch03-restored-home 2>/dev/null || true
docker volume create jenkins-ch03-restored-home

# Restore ordinary controller data.
docker run --rm --user root --entrypoint /bin/sh \
  -v jenkins-ch03-config-backup:/config-backup:ro \
  -v jenkins-ch03-restored-home:/restore \
  jenkins/jenkins:2.568.3-jdk21 -c '
    cd /restore && tar -xzf /config-backup/jenkins-home-nonsecrets.tgz
  '

# Restore the separately protected secrets directory.
docker run --rm --user root --entrypoint /bin/sh \
  -v jenkins-ch03-secret-backup:/secret-backup:ro \
  -v jenkins-ch03-restored-home:/restore \
  jenkins/jenkins:2.568.3-jdk21 -c '
    cd /restore && tar -xzf /secret-backup/jenkins-secrets.tgz
  '

# Normalize ownership to the Jenkins user defined by the image.
JENKINS_UID=$(docker run --rm --entrypoint /bin/sh jenkins/jenkins:2.568.3-jdk21 -c 'id -u jenkins')
JENKINS_GID=$(docker run --rm --entrypoint /bin/sh jenkins/jenkins:2.568.3-jdk21 -c 'id -g jenkins')
docker run --rm --user root --entrypoint /bin/sh \
  -e JUID="$JENKINS_UID" -e JGID="$JENKINS_GID" \
  -v jenkins-ch03-restored-home:/restore \
  jenkins/jenkins:2.568.3-jdk21 -c 'chown -R "$JUID:$JGID" /restore'

docker run -d \
  --name jenkins-ch03-restored \
  --restart=no \
  -p 127.0.0.1:18084:8080 \
  -v jenkins-ch03-restored-home:/var/jenkins_home \
  jenkins/jenkins:2.568.3-jdk21

docker logs --tail 100 jenkins-ch03-restored

Use a different loopback port so a mistaken source controller cannot be confused with the restore. The original source volume still exists and is not mounted by the restored controller.

8. Verify each prediction independently

docker exec jenkins-ch03-restored sh -lc '
  printf "home=%s\n" "$JENKINS_HOME"
  id
  java -version 2>&1 | head -4
  grep -n -E "systemMessage|ACADEMY_CHAPTER" "$JENKINS_HOME/config.xml" || true
  printf "durable marker: "; cat "$JENKINS_HOME/ch03-durable-marker.txt"
  if [ -e /tmp/jenkins-ch03-ephemeral.txt ]; then
    echo "UNEXPECTED: old ephemeral marker exists"
    exit 2
  else
    echo "expected: old ephemeral marker absent"
  fi
' | tee jenkins-ch03-checkpoint/restored-state.txt

docker inspect --format='image={{.Config.Image}} mount={{range .Mounts}}{{.Name}}:{{.Destination}}{{end}}' \
  jenkins-ch03-restored > jenkins-ch03-checkpoint/restored-container.txt

Open http://127.0.0.1:18084, authenticate using the restored local training account, and verify the System Message and global property. Confirm the installed plugin baseline matches the source evidence. A dashboard that loads but has missing configuration is a partial restore, not success.

9. Required evidence packet

10. Deliberate failure branch

To test your diagnostic model without damaging the successful restore, create a throwaway second restore volume and omit the separate secrets archive. Start it on another loopback port, inspect startup/login/credential-related evidence, then delete it. Do not copy random keys from another controller as a shortcut.

This optional branch exists to teach causality. The required checkpoint is the successful matched restore above.

11. Cleanup / rollback

docker rm -f jenkins-ch03-restored 2>/dev/null || true

docker volume rm jenkins-ch03-restored-home 2>/dev/null || true
# Keep source/backup volumes only if you want to repeat the exercise.
# When finished intentionally:
# docker volume rm jenkins-ch03-home jenkins-ch03-config-backup jenkins-ch03-secret-backup

# Evidence packet contains no secret archive. Review before retaining or sharing.
find jenkins-ch03-checkpoint -maxdepth 1 -type f -print

12. What the checkpoint proves—and does not prove

13. Summary

• A credible Jenkins backup is a tested recovery procedure, not just a mounted volume or tar file.
• Controller configuration, secret-key recovery material, container-local state, agent workspaces, and external systems have separate ownership.
• Consistent snapshots, exact core/Java/plugin baseline, filesystem ownership, and matched keys all affect restore success.
• Evidence should prove recovery without disclosing sensitive key material.
• This chapter establishes the durable-state foundation needed before Chapter 04 adds users, authentication, authorization, folders, and least privilege.

Knowledge check

Official references and version notes

• Configuring the System — current Jenkins home-directory and global system-configuration guidance.
• Managing Jenkins — current administrative surfaces for System, Tools, Plugins, status, and troubleshooting.
• System Information — controller system properties, environment variables, plugins, memory information, and diagnostics.
• Managing Tools — built-in tool-provider concepts and global tool configuration.
• Backing-up/Restoring Jenkins — backup scope, controller-key separation, and restore validation.
• Credentials security — why Jenkins secret-key material needs protection and separation from ordinary backups.
• Storing Secrets — technical description of Jenkins encryption keys under $JENKINS_HOME/secrets.
• Controller Isolation — current guidance for keeping routine builds off the built-in controller node.
• Jenkins LTS changelog and Java Support Policy — version/runtime assumptions for this chapter.