Checkpoint Lab — Backup, Restore, Disaster Recovery, Controller Migration, Configuration Recovery, and Recovery Testing

1. Safety rules

• The source path must equal the exact lab path before deletion.
• The backup and checksum must be verified before deletion.
• The separate key dependency must be present in the synthetic key vault before deletion.
• The Jenkins core/Java/plugin baseline must be recorded.
• The restore target must use a different path and HTTP port.
• No cleanup occurs until the final evidence packet is copied outside the lab paths.

2. Required predictions before destruction

1. The restored controller will recover the synthetic job full name and required build record only if those files were inside the validated backup scope.
2. The fake Jenkins-managed credential will be unusable if the separately protected controller key dependency is not restored correctly.
3. The workspace does not need to survive; it should be recreated from the synthetic source.
4. The local “artifact repository” directory is an external dependency and will not magically return from a controller-only backup.
5. Changing Jenkins core/plugins during restore may invalidate the checkpoint because it mixes disaster recovery with upgrade.

Record predictions before the destructive step.

3. Create the disposable recovery target state

Your source controller should contain: job dr-lab/pipeline, at least two builds, one archived text artifact, and one fake credential used only against a local mock service. Record exact source SHA/Jenkinsfile ref if the job comes from SCM.

Required identities
-------------------
controller=jenkins-dr-source
job=dr-lab/pipeline
expected_build=<number>
source_sha=<sha>
fake_credential_id=dr-lab-token
external_artifact=/tmp/jenkins-dr-external/artifacts/<digest>.txt
restore_home=/tmp/jenkins-dr-restore
restore_port=9999

4. Preflight and version inventory

set -euo pipefail
SOURCE_HOME='/tmp/jenkins-dr-source'
BACKUP_ROOT='/tmp/jenkins-dr-backups'
KEY_ROOT='/tmp/jenkins-dr-key-vault'
RESTORE_HOME='/tmp/jenkins-dr-restore'
EVIDENCE='/tmp/jenkins-dr-evidence'
mkdir -p "$EVIDENCE"
[ "$SOURCE_HOME" = '/tmp/jenkins-dr-source' ] || exit 70
[ -d "$SOURCE_HOME" ] || exit 66
{
  printf 'preflight_utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf 'source_home=%s\n' "$SOURCE_HOME"
  printf 'restore_home=%s\n' "$RESTORE_HOME"
  printf 'restore_port=9999\n'
  printf 'expected_core=2.568.3\n'
  printf 'expected_java=21\n'
} | tee "$EVIDENCE/preflight.txt"

Add an exported plugin inventory from the disposable controller through an authorized read-only method. Do not use Script Console solely for this checkpoint.

5. Take and verify the backup

set -euo pipefail
SOURCE_HOME='/tmp/jenkins-dr-source'
BACKUP_ROOT='/tmp/jenkins-dr-backups'
BACKUP_ID="jenkins-dr-checkpoint-$(date -u +%Y%m%dT%H%M%SZ)"
BACKUP_DIR="$BACKUP_ROOT/$BACKUP_ID"
mkdir -p "$BACKUP_DIR/jenkins_home"

# Precondition: disposable source is stopped or snapshot-consistent.
cp -a "$SOURCE_HOME/." "$BACKUP_DIR/jenkins_home/"
(
  cd "$BACKUP_DIR"
  find jenkins_home -type f -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS
  sha256sum -c SHA256SUMS
)
printf '%s\n' "$BACKUP_ID" > /tmp/jenkins-dr-evidence/backup-id.txt

6. Verify the separate lab key dependency

set -euo pipefail
KEY_ROOT='/tmp/jenkins-dr-key-vault'
[ -d "$KEY_ROOT" ] || { echo 'key vault missing' >&2; exit 66; }
[ -f "$KEY_ROOT/master.key" ] || { echo 'lab controller key dependency missing' >&2; exit 67; }
# Never print the key. Record only presence, mode/ownership metadata as appropriate.
printf 'key_dependency_present=yes\n' >> /tmp/jenkins-dr-evidence/preflight.txt

7. Stop and ask: are we allowed to destroy the source?

Before deletion, verify all of these are true: backup checksum passes, backup ID is explicit, key dependency exists separately, version/plugin inventory is captured, external dependency identities are recorded, and the path guard is exact. If any check fails, the checkpoint stops.

8. Destroy only the disposable source

set -euo pipefail
SOURCE_HOME='/tmp/jenkins-dr-source'
[ "$SOURCE_HOME" = '/tmp/jenkins-dr-source' ] || exit 70
[ -d /tmp/jenkins-dr-backups ] || exit 71
[ -f /tmp/jenkins-dr-key-vault/master.key ] || exit 72
rm -rf -- "$SOURCE_HOME"
[ ! -e "$SOURCE_HOME" ] || exit 73
printf 'source_destroyed_utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" | tee -a /tmp/jenkins-dr-evidence/timeline.txt

9. Restore into a clean alternate location

set -euo pipefail
BACKUP_DIR="$(find /tmp/jenkins-dr-backups -mindepth 1 -maxdepth 1 -type d | sort | tail -n 1)"
RESTORE_HOME='/tmp/jenkins-dr-restore'
KEY_ROOT='/tmp/jenkins-dr-key-vault'
[ -n "$BACKUP_DIR" ] || exit 66
[ "$RESTORE_HOME" = '/tmp/jenkins-dr-restore' ] || exit 70
rm -rf -- "$RESTORE_HOME"
mkdir -p "$RESTORE_HOME"
cp -a "$BACKUP_DIR/jenkins_home/." "$RESTORE_HOME/"
install -m 0600 "$KEY_ROOT/master.key" "$RESTORE_HOME/secrets/master.key"
printf 'restore_copy_complete_utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" | tee -a /tmp/jenkins-dr-evidence/timeline.txt

10. Start the compatible Jenkins runtime

export JENKINS_HOME='/tmp/jenkins-dr-restore'
java -jar ./jenkins-2.568.3.war --httpPort=9999

Do not bind the restored test to the production hostname or enable real outgoing webhooks. Keep it isolated until validation is complete.

11. Validate independently

12. Measure RPO and RTO

Use recorded UTC timestamps, not memory. RPO is the time between the latest required state successfully recovered and the incident/destruction point. RTO is measured according to your stated service definition—for this lab, from destruction start until the restored controller passes all required validation checks.

Checkpoint metrics
------------------
source_destroyed_utc=
latest_recovered_required_state_utc=
achieved_rpo=
restore_start_utc=
validation_complete_utc=
achieved_rto=
rpo_target=15m
rto_target=30m
pass_or_fail=

13. Required evidence packet

14. Cleanup after review

First copy the evidence packet to a path outside the disposable resources. Then remove only exact lab directories. Keep no synthetic key material unnecessarily.

set -euo pipefail
for p in /tmp/jenkins-dr-restore /tmp/jenkins-dr-backups /tmp/jenkins-dr-key-vault /tmp/jenkins-dr-external; do
  case "$p" in
    /tmp/jenkins-dr-restore|/tmp/jenkins-dr-backups|/tmp/jenkins-dr-key-vault|/tmp/jenkins-dr-external) rm -rf -- "$p" ;;
    *) echo "refusing unexpected cleanup path" >&2; exit 70 ;;
  esac
done

15. What this checkpoint proves—and what it does not

16. What Chapter 36 adds to the production Jenkins operating model

Chapter 35 taught you to recover through ordinary controller restart and agent loss. Chapter 36 proves something stronger: when controller state is lost or moved, you can rebuild a compatible Jenkins service from protected state, independently protected key material, versioned configuration sources and explicit external dependencies—and you can demonstrate that claim with a clean-room restore.

Chapter 37 adds observability. Recovery is much faster when metrics, logs, health checks, audit trails and queue telemetry make the pre-failure and post-restore state measurable instead of anecdotal.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

Recovery procedures are version-sensitive. Re-check current primary documentation and your own controller/plugin inventory before using these patterns on a real system.

• Jenkins — Backing-up/Restoring Jenkins
• Jenkins — System configuration and JENKINS_HOME
• Jenkins — Credentials and secret protection
• Jenkins Configuration as Code
• Jenkins LTS changelog
• Jenkins LTS Upgrade Guide
• Jenkins Security Advisories
• thinBackup plugin
• Using Jenkins agents
• Controller Isolation