Backup, Restore, Disaster Recovery, Controller Migration, Configuration Recovery, and Recovery Testing: Guided Hands-On Workflow and Core Operations

1. Lab boundary

The commands are Linux-oriented because they make paths and permissions explicit. Windows users can apply the same state model with equivalent file-copy/snapshot tooling.

2. Scenario

The source controller is named jenkins-dr-source. It has one synthetic Pipeline job, one fake string credential, one archived text artifact and no production integrations. We will create a versioned backup, record its digest, keep key material in a separate protected location, restore into /tmp/jenkins-dr-restore, start Jenkins on port 9999, and verify the recovery scope.

3. Preflight: identify the exact source

set -euo pipefail
SOURCE_HOME='/tmp/jenkins-dr-source'
BACKUP_ROOT='/tmp/jenkins-dr-backups'
KEY_ROOT='/tmp/jenkins-dr-key-vault'
RESTORE_HOME='/tmp/jenkins-dr-restore'

[ -d "$SOURCE_HOME" ] || { echo "source JENKINS_HOME missing" >&2; exit 66; }
[ "$SOURCE_HOME" = '/tmp/jenkins-dr-source' ] || exit 70
mkdir -p "$BACKUP_ROOT" "$KEY_ROOT"
printf 'source=%s\nbackup_root=%s\nrestore=%s\n' "$SOURCE_HOME" "$BACKUP_ROOT" "$RESTORE_HOME"

Use exact guarded paths. Never write a cleanup command that targets /tmp, /var/lib/jenkins, a wildcarded cloud bucket or “latest backup” without identity checks.

4. Record the runtime/plugin inventory

Use UI/API/CLI as appropriate to record the exact Jenkins core, Java and installed plugin versions. The lab does not require Script Console. Store the inventory beside—but not inside—the controller key location.

recovery-manifest.txt
---------------------
backup_id=jenkins-dr-20260917T153000Z
controller=jenkins-dr-source
jenkins_core=2.568.3
controller_java=21
jenkins_home=/tmp/jenkins-dr-source
plugin_inventory_file=plugins.tsv
jcasC_ref=synthetic-ref-if-used
job_dsl_ref=none
external_dependencies=local-only
rpo_target=15m
rto_target=30m

Why record plugin versions? A future controller may start with a different dependency graph and fail to interpret stored plugin configuration. Recovery should avoid opportunistic upgrade unless the runbook intentionally combines restore and upgrade.

5. Establish a consistency point

Filesystem snapshots are preferred by Jenkins documentation when available because they capture files at one logical point and reduce mixed-time copies. In this disposable lab, stop or quiesce the source before a plain file copy. Record the start/end timestamps so the recovery point is explicit.

set -euo pipefail
printf 'backup_copy_begin_utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
# Lab assumption: source controller process is stopped or storage is snapshot-consistent here.
# Verify with your container/service manager instead of guessing.

6. Create a uniquely named backup

set -euo pipefail
SOURCE_HOME='/tmp/jenkins-dr-source'
BACKUP_ROOT='/tmp/jenkins-dr-backups'
BACKUP_ID="jenkins-dr-$(date -u +%Y%m%dT%H%M%SZ)"
BACKUP_DIR="$BACKUP_ROOT/$BACKUP_ID"
mkdir -p "$BACKUP_DIR"

# Copy the disposable source tree. For real systems choose snapshot/backup tooling
# appropriate to filesystem, scale and consistency requirements.
cp -a "$SOURCE_HOME/." "$BACKUP_DIR/jenkins_home/"
printf '%s\n' "$BACKUP_ID" > "$BACKUP_DIR/backup-id.txt"
printf 'backup_copy_end_utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" > "$BACKUP_DIR/timing.txt"
printf 'backup_dir=%s\n' "$BACKUP_DIR"

7. Separate the controller key dependency

In the conceptual production design, key material is protected separately from routine backup storage. In this synthetic lab, copy only fake lab key material into the separate KEY_ROOT; do not print it, archive it with the main backup or commit it to SCM.

set -euo pipefail
SOURCE_HOME='/tmp/jenkins-dr-source'
KEY_ROOT='/tmp/jenkins-dr-key-vault'
install -d -m 0700 "$KEY_ROOT"
# Demonstration only: exact key handling must follow your Jenkins version/runbook.
# Do not cat/echo the key.
if [ -f "$SOURCE_HOME/secrets/master.key" ]; then
  install -m 0600 "$SOURCE_HOME/secrets/master.key" "$KEY_ROOT/master.key"
fi
printf 'key_dependency_present=%s\n' "$(test -f "$KEY_ROOT/master.key" && echo yes || echo no)"

The evidence packet records that the key dependency exists and where it is controlled. It never records key contents.

8. Create integrity evidence

A checksum does not prove restorability, but it detects later byte changes. Create deterministic checksums for the backup files or for a tar archive produced by your controlled backup workflow.

set -euo pipefail
BACKUP_DIR="$(find /tmp/jenkins-dr-backups -mindepth 1 -maxdepth 1 -type d | sort | tail -n 1)"
[ -n "$BACKUP_DIR" ] || exit 66
(
  cd "$BACKUP_DIR"
  find jenkins_home -type f -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS
  sha256sum -c SHA256SUMS
)

9. Copy the completed backup off the controller

Keeping the only backup on the same disk/host leaves the same failure domain. For the mandatory local lab, a second disposable directory simulates off-controller storage. In production use separately protected storage with retention, immutability/access controls as required, and restore-path access that is tested—not assumed.

10. Restore into an isolated path

set -euo pipefail
BACKUP_DIR="$(find /tmp/jenkins-dr-backups -mindepth 1 -maxdepth 1 -type d | sort | tail -n 1)"
RESTORE_HOME='/tmp/jenkins-dr-restore'
KEY_ROOT='/tmp/jenkins-dr-key-vault'
[ "$RESTORE_HOME" = '/tmp/jenkins-dr-restore' ] || exit 70
rm -rf -- "$RESTORE_HOME"
mkdir -p "$RESTORE_HOME"
cp -a "$BACKUP_DIR/jenkins_home/." "$RESTORE_HOME/"

# Apply the separately protected lab key only at restore time.
if [ -f "$KEY_ROOT/master.key" ]; then
  install -m 0600 "$KEY_ROOT/master.key" "$RESTORE_HOME/secrets/master.key"
fi
printf 'restore_home=%s\n' "$RESTORE_HOME"

11. Start the restored controller on a non-conflicting port

Jenkins documentation demonstrates validation by pointing JENKINS_HOME at the restored directory and using a random/non-conflicting HTTP port. Use the same Jenkins WAR/core version as the source baseline where practical; do not silently upgrade during disaster recovery.

export JENKINS_HOME='/tmp/jenkins-dr-restore'
# Use the explicitly retained Jenkins 2.568.3 WAR for this dated lab baseline.
java -jar ./jenkins-2.568.3.war --httpPort=9999

If the controller fails to start, preserve the startup log and stop. Do not delete plugins/configuration until you understand the compatibility error.

12. Verify the recovery scope

13. Rehearse a migration without confusing it with upgrade

A controller migration changes host/path/container/storage/network identity. An upgrade changes Jenkins core/Java/plugins. Combining both increases variables. In the lab, migrate the restored state to an alternate directory/port while keeping the same version baseline. Record what must change intentionally—URL, reverse proxy, agent endpoints, filesystem permissions—and what must remain invariant—job/build identities, credential usability, source configuration and artifact references.

14. Cleanup

set -euo pipefail
for p in /tmp/jenkins-dr-restore /tmp/jenkins-dr-backups /tmp/jenkins-dr-key-vault; do
  case "$p" in
    /tmp/jenkins-dr-restore|/tmp/jenkins-dr-backups|/tmp/jenkins-dr-key-vault) rm -rf -- "$p" ;;
    *) echo "refusing unexpected path: $p" >&2; exit 70 ;;
  esac
done

Cleanup is performed only after the evidence packet is archived somewhere outside the paths being removed.

15. Challenge: choose the failing layer

The restored controller starts and the job exists, but the synthetic credential cannot authenticate to the local mock service. Which layer do you inspect first?

Do not recreate the job. Verify the separately protected key was restored, the credential metadata is present, and the mock external credential itself still exists. This is credential/recovery-state diagnosis, not a queue or agent problem.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

Recovery procedures are version-sensitive. Re-check current primary documentation and your own controller/plugin inventory before using these patterns on a real system.

• Jenkins — Backing-up/Restoring Jenkins
• Jenkins — System configuration and JENKINS_HOME
• Jenkins — Credentials and secret protection
• Jenkins Configuration as Code
• Jenkins LTS changelog
• Jenkins LTS Upgrade Guide
• Jenkins Security Advisories
• thinBackup plugin
• Using Jenkins agents
• Controller Isolation