Checkpoint Lab — Controller Resilience, Queue Recovery, Agent Loss, Pipeline Durability, Maintenance Windows, and Failure Modes

1. Safety rules

• Controller identity must be explicitly recorded.
• Built-in node must have no routine build workload.
• Agent label must be resilience-lab or another explicitly disposable label.
• All side effects use the local idempotent ledger from Lesson 2.
• Preserve evidence before each disruption and cleanup step.

2. Required predictions before starting

1. When the controller restarts during the resumable wait, the same Jenkins build number should return and continue if persistence/plugins are healthy.
2. When the disposable agent is stopped during the safe test block, the build may wait/fail/retry according to the agent failure and retry condition, but the external ledger must remain untouched until the side-effect stage.
3. If the connection is lost after the ledger write, the ledger query—not Jenkins result color—determines whether the side effect already exists.

Write these predictions into recovery-evidence/predictions.txt before the drill.

3. Preflight packet

set -euo pipefail
mkdir -p recovery-evidence
{
  printf 'utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf 'job=%s\n' "${JOB_NAME:-unknown}"
  printf 'build=%s\n' "${BUILD_NUMBER:-unknown}"
  printf 'build_url=%s\n' "${BUILD_URL:-unknown}"
  printf 'node=%s\n' "${NODE_NAME:-unknown}"
  printf 'workspace=%s\n' "${WORKSPACE:-unknown}"
  printf 'source_sha=%s\n' "$(git rev-parse HEAD)"
} | tee recovery-evidence/preflight.txt
[ "${NODE_NAME:-built-in}" != 'built-in' ] || {
  echo 'Refusing checkpoint on built-in node' >&2; exit 70;
}

From Manage Jenkins / API, add: Jenkins LTS/core, Java, installed Pipeline/Durable Task versions, controller uptime, quiet-down state, queue snapshot and exact lab agent online state.

4. Checkpoint Pipeline

Use the Lesson 2 Pipeline, but make the interruption markers explicit. The external operation ID must be derived from this build, and its query must be available independently of workspace state.

pipeline {
  agent none
  options { timestamps(); disableConcurrentBuilds() }
  stages {
    stage('Controller restart window') {
      steps {
        echo "RESTART_WINDOW build=${env.BUILD_NUMBER}"
        sleep time: 120, unit: 'SECONDS'
      }
    }
    stage('Agent-loss window') {
      steps {
        retry(count: 2, conditions: [agent()]) {
          node('resilience-lab') {
            checkout scm
            sh '''#!/usr/bin/env bash
              set -euo pipefail
              mkdir -p recovery-evidence
              printf 'attempt_start=%s node=%s\\n' "$(date -u +%FT%TZ)" "$NODE_NAME" \
                | tee -a recovery-evidence/agent-attempts.txt
              sleep 120
              printf 'attempt_end=%s node=%s\\n' "$(date -u +%FT%TZ)" "$NODE_NAME" \
                | tee -a recovery-evidence/agent-attempts.txt
            '''
          }
        }
      }
    }
    stage('Idempotent external operation') {
      steps {
        node('resilience-lab') {
          sh '''#!/usr/bin/env bash
            set -euo pipefail
            op="${JOB_NAME}#${BUILD_NUMBER}:publish"
            python3 ci/ledger.py apply "$op" | tee recovery-evidence/apply.json
            printf '%s\\n' "$op" > recovery-evidence/operation-id.txt
          '''
        }
      }
    }
    stage('Verify external operation') {
      steps {
        node('resilience-lab') {
          sh '''#!/usr/bin/env bash
            set -euo pipefail
            op="$(cat recovery-evidence/operation-id.txt)"
            python3 ci/ledger.py query "$op" | tee recovery-evidence/query.json
          '''
        }
      }
    }
  }
  post {
    always {
      node('resilience-lab') {
        archiveArtifacts artifacts: 'recovery-evidence/**', allowEmptyArchive: true, fingerprint: true
      }
    }
  }
}

5. Required drill sequence

6. Required recovery classification

For every interrupted activity, fill this table in your evidence packet.

7. Required evidence packet

8. Verification checklist

• Build N after restart is the same build number, not a fresh rerun.
• Queued Build N+1 did not start while quiet-down was active.
• Agent loss evidence names the exact disposable node and first failure timestamp.
• No routine build ran on the built-in node.
• No release artifact was rebuilt as a recovery shortcut.
• The external operation was queried by exact operation ID before any retry decision.
• Archived evidence contains no real credentials or production URLs/data.
• Quiet-down was cancelled only after the recovery state was understood.
• Cleanup removed only exact lab resources.

9. Cleanup / rollback

Cancel quiet-down on the disposable controller, restore the lab agent only if you still need it, and retain Jenkins archived evidence until the checkpoint is reviewed. Then remove the synthetic ledger by exact path and optionally delete the lab job/agent by exact name.

set -euo pipefail
ledger='/tmp/jenkins-resilience-ledger'
[ "$ledger" = '/tmp/jenkins-resilience-ledger' ] || exit 70
rm -rf -- "$ledger"

If a container image/plugin/controller version was temporarily changed for the drill, restore only through the lab’s documented configuration source. Do not perform unsupported core/plugin downgrades as a casual rollback.

10. What the checkpoint proves—and does not prove

11. What Chapter 35 adds to the Jenkins operating model

You now have a recovery vocabulary grounded in state: controller process, persisted Pipeline/queue state, agent/workspace state and external side effects are inspected independently. A restart is no longer a generic troubleshooting button, and retry is no longer a synonym for resilience.

Chapter 36 turns this into disaster-recovery engineering: backups, restores, controller migration, secret/key recovery and recovery testing. Resilience keeps work recoverable through ordinary failures; disaster recovery proves you can rebuild the controller when the controller state itself is lost or moved.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

Resilience behavior depends on Jenkins core, Pipeline plugins, agent launchers and individual steps. Re-check current primary documentation before applying these patterns to a real controller.

• Jenkins LTS changelog
• Jenkins Security Advisories
• Managing Jenkins — Prepare for Shutdown and restart guidance
• Scaling Pipelines — speed/durability settings
• Pipeline: Basic Steps — retry conditions
• Running Pipelines — restart from stage
• Durable Task plugin
• Pipeline: Groovy plugin
• Using Jenkins agents
• Controller Isolation
• Remote Access API
• Jenkins CLI