Nodes, Agents, Labels, Executors, Workspaces, Offline Causes, and Agent Capacity Design: Guided Hands-On Workflow and Core Operations

1. Disposable topology

Use Jenkins 2.568.3 LTS, Java 21, built-in node executors 0, and two disposable agents. Each starts with one executor:

The different labels deliberately prevent the release pool from silently absorbing generic CI queue pressure. This lab demonstrates trust-aware scheduling with fake data only.

2. Attach agents without exposing bootstrap secrets

Create two Permanent Agents under Manage Jenkins → Nodes → New Node. Give each its remote root directory, labels and one executor. Use a launcher available in your environment. For a free local path, an inbound agent can connect over WebSocket using the controller-generated secret.

# Illustrative agent-side shape; obtain agent.jar and the secret from YOUR disposable controller.
# Do not paste a real secret into a Jenkinsfile or evidence packet.
java -jar agent.jar \
  -url "$JENKINS_URL" \
  -secret "$AGENT_SECRET" \
  -name 'agent-ci-1' \
  -webSocket \
  -workDir "$HOME/jenkins-agent"

Run the second agent under a separate disposable process/account/root directory. Confirm both show online before creating workload.

3. Route work and prove execution context

pipeline {
  agent none
  stages {
    stage('CI pool evidence') {
      agent { label 'linux-ci' }
      steps {
        sh '''
          set -eu
          mkdir -p out
          {
            echo "node=$NODE_NAME"
            echo "labels=$NODE_LABELS"
            echo "workspace=$WORKSPACE"
            uname -m | sed 's/^/arch=/'
          } | tee out/ci-context.txt
        '''
        archiveArtifacts artifacts: 'out/ci-context.txt', fingerprint: true
      }
    }
    stage('Release pool evidence') {
      agent { label 'release-trusted' }
      steps {
        sh '''
          set -eu
          mkdir -p out
          printf 'node=%s\nlabels=%s\nworkspace=%s\n' \
            "$NODE_NAME" "$NODE_LABELS" "$WORKSPACE" | tee out/release-context.txt
        '''
        archiveArtifacts artifacts: 'out/release-context.txt', fingerprint: true
      }
    }
  }
}

The Jenkinsfile reads scheduling metadata and writes non-sensitive evidence. It does not infer trust from the label alone; the administrator-created pool and permissions establish the intended boundary.

4. Generate bounded queue pressure

Create a disposable Pipeline capacity-hold that uses linux-ci and sleeps for 90 seconds. Trigger it twice within a few seconds. With one executor on agent-ci-1, one build should run and the other should wait.

pipeline {
  agent { label 'linux-ci' }
  options { timeout(time: 3, unit: 'MINUTES') }
  stages {
    stage('Hold one executor') {
      steps {
        echo "build=${env.BUILD_NUMBER} node=${env.NODE_NAME} workspace=${env.WORKSPACE}"
        sleep time: 90, unit: 'SECONDS'
      }
    }
  }
}

Capture both build numbers, the queue item ID for the waiting build, the node's busy/total executors and the queue reason. Do not trigger an unbounded loop of builds.

5. Compare saturation with a deliberate label mismatch

Temporarily change a disposable copy of the job to require linux-ci && arm64 when neither lab node has arm64. The queue reason should now indicate no suitable/eligible node rather than merely waiting for a busy executor.

6. Take one agent temporarily offline

After the hold build ends, mark agent-ci-1 temporarily offline with a cause such as Chapter17 maintenance drill. Queue one linux-ci build and capture the queue reason plus node offline cause. Then bring the node back online and verify the queued build starts without deleting/recreating the job.

7. Observe workspace paths and reuse

Run two sequential CI builds and compare WORKSPACE. Then, only in this disposable lab, create a file named leftover-demo.txt and show that a later build may see persistent workspace state unless the job cleans or recreates what it needs.

node('linux-ci') {
  sh '''
    set -eu
    echo "node=$NODE_NAME workspace=$WORKSPACE"
    if [ -e leftover-demo.txt ]; then echo 'leftover-present=true'; else echo 'leftover-present=false'; fi
    printf 'created-by-build=%s\n' "$BUILD_NUMBER" > leftover-demo.txt
  '''
}

This is a correctness demonstration, not permission to store secrets in workspaces. Clean the demo file afterward.

8. Tune only after measurement

Record host CPU count, memory availability, disk capacity and the observed build duration with one executor. If the host has clear spare capacity, temporarily raise agent-ci-1 to two executors, rerun two bounded hold builds and compare queue time and host/resource/build-duration evidence. Then return to one executor unless your evidence supports the new setting.

Never increase executor count on the built-in node as a capacity shortcut.

9. Small challenge: choose the layer

A build waits with “There are no nodes with the label release-trusted”, while the release agent appears online under the label trusted-release. Should you add executors, restart Jenkins, change the workspace, or repair scheduling metadata?

Expected reasoning: preserve the queue item and node labels, then repair the label expression/taxonomy. Capacity, Pipeline CPS and workspace cleanup do not solve zero eligibility.

10. Cleanup

1. Restore any deliberately changed labels and executor counts.
2. Bring agent-ci-1 online if the drill left it offline.
3. Delete leftover-demo.txt from the disposable workspace.
4. Stop agent processes and delete the disposable nodes only after exporting reviewed evidence.
5. Revoke/delete disposable API or inbound-agent credentials created only for the lab.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

• Jenkins LTS changelog and Java support policy — lab baseline Jenkins 2.568.3 LTS, Java 21; this LTS line supports Java 21 and 25.
• Managing Nodes — controller, node, agent, executor and node-monitor concepts.
• Using Jenkins agents — labels, executor counts, usage modes and distributed builds.
• Controller Isolation — do not run builds on the built-in node; set its executor count to 0 once agents exist.
• Securing Builds — isolate builds and separate trust domains instead of treating every shared agent as equivalent.
• Pipeline: Nodes and Processes step reference — node, ws, workspace context and supported label-expression operators.
• Pipeline: Nodes and Processes plugin — baseline 1479.v56e587f413a_7, requires Jenkins 2.479.3 or newer.
• SSH Build Agents plugin — optional launcher baseline 3.1097.v868116049892; use verified host keys and least-privilege agent accounts.
• Built-In Node Name and Label Migration — terminology and NODE_NAME/NODE_LABELS behavior.