Nodes, Agents, Labels, Executors, Workspaces, Offline Causes, and Agent Capacity Design: Concepts, Architecture, and Mental Model

1. The practical problem: “Why is my build waiting?”

After Chapters 9–16, a Jenkinsfile can be valid, credentials can be scoped correctly, and every build step can still sit in the queue. Jenkins must decide where work may run. That decision combines job/Pipeline requirements, labels, node state, usage restrictions and currently available executors.

A useful capacity model therefore starts before an agent executes a command. Preserve the queue item and its reason first. A waiting task may have no matching node, a matching node may be offline, every matching executor may be busy, or policy may forbid the job from the otherwise-capable node.

2. Mental model: requirements become capacity

flowchart TD
A[Queued task + label requirements] --> B[Evaluate eligible nodes]
B --> C{Eligible online node?}
C -- no --> D[Remain queued with reason]
C -- yes --> E{Free executor?}
E -- no --> F[Wait for matching capacity]
E -- yes --> G[Allocate executor + workspace]
G --> H[Run Pipeline/Freestyle work]
H --> I[Record result/evidence]
I --> J[Cleanup policy + release executor]

The queue item is controller state. Labels and node configuration are controller-managed scheduling metadata. The agent process and tools live on the execution host. An executor is a Jenkins scheduling slot on that node. The workspace is a filesystem location allocated in that node context. None of these are interchangeable.

3. Node, agent and executor are different things

4. Labels express eligibility, not capacity

A label says what an agent represents: linux, arm64, jdk21, high-memory, release-trusted. A Pipeline can require linux && jdk21. The node step supports !, &&, ||, implication -> and equivalence <-> expressions. Wildcards and regular expressions are not label matching syntax.

Eligibility does not mean available capacity. Ten online agents may all match linux, yet if all matching executors are busy the task still waits. Conversely, adding executors cannot help when no node matches the expression.

5. Online, offline and temporarily offline

An agent can be disconnected because its process/network/launcher failed, or an administrator/monitor can make the node unavailable with a recorded offline cause. Temporarily taking a node offline is a scheduling mutation: existing work may need separate handling, while new work will not be assigned there.

Node monitors can observe disk space, temporary space, swap, clock synchronization and response time. A node that crosses configured thresholds may be taken offline. Preserve the cause; “agent offline” is not equivalent to “queue starved because all executors are busy.”

6. Workspaces are execution state, not durable evidence

Workspace paths live on agents and can be reused. Concurrent executions may receive alternate workspace paths such as suffix variants. A build that depends on an old workspace file without recreating or validating it is not reproducible.

Archive durable evidence, publish reports and use controlled caches. Clean or isolate workspaces according to trust and correctness needs. Never assume that a workspace disappearing means the build record or archived artifact should disappear, and never assume a successful workspace cleanup revokes a credential already exfiltrated elsewhere.

7. Capacity is also a trust design

Labels are often capability labels, but some capabilities imply privilege: signing keys, production network access, hardware devices or release credentials. Do not schedule untrusted pull-request code onto the same persistent agent that handles privileged release work merely because both are linux.

Use distinct pools and source/authorization boundaries. Treat the agent host, OS identity, launcher, installed tools, co-located workloads and workspace/cache policy as part of the security context.

8. Read-only inspection before tuning

// Evidence produced by an ordinary Pipeline on an agent
node('linux-ci') {
  echo "node=${env.NODE_NAME}"
  echo "labels=${env.NODE_LABELS}"
  echo "workspace=${env.WORKSPACE}"
  sh 'uname -a || true'
  sh 'java -version 2>&1 | head -n 2'
}

Also inspect Manage Jenkins → Nodes, each node's executor occupancy, connection/offline cause, and the build queue. If using the authenticated Remote API in your disposable lab, keep the API token in an environment variable and capture only non-secret JSON fields.

curl -fsS -u "$JENKINS_USER:$JENKINS_API_TOKEN" \
  "$JENKINS_URL/queue/api/json?tree=items[id,why,task[name,url]]" | jq .

curl -fsS -u "$JENKINS_USER:$JENKINS_API_TOKEN" \
  "$JENKINS_URL/computer/api/json?tree=computer[displayName,offline,temporarilyOffline,numExecutors,busyExecutors,assignedLabels[name]]" | jq .

9. Keep the built-in node out of the worker pool

Jenkins documentation recommends setting the built-in node executor count to 0 once agents exist. Build code otherwise runs in the controller trust boundary and can contend with controller CPU, heap and filesystem operations.

If the lab has only one physical computer, separate agent processes can still run under different OS identities with no read/write access to JENKINS_HOME and no privilege escalation. That improves isolation compared with running builds inside the controller process, though it is not equivalent to separate hosts.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

• Jenkins LTS changelog and Java support policy — lab baseline Jenkins 2.568.3 LTS, Java 21; this LTS line supports Java 21 and 25.
• Managing Nodes — controller, node, agent, executor and node-monitor concepts.
• Using Jenkins agents — labels, executor counts, usage modes and distributed builds.
• Controller Isolation — do not run builds on the built-in node; set its executor count to 0 once agents exist.
• Securing Builds — isolate builds and separate trust domains instead of treating every shared agent as equivalent.
• Pipeline: Nodes and Processes step reference — node, ws, workspace context and supported label-expression operators.
• Pipeline: Nodes and Processes plugin — baseline 1479.v56e587f413a_7, requires Jenkins 2.479.3 or newer.
• SSH Build Agents plugin — optional launcher baseline 3.1097.v868116049892; use verified host keys and least-privilege agent accounts.
• Built-In Node Name and Label Migration — terminology and NODE_NAME/NODE_LABELS behavior.