Performance Engineering: Executors, Queueing, Heap, Garbage Collection, Disk I/O, Workspaces, and Controller Load: Guided Hands-On Workflow and Core Operations

1. Scenario and safety boundary

You operate a disposable controller called jenkins-perf-lab with the built-in node set to 0 executors. One disposable Linux agent is labeled perf-linux. The job perf-lab/bounded-workload performs synthetic CPU-light work, bounded filesystem work, modest log output and a simulated external wait. Nothing points at production systems.

2. Preflight and record the baseline

set -euo pipefail
OUT='/tmp/jenkins-perf-lab-evidence'
mkdir -p "$OUT"
date -u +baseline_utc=%Y-%m-%dT%H:%M:%SZ | tee "$OUT/baseline.txt"
java -version 2>&1 | tee -a "$OUT/baseline.txt"
printf 'jenkins_lts=2.568.3\nagent=perf-agent-01\nlabel=perf-linux\nexecutors=1\n' | tee -a "$OUT/baseline.txt"

3. Use a bounded, parameterized synthetic Pipeline

The Pipeline below keeps the work intentionally small. The point is not to benchmark hardware; it is to learn measurement discipline.

pipeline {
  agent { label 'perf-linux' }
  parameters {
    choice(name: 'MODE', choices: ['serial', 'parallel2'], description: 'bounded workload shape')
  }
  options { timestamps(); disableConcurrentBuilds(abortPrevious: false) }
  stages {
    stage('Identity') {
      steps {
        sh '''set -eu
          printf 'job=%s\n' "$JOB_NAME"
          printf 'build=%s\n' "$BUILD_NUMBER"
          printf 'node=%s\n' "$NODE_NAME"
          printf 'workspace=%s\n' "$WORKSPACE"
          printf 'mode=%s\n' "$MODE"
          date -u +started_utc=%Y-%m-%dT%H:%M:%SZ
        '''
      }
    }
    stage('Bounded work') {
      steps {
        script {
          def task = {
            sh '''set -eu
              mkdir -p perf-data
              dd if=/dev/zero of=perf-data/payload.bin bs=1M count=8 status=none
              sha256sum perf-data/payload.bin > perf-data/payload.sha256
              sleep 12
              printf 'bounded-complete\n'
            '''
          }
          if (params.MODE == 'parallel2') {
            parallel a: task, b: task
          } else {
            task()
            task()
          }
        }
      }
    }
    stage('Evidence') {
      steps {
        sh '''set -eu
          du -sk . | tee workspace-kib.txt
          find perf-data -maxdepth 1 -type f -printf '%f %s bytes\n' | sort > files.txt
        '''
        archiveArtifacts artifacts: 'workspace-kib.txt,files.txt,perf-data/*.sha256', fingerprint: true
      }
    }
  }
}

Because the two tasks create the same bounded file path, a production-quality parallel version would allocate branch-specific directories. For this training example, Jenkins gives parallel branches distinct workspace suffixes only in some allocation patterns, so the safer exercise is to put each branch inside an explicit subdirectory if your environment shares the same workspace. Do not rely on accidental filesystem separation.

4. Capture a repeatable measurement snapshot

set -euo pipefail
JENKINS_URL='http://127.0.0.1:8080'
OUT='/tmp/jenkins-perf-lab-evidence'
TS="$(date -u +%Y%m%dT%H%M%SZ)"
# Add approved read-only lab authentication through your environment when required.
curl -fsS "$JENKINS_URL/queue/api/json?tree=items[id,inQueueSince,why,task[name,url]]" > "$OUT/queue-$TS.json"
curl -fsS "$JENKINS_URL/computer/api/json?tree=computer[displayName,offline,numExecutors,assignedLabels[name]]" > "$OUT/nodes-$TS.json"
if curl -fsS "$JENKINS_URL/prometheus/" > "$OUT/prometheus-$TS.txt"; then
  printf 'prometheus_snapshot=%s\n' "$TS" >> "$OUT/timeline.txt"
fi

For controller JVM evidence, prefer the mechanisms already approved for your lab. If you have local process access and a full JDK, commands such as jcmd <pid> VM.info, jcmd <pid> GC.heap_info and a thread dump can be useful. Treat their output as potentially sensitive and do not publish it unreviewed.

5. Experiment A — baseline: one executor, serial workload

1. Confirm perf-agent-01 has exactly one executor.
2. Trigger exactly four MODE=serial builds within a short interval.
3. Capture queue snapshots every 5–10 seconds until all builds have started.
4. Record build start/end times, node name, workspace, queue reason and controller metrics.
5. After completion, record artifact sizes and workspace size.

Expected shape: one build runs while later builds wait for the single eligible executor. The exact queue wait is hardware/environment dependent; the lesson does not hard-code a “correct” result.

6. Normalize the evidence into one comparison table

Do not compare a warm-cache run with a cold-cache run and then attribute the difference to executors. Record cache/tooling state or recreate the same state between experiments.

7. Experiment B — change only executor count

1. Preserve Experiment A evidence.
2. Change perf-agent-01 from 1 executor to 2.
3. Keep the Pipeline in serial mode and trigger the same four builds.
4. Capture the same queue/JVM/agent/workspace measurements.
5. Compare queue wait, per-build duration and controller/agent resource use.
6. Return the agent to 1 executor before Experiment C.

A successful result is not “two is faster.” A successful result is a defensible conclusion such as: “queue wait fell while per-build duration and resource headroom stayed acceptable,” or “queue wait fell but agent I/O contention doubled build duration, so total completion did not improve.”

8. Experiment C — change only Pipeline parallelism

With the agent back at one executor, run the same four builds with MODE=parallel2. Depending on how the workload allocates node/executor context, parallel branches may compete inside one allocation or require additional capacity. Observe what Jenkins actually schedules; do not assume the syntax guarantees hardware concurrency.

Record Pipeline step/stage timing, controller CPU, number of active steps, queue wait, workspace traffic and total elapsed time. Parallelism that reduces a single build’s step duration but increases platform queueing for all builds may not be an overall improvement.

9. Inspect workspace, console and artifact volume

set -euo pipefail
# Run only inside the disposable agent workspace whose identity you already recorded.
pwd
du -sk .
find . -xdev -type f -printf '%s %p\n' | sort -nr | head -n 20
# Inspect Jenkins-retained artifact sizes through the build UI/API rather than deleting files by hand.

Do not delete a live workspace to “fix performance” before proving it is the cause. Workspace cleanup changes cache state and may erase first-failure evidence; it must be an intentional experiment with before/after measurements.

10. Choose one actual bottleneck and make one change

Use your measured data—not the tutorial—to choose the next change. Examples:

11. Verification checklist

• Built-in node remained at 0 executors.
• All workload and resource changes were limited to disposable lab resources.
• Each experiment used the same job/source/tool versions and bounded trigger count.
• Only one intended variable changed between comparable runs.
• Queue wait was measured separately from build duration.
• Controller/JVM and agent resource evidence was captured before claiming a bottleneck.
• Workspace/log/artifact volume was recorded rather than guessed.
• No credentials, heap dump or sensitive support data was included in public artifacts.

12. Cleanup and rollback

Restore the agent to its original executor count, preserve the non-sensitive comparison report, dispose of the lab controller/agent, then remove only the exact evidence directory.

set -euo pipefail
OUT='/tmp/jenkins-perf-lab-evidence'
case "$OUT" in
  /tmp/jenkins-perf-lab-evidence) rm -rf -- "$OUT" ;;
  *) echo 'refusing unexpected cleanup path' >&2; exit 70 ;;
esac

13. Challenge: choose the next measurement, not the next knob

After adding a second executor, queue wait drops from 150 seconds to 30 seconds but median build duration rises from 70 seconds to 170 seconds. Controller metrics are normal; agent disk latency rises sharply. What should you do?

Do not add a third executor. The new evidence indicates agent storage contention. Restore the baseline concurrency and test a storage/workspace hypothesis while keeping the workload constant.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

Performance behavior depends on controller scale, workload shape, storage, plugins and runtime versions. Re-check current primary documentation before applying any tuning change to a real controller.

• Jenkins — Scaling Jenkins
• Jenkins — Hardware Recommendations
• Jenkins — Architecting for Scale
• Jenkins — Scaling Pipelines / durability
• Jenkins — Pipeline Best Practices
• Jenkins — Using agents
• Jenkins — Managing nodes
• Jenkins — Java Support Policy
• Metrics plugin
• Prometheus metrics plugin
• Pipeline: Groovy plugin
• Pipeline: Supporting APIs plugin
• Jenkins LTS changelog
• Jenkins Security Advisories