Checkpoint Lab — Performance Engineering: Executors, Queueing, Heap, Garbage Collection, Disk I/O, Workspaces, and Controller Load

1. Mission

You are given the disposable jenkins-perf-lab controller and the bounded workload from Lesson 2. Your goal is not to make the best benchmark number. Your goal is to demonstrate a defensible performance-engineering method.

The checkpoint is complete only when another engineer can reproduce your baseline, understand your bottleneck hypothesis, identify the one variable you changed, and verify from evidence whether that change improved the intended service objective.

2. Safety and experiment rules

• Use only the disposable local controller/agent and synthetic repository/data.
• Keep the built-in/controller node at 0 executors.
• Use at most four concurrently triggered builds, two agent executors and two Pipeline branches.
• No production SCM, artifact repository, cloud, Kubernetes, IdP or secret is required.
• Do not disable Jenkins security controls.
• Do not create unbounded logs, files, threads, loops or parallel branches.
• Do not delete evidence before the comparison report is complete.
• Thread/heap/support diagnostics, if used, remain restricted and are not placed in the public evidence packet.

3. Record controller, job, source and execution identity

4. Write predictions before changing anything

Required prediction format:

Prediction A (baseline state)
If four identical builds arrive while perf-agent-01 has one executor,
then at least three queue items should wait initially, and queue reasons should
identify eligible-capacity pressure rather than controller failure.

Prediction B (candidate change)
If the measured bottleneck is eligible execution capacity AND agent/controller
resource headroom exists, adding exactly one executor or one disposable agent
should reduce queue wait. Per-build duration and controller load should remain
within the baseline confidence range. If they worsen materially, reject the change.

You may choose a different bottleneck and prediction, but it must be based on evidence and include a falsification condition.

5. Run the baseline workload

1. Ensure the lab starts from the documented baseline: one eligible agent executor, serial workload mode, same source revision and cache state.
2. Trigger exactly four builds.
3. Capture queue snapshots through queue-to-build transitions.
4. Capture node/executor snapshots and optional metrics.
5. Record build start/end, stage timing, workspace size, console/artifact sizes and external mock latency.
6. Repeat the baseline at least twice if practical so one anomalous run does not define the conclusion.

6. Baseline evidence packet

7. State one bottleneck hypothesis

Your hypothesis must quote measurements. Examples:

Example 1 — eligible capacity
p95 queue wait = 145 s; perf-agent-01 single executor is continuously busy;
agent CPU peaks 45%, disk latency stays low, controller CPU/GC normal.
Hypothesis: eligible executor capacity is the dominant bottleneck.

Example 2 — agent I/O
queue wait < 5 s; build execution = 190 s; workspace write latency and iowait
rise during perf-data generation; controller CPU/GC normal.
Hypothesis: agent workspace storage is the dominant bottleneck.

Example 3 — controller/Pipeline
queue wait low; agent work < 20 s; controller CPU and Pipeline step persistence
spike during thousands of tiny steps; build completion waits on orchestration.
Hypothesis: Pipeline/controller orchestration is the dominant bottleneck.

8. Change exactly one variable

Do not alter source revision, tool versions, agent image, cache state and executor count simultaneously. If multiple conditions must change, this checkpoint cannot attribute causality.

9. Rerun the identical workload

Trigger the same number of builds with the same job/source/parameters and collect the same evidence fields. Record the exact change timestamp so the before/after sets cannot be mixed.

10. Compare outcome and resource cost

11. Accept, reject or classify as inconclusive

Use one of these outcomes:

• Accept: the intended SLI improved and resource/reliability costs remain within documented bounds.
• Reject: the intended metric did not improve, another important metric regressed, or the change introduced unacceptable reliability/security cost.
• Inconclusive: workload/cache/environment changed, measurements are too noisy, or the evidence does not isolate the variable. Rebuild the experiment rather than inventing a conclusion.

12. Required performance report

Performance checkpoint report
=============================
Controller/core/Java/plugins:
Job/source/Jenkinsfile revision:
Agent/image/tools/executors:
Baseline workload definition:
Baseline run IDs / queue IDs:
Baseline measurements:
Bottleneck hypothesis:
Competing hypothesis:
Single controlled change:
After-change run IDs / queue IDs:
After-change measurements:
End-to-end result:
Controller resource result:
Agent/storage result:
Durability/reliability impact:
Decision: ACCEPT / REJECT / INCONCLUSIVE
Rollback performed or retained change:
Known limitations:
Evidence paths / redaction review:

13. Verification checklist

• Exactly five chapter files aside, the lab modified no shared Academy/course resources.
• Controller built-in executors stayed at 0.
• At least two predicted state/resource changes were written before the tuning change.
• Baseline and post-change runs used the same source, toolchain and bounded workload.
• Queue wait was not confused with build duration.
• Controller and agent resource costs were both considered.
• Workspace/log/artifact volume and external latency were explicit.
• Only one tuning variable changed.
• First-failure/slow evidence was preserved before cleanup or retry.
• The final conclusion can be rejected by the recorded measurements if the hypothesis was wrong.
• No secret, token, private key, heap dump or sensitive support bundle is in shareable evidence.

14. Cleanup and rollback

If the experiment changed executor count, heap, Pipeline durability, workspace location or any other lab setting, either restore the documented baseline or explicitly record why the accepted change remains. Dispose of the controller/agent and remove only exact temporary paths.

set -euo pipefail
OUT='/tmp/jenkins-perf-lab-evidence'
case "$OUT" in
  /tmp/jenkins-perf-lab-evidence) rm -rf -- "$OUT" ;;
  *) echo 'refusing unexpected cleanup path' >&2; exit 70 ;;
esac

15. What Chapter 38 adds to a production Jenkins operating model

Chapter 37 established how to observe Jenkins. Chapter 38 converts that telemetry into controlled engineering decisions. You can now distinguish queue/eligibility from executor saturation, controller/JVM pressure from agent execution, CPS persistence from project computation, and local storage from external-service latency. More importantly, you can prove whether a change helped rather than relying on intuition.

Chapter 39 builds directly on this discipline for LTS, Java and plugin upgrades: performance baselines become part of compatibility testing, and rollback planning must preserve both correctness and service behavior.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

Performance behavior depends on controller scale, workload shape, storage, plugins and runtime versions. Re-check current primary documentation before applying any tuning change to a real controller.

• Jenkins — Scaling Jenkins
• Jenkins — Hardware Recommendations
• Jenkins — Architecting for Scale
• Jenkins — Scaling Pipelines / durability
• Jenkins — Pipeline Best Practices
• Jenkins — Using agents
• Jenkins — Managing nodes
• Jenkins — Java Support Policy
• Metrics plugin
• Prometheus metrics plugin
• Pipeline: Groovy plugin
• Pipeline: Supporting APIs plugin
• Jenkins LTS changelog
• Jenkins Security Advisories