Performance Engineering: Executors, Queueing, Heap, Garbage Collection, Disk I/O, Workspaces, and Controller Load: Configuration, Design Choices, and Tradeoffs

1. Performance changes are architecture changes

Every tuning knob changes a resource contract. Adding executors changes concurrency and contention. Adding an agent changes scheduling capacity, isolation and operational cost. Increasing heap changes GC behavior and host memory reserve. Moving artifacts changes network, storage and recovery boundaries. Changing Pipeline durability changes crash-recovery semantics.

Therefore the design question is never just “what is faster?” It is “which layer is constrained, which component owns that layer, what side effects will this change introduce, and what evidence proves the outcome?”

2. More executors versus more agents

Current Jenkins guidance treats agents as the place for build execution and recommends zero controller executors. That boundary should not be sacrificed as a “performance optimization.”

3. Controller heap increase versus workload reduction

A larger heap can reduce collection frequency or prevent genuine out-of-memory conditions, but it can also lengthen some GC cycles, consume host memory reserved for filesystem cache/other processes, and mask unbounded retained objects. Before changing heap, establish the live-set trend, allocation/GC behavior, host memory/swap state and controller workload.

4. Artifact manager/external storage versus controller disk

Jenkins archived artifacts are convenient build evidence, but very large binaries and long retention can make controller backup, restore and disk I/O expensive. External artifact repositories or artifact-manager integrations can move durable binary storage to systems built for that role. The move is not “free”: network latency, credential scope, repository availability and traceability become external dependencies.

Keep immutable artifact coordinates/digests tied to the Jenkins build. Do not “optimize” by leaving release bytes only in an agent workspace.

5. Pipeline structure optimization versus plugin change

When controller load is driven by huge dynamic Pipeline graphs, many tiny steps or controller-side Groovy computation, adding or swapping plugins may not solve the causal workload. Simplify the Pipeline first: run data processing in scripts/binaries on agents, group excessively granular shell calls where sensible, bound matrix/parallel cardinality, and preserve only necessary step-level orchestration.

Plugin upgrades may still deliver important performance fixes. Treat them as privileged controller dependency changes: read changelogs/security advisories, test representative Pipelines and define rollback. Never change multiple major plugins and Pipeline structure in the same performance experiment if you want causal evidence.

6. Durability: speed versus restart survivability

Jenkins documents three broad Pipeline durability levels. The performance-optimized mode writes execution state less frequently and can greatly reduce disk I/O; maximum survivability persists more aggressively and is the slowest. “Survivable nonatomic” sits between them.

Change durability on a test controller/job and verify restart semantics as part of acceptance, not just throughput.

7. Cleanup frequency versus diagnostic retention

Frequent workspace/log cleanup can reduce disk pressure and cache staleness, but aggressive deletion can increase dependency download time and erase incident evidence. Retention should be policy-driven by data class:

8. Worked scenario: a controller with high queue wait and rising GC

Suppose p95 queue wait is 4 minutes, every linux-small agent is 90–100% busy, controller CPU averages 75%, GC pauses spike during morning bursts, and Pipeline runs create hundreds of tiny steps. The temptation is to add executors and heap simultaneously.

A better sequence is:

1. Split scheduling evidence from controller evidence.
2. Test whether adding one disposable agent reduces queue wait without materially increasing controller load.
3. Independently refactor one representative Pipeline to reduce tiny controller-orchestrated steps and compare controller CPU/GC.
4. Only if the stable live set still requires more memory, test one heap change on the same workload.
5. Accept changes separately so rollback remains possible.

9. Decision table

10. Cost and developer experience

Performance improvements consume capacity, operational complexity or durability budget. A 30-second improvement that doubles cloud-agent cost or removes crucial recovery evidence may be a poor trade. Likewise, extremely conservative durability and retention can make feedback unnecessarily slow. Tie every platform choice to an explicit service objective and owner.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

Performance behavior depends on controller scale, workload shape, storage, plugins and runtime versions. Re-check current primary documentation before applying any tuning change to a real controller.

• Jenkins — Scaling Jenkins
• Jenkins — Hardware Recommendations
• Jenkins — Architecting for Scale
• Jenkins — Scaling Pipelines / durability
• Jenkins — Pipeline Best Practices
• Jenkins — Using agents
• Jenkins — Managing nodes
• Jenkins — Java Support Policy
• Metrics plugin
• Prometheus metrics plugin
• Pipeline: Groovy plugin
• Pipeline: Supporting APIs plugin
• Jenkins LTS changelog
• Jenkins Security Advisories