Performance Testing Foundations: Load, Stress, Spike, Soak, and Capacity: Configuration, Design Patterns, and Trade-Offs

1. The design starts with the decision

A load test for a release gate and a stress test for architecture discovery may use some of the same samplers, but they should not use the same acceptance logic. A gate asks a binary reviewable question against a stable baseline. A characterization run asks how the system behaves across a range and may intentionally cross a failure boundary.

2. Closed-user and open-arrival mental models

In a closed workload, a fixed pool of virtual users repeatedly executes work. When responses slow down, those users wait longer and naturally offer less new work. In an open model, new arrivals are scheduled independently of how long earlier requests take; the system may accumulate concurrency as response times rise.

flowchart TB
subgraph Closed[Closed / concurrency-oriented]
U[Fixed user pool] --> W[Request]
W --> R[Wait for response]
R --> Z[Think / pacing]
Z --> W
end
subgraph Open[Open / arrival-oriented]
A[Arrival schedule] --> Q1[New request]
A --> Q2[New request]
A --> Q3[New request]
Q1 --> S[System]
Q2 --> S
Q3 --> S
end

Neither model is universally “more realistic.” Choose the model that resembles how demand reaches the real system. Human sessions often have closed-user aspects; queues, events, or externally scheduled API calls can be closer to open arrivals.

3. Exploratory characterization versus enforceable gate

An exploratory run can answer, “Where does the error curve become unacceptable?” A release gate must answer, “Did this version satisfy a rule that was defined before the run?” Mixing the two creates moving thresholds: engineers see the result, then choose a threshold that happens to pass.

4. Average versus percentiles: summarize without erasing the distribution

Consider ten elapsed times: 40, 41, 41, 42, 42, 43, 44, 45, 48, and 300 ms. The average is 72.6 ms. That average is mathematically correct but hides the one severe tail observation. A percentile is also only a summary, yet a high percentile makes tail behavior harder to hide.

Percentile definitions and estimators can vary between tools. When comparing results, keep the tool/version and population consistent. Do not compare a p95 computed over one set of labels and success rules with a p95 from a differently filtered population and call the difference a regression.

5. One run is an observation; repeated controlled runs support a claim

JIT compilation, filesystem cache, application cache, DNS, connection reuse, background jobs, GC, shared infrastructure, and many other sources create run-to-run variation. A robust comparison keeps important factors fixed, separates warm-up from steady state when relevant, and repeats the experiment enough to understand normal variance.

For the Chapter 01 local fixture, a useful teaching sequence is three tiny runs with identical JMX and fixture delay. The aim is not capacity; it is to observe whether the measured elapsed times are reasonably consistent. Then change only delay_ms and repeat. The causal story is much stronger because one major factor changed.

6. Keep configuration layers separate

A performance experiment becomes uninterpretable when engineers change several of these layers and attribute the result to only one of them.

7. Worked decision: a synthetic service has an expected 10 requests/second peak

Suppose a dedicated local/private test environment expects roughly 10 requests/second during normal operation and the release requirement is p95 elapsed time below 200 ms with less than 1% failed samples. A useful first gate design targets that representative load and verifies injector headroom. A stress experiment is separate: perhaps step above 10 requests/second to learn where the SLO stops holding. Do not use the stress ceiling as the normal-load gate.

If the real production demand arrives independently of response completion, open-arrival reasoning may better represent it. If the application is mostly a fixed population of interactive sessions, a closed user model with realistic think time may be a better approximation. The choice comes from demand semantics, not from which JMeter component is easiest to configure.

8. Generator cost and validity trade-offs

Higher traffic is not automatically a better test. More threads increase JVM, socket, memory, result, and target cost. More result fields can increase disk/network overhead. Longer tests improve time coverage but cost more environment time. Distributed generation can extend load but adds synchronization and network evidence complexity.

The design goal is the smallest controlled experiment that can answer the question with adequate confidence. Chapter 26 will cover generator sizing in depth; the invariant begins now: measure injector health before believing target-capacity conclusions.

9. Free/local design path

Every mandatory concept in this chapter can be practiced with the loopback Python fixture, Apache JMeter, the JDK, raw JTL files, and OS resource monitors. Managed load clouds, commercial observability, shared staging, and Kubernetes are not prerequisites. If a later production architecture uses those systems, the same evidence contract still applies.

Knowledge check

Official references and version notes

• Apache JMeter downloads — current production-release and Java requirement baseline.
• Getting Started — GUI authoring, CLI load execution, Java requirements, CLI flags, and operational guidance.
• Best Practices — load-generation and result-collection practices.
• Component Reference — Thread Group and current Open Model Thread Group status.
• HTML Dashboard Report — result-report terminology and percentile-oriented reporting.