Test Validity, Coordinated Omission, Warm-Up, Noise, and Experimental Design: Core Concepts and Mental Model

1. The practical problem: a measured difference can be an artifact

Suppose branch B is 12 ms slower than branch A in one run. That difference could be product code, JVM/JIT warm-up, a cold cache, different test data, background CPU noise, a runner change, clock skew, an arrival-model artifact, or normal run-to-run variation. A valid experiment controls or measures these competing explanations before calling the difference a regression.

2. Mental model: the experiment is a chain of assumptions

flowchart TD
Q[Question] --> H[Hypothesis + predicted direction]
H --> E[Controlled environment + versions/data]
E --> W[Workload model
closed or arrival-oriented]
W --> U[Warm-up]
U --> M[Measurement window]
M --> R[Repeated runs]
R --> C[Engineering/statistical comparison]
C --> K[Conclusion + uncertainty]
V[Generator/target validity evidence] --> C
N[Noise/background state] --> C

The question defines what decision the test should support. A hypothesis predicts what metric should change. Environment/data/tool versions define comparability. The workload model determines how arrivals react to response time. Warm-up removes intentionally non-representative cold behavior. A predeclared measurement window defines which samples count. Repetitions reveal variation. Only then do percentiles/means/throughput support a conclusion—and the conclusion must still list uncertainty.

3. New terms

4. State that must be frozen or observed

5. Closed versus arrival-oriented interpretation

In the stable JMeter Thread Group, a thread waits for its current sample to finish before proceeding, so the realized arrival rate depends on response time. That is often correct for “N concurrent users.” It is not the same question as “the system must receive 10 arrivals/s even during a 700 ms stall.” The latter is arrival-oriented and must schedule work independently of previous completion.

JMeter 5.6.3 includes an experimental Open Model Thread Group that can express schedules such as rate(10/sec) random_arrivals(6 sec). Because the component is explicitly experimental, this chapter's mandatory coordinated-omission demonstration uses a small Python standard-library scheduler as a stable fallback; the JMeter Open Model path is optional.

6. Timing boundaries and warm-up

JMeter elapsed/latency/connect fields begin at sampler execution—not during a Constant Timer. Warm-up can occur in multiple layers: Java JIT, DNS/TLS/connection pool, target cache, database pages, application code paths and generated data. Do not assume “30 seconds” is universally enough. Inspect time series and target state, choose a bounded warm-up rule, and keep it identical between A/B conditions.

7. Repetition and uncertainty

One run estimates one realization. Three repetitions are still a small sample, but they expose whether a 15 ms difference is large relative to run variation. This chapter reports median, sample standard deviation, coefficient of variation and a simple t-based interval for the mean. The interval is an uncertainty illustration, not a magical significance certificate.

8. Read-only/non-destructive inspection first

& "$env:JMETER_HOME\bin\jmeter.bat" -v
java -version

Get-FileHash `
  .\plans\warmup.jmx, .\plans\measure.jmx, `
  .\config\validity.properties, .\data\* `
  -Algorithm SHA256

Get-Content .\experiment\protocol.json
Get-Content .\results\*\summary.json -ErrorAction SilentlyContinue

Also inspect generator process/JVM state and target /health. Do not change warm-up, data, runtime or thresholds until the actual executed state is proven.

9. DevOps connection

Performance engineering becomes credible when a team can distinguish a product regression from cold startup, workload-model bias, a noisy CI runner or random variation. The same evidence discipline improves release gates, incident analysis, capacity experiments and optimization reviews.

Knowledge check

Official references and version notes

• Apache JMeter downloads — current JMeter 5.6.3 and Java 8+ requirement.
• JMeter current changes — Java 17+ recommendation for the 5.6.x line.
• Open Model Thread Group — currently experimental and schedule-driven.
• JMeter Dashboard Report — CSV requirements, report windows/percentiles and estimator caveats.
• JMeter Glossary — elapsed, latency and connect-time definitions.
• JMeter Listeners / result fields — timestamps, elapsed, success, active-thread counts and other JTL fields.