Test Validity, Coordinated Omission, Warm-Up, Noise, and Experimental Design: Diagnostics, Failure Modes, and Production Practices

1. Preserve-first diagnostic sequence

flowchart TD
E[Preserve JTL + jmeter.log + engine + target evidence] --> V[Confirm JMeter/Java/plugin/tool versions]
V --> C[Confirm JMX/data/properties/CLI + authorized target]
C --> S[Validate scope/labels/resolved variables + warm-up/window/model]
S --> P[Inspect protocol/session/data/cache state]
P --> G[Inspect generator JVM/OS/network]
G --> T[Inspect SUT build/cache/DB/noise/telemetry]
T --> D[Inspect distributed/CI/container/time-sync state]
D --> H[Identify validity threat / competing cause]
H --> F[Least-destructive protocol correction]
F --> R[Small comparable rerun]

2. Failure mode: startup/JIT warm-up unknowingly included

Broken protocol starts a fresh service and immediately compares its first 100 samples with a long-lived baseline. Target events show the first 20 requests paid an intentional cold penalty.

Repair: keep the contaminated JTL, restart both conditions, run the same 20-request warm-up, then measure. Do not simply remove the first 20 rows after seeing them unless that exclusion was the predeclared protocol.

3. Failure mode: datasets/build configurations differ

A uses cached 1 KiB records; B uses uncached 5 MiB records or a debug build. The performance difference may be real, but it is not attributable to the code change under test.

Repair: hash/manifest build artifacts, data/config, feature flags and dependency versions. Re-run with equivalent datasets/config or explicitly change the question to test those differences.

4. Failure mode: declaring significance from one run

One A/B pair cannot show whether a 10 ms difference is larger than normal run variation. Repeat under the same protocol; report variation/effect magnitude. Avoid saying “statistically significant” unless the assumptions/test/design actually support that claim.

5. Failure mode: coordinated omission hides stalls

A one-thread closed test aims for “10 requests/s,” but when one request stalls for 750 ms the next requests do not exist. The JTL might contain only one slow sample during a wall-clock incident. A fixed-arrival scheduler still launches several arrivals that experience the bad window.

Repair the workload model for the performance question. Do not post-process nonexistent requests into the JTL and call them measured samples. Document whether you used a closed user population or an independent arrival schedule.

6. Failure mode: clock skew across telemetry

JMeter and target metrics appear offset by five seconds, so GC/queue spikes seem unrelated. In distributed/CI/container tests record timezone/UTC, NTP/clock offsets or correlate using run IDs/sequence markers. Do not manually slide graphs until they “look correlated” without recording the offset evidence.

7. Failure mode: tuning until the test passes

Changing warm-up, timeout, heap, threshold, thread count and dataset after every failure until the latest run is green creates a different experiment. Keep a protocol/version history. A product fix changes the product; a measurement fix changes the protocol and normally requires re-baselining both A and B.

8. Intentionally broken example: discard unexplained outliers

# anti-pattern
samples = [x for x in samples if x < 500]  # "500ms looks abnormal"
report_p95(samples)

If 500+ ms samples coincide with the fixture's logged periodic stall, they are exactly the phenomenon under study. Preserve them. Exclude a point only for a documented invalidity cause defined consistently (for example, target process crashed and the run is invalid), not because it harms the metric.

9. Background-noise diagnosis

Compare generator CPU/GC/disk/network and SUT process/background state across repetitions. If one run contains an OS update/backup/CI-neighbor event, mark it as a validity threat. The correct action may be to rerun the entire predefined repetition—not silently delete a few slow samples.

10. Causal table

11. Safety-sensitive boundaries

Sustained traffic, credentials, DB/message/API mutation, recorder certs, RMI engines, CI secrets, containers, OS/JVM tuning and failure injection require authorization and rollback. This chapter uses fake/local data and deterministic delay only. Never experiment on uncontrolled public/production systems.

12. Troubleshooting shortcuts to reject

• Do not add blanket retries or arbitrary long sleeps.
• Do not assign giant heaps without generator evidence.
• Do not mass-disable evidence/listeners without measured cost.
• Do not use global property hacks.
• Do not disable TLS/RMI verification.
• Do not increase workload unboundedly.
• Do not delete result files or unexplained outliers.
• Do not weaken thresholds/warm-up rules until a run passes.

Knowledge check

Official references and version notes

• Apache JMeter downloads — current JMeter 5.6.3 and Java 8+ requirement.
• JMeter current changes — Java 17+ recommendation for the 5.6.x line.
• Open Model Thread Group — currently experimental and schedule-driven.
• JMeter Dashboard Report — CSV requirements, report windows/percentiles and estimator caveats.
• JMeter Glossary — elapsed, latency and connect-time definitions.
• JMeter Listeners / result fields — timestamps, elapsed, success, active-thread counts and other JTL fields.