Latency Percentiles, Throughput, Errors, Saturation, and Bottleneck Diagnosis: Core Concepts and Mental Model

1. One graph is a symptom, not root cause

p95 can rise because of a server queue, database pool, generator CPU/GC, DNS/connect behavior, payload changes, assertion/parser cost, or mixed labels. Diagnosis must explain multiple signals together.

2. Mental model

flowchart TD
O[Configured concurrency + pacing] --> Q[Queueing + service work]
Q --> L[Latency distribution]
Q --> T[Achieved throughput]
Q --> E[Errors]
G[Generator CPU/GC/network/disk] --> L
G --> T
S[SUT CPU/memory/DB/network/queue] --> Q
L --> H[Competing hypotheses]
T --> H
E --> H
G --> H
S --> H
H --> X[One-variable rerun]

Configured concurrency/pacing creates pressure. Work can queue before service. JMeter elapsed time includes the client-visible request path; throughput/errors describe achieved work. Generator and SUT telemetry locate pressure. The one-variable rerun tests whether the leading hypothesis predicts the observed response.

3. Metrics

4. Timing boundaries

JMeter elapsed time runs from just before request send until the last response byte. Latency ends at the first response; connect time covers connection establishment including SSL. A Constant Timer delays sampler execution and is not target service time.

5. Offered versus achieved

This lab uses a closed Thread Group: increasing threads while holding pacing constant increases offered pressure, but arrivals depend on response time. Achieved throughput is measured from JTL. Chapter30 will examine coordinated omission and other validity consequences.

6. Saturation knee

A defensible knee often combines diminishing throughput gains, rising tail latency/queue wait, a constrained resource pinned near its limit, possible errors, and a generator that is still healthy. No one signal alone proves the bottleneck.

7. Read-only inspection first

& "$env:JMETER_HOME\bin\jmeter.bat" -v\njava -version\nGet-Content .\config\bottleneck.properties\nGet-FileHash .\plans\bottleneck.jmx,.\config\bottleneck.properties -Algorithm SHA256\ncurl.exe --fail --silent http://127.0.0.1:8029/health

Prove JMeter/Java, workload, sample label, result schema, fixture capacity/service delay and target authorization before changing anything.

8. Evidence contract

Every diagnosis retains the source CSV JTL, matching jmeter.log, dashboard, exact JMX/properties, configured-versus-achieved workload, generator-health snapshot and target queue/service events before any correction. The mandatory lab stays on 127.0.0.1:8029 and uses no real credentials.

9. DevOps connection

Bottleneck diagnosis is experimental reasoning: JMeter supplies client symptoms, but root cause requires correlated system evidence and controlled changes.

Knowledge check

Official references and version notes

• Apache JMeter downloads — JMeter 5.6.3 and Java 8+ requirement.
• Dashboard Report — CSV requirements, percentile settings, response-time/throughput/error/thread graphs and estimator caveat.
• Properties Reference — result-save fields, thread counts and report-generator properties.
• JMeter Glossary — elapsed, latency and connect-time definitions.