Chapter 29Lesson 03~220 minutes

Latency Percentiles, Throughput, Errors, Saturation, and Bottleneck Diagnosis: Configuration, Design Patterns, and Trade-Offs

Metric choices determine what a diagnosis can legitimately say. Percentile estimator/window, label scope, transaction aggregation and target telemetry all change interpretation.

Learning objectives

Compare the principal configuration and design choices for Latency Percentiles, Throughput, Errors, Saturation, and Bottleneck Diagnosis without changing the workload question unintentionally.
Identify which settings belong to the JMeter plan, JVM, OS/network, target, extensions, CI/container, or distributed-engine layers.
Explain the trade-offs among performance cost, reliability, reproducibility, security, portability, and operational complexity.
Choose an appropriate pattern from measured evidence and explicit constraints rather than from convenience or folklore.
Preserve measurement validity and a stable evidence baseline before moving into failure diagnosis.

1. Percentile set and window

p50/p90/p95/p99 need enough comparable samples; extreme percentiles are sensitive on short runs. JMeter dashboard defaults to 90/95/99 and warns its estimator can differ from Aggregate Report. Document estimator/filter/window.

2. Request versus transaction metrics

Request	Transaction
One sampler/endpoint; good for locating a slow dependency.	Business/user operation spanning multiple requests.
Filter exact label.	Use a named Transaction Controller when the SLO is journey-level.
Do not mix fast health/static calls.	Do not mix parent and child samples into one unlabeled percentile.

3. Client-only versus full-stack telemetry

JMeter gives elapsed/latency/connect/success/threads/throughput. Full-stack diagnosis adds queue/worker/DB/CPU/memory/GC/network/downstream signals. Add the smallest second metric that can discriminate current hypotheses.

4. Full-run versus steady-state window

Full-run summaries include startup/ramp/warm-up/teardown. A steady-state window is valid only if defined before comparison; do not trim bad seconds after seeing the graph. Chapter30 deepens warm-up/noise design.

5. Timeline correlation

Bucketed p95/errors/throughput can be aligned with target queue/in-service and generator GC/CPU. Correlation narrows hypotheses; the controlled perturbation supplies stronger causal evidence.

6. Hypothesis-driven perturbation

Good experiment	Uncontrolled tuning
One change tied to a stated mechanism.	Heap, worker pool, timeout, retries and DB changed together.
Predict metric response before rerun.	Success defined after seeing results.
Same workload/scope/window.	Workload/version/filter differs.
Explicit rollback.	Configuration drift.

7. Configuration-layer boundaries

Layer	Examples
JMeter	Threads/timers/samplers/transactions/assertions/labels.
JVM	Java17 heap/GC.
OS/network	DNS/sockets/NIC/disk.
SUT	Worker/DB pool/service delay/cache/downstream.
Plugin/driver	Backend/monitoring/protocol extensions.
CI	Runner resources/timeouts/artifacts.
Container/orchestrator	CPU/memory quota, DNS/network, Pod count.

8. Worked scenario

Run A (8 threads): p95=330, p95 queue wait=240, in-service2/2, JMeter CPU20%. Run B after capacity6: p95=95, queue wait12, in-service reaches6, JMeter CPU28%. This strongly supports prior target worker contention but does not establish fixed open-arrival production capacity.

9. Decision table

Question	Measure
Typical vs tail	p50 + p95/p99.
Which HTTP call	Exact request-label percentiles.
Journey SLO	Transaction metric.
Server queue hypothesis	Queue wait/depth + in-service/capacity.
Generator hypothesis	JMeter CPU/GC/network/disk + achieved RPS.
Warm-up concern	Predeclared steady-state window + full-run context.

10. Evidence and safety contract

Keep raw CSV JTL, matching jmeter.log, dashboard, analysis window/filter/formula, generator evidence and target telemetry for every comparison. Mandatory runnable work remains at 127.0.0.1:8029 with synthetic data and no real credentials.

11. Validity before claims

Compare identical JMX/data/properties/labels/windows and generator class, except the planned perturbation. Report configured/achieved samples/RPS/errors and generator headroom before claiming target capacity or regression.

Knowledge check

Why can p99 be unstable on tiny samples?

When use transaction metrics?

Does queue/latency correlation prove causation?

Why predeclare a steady-state window?

What accompanies a bottleneck claim?

Next lesson

Common metric traps

Lesson 4 turns these trade-offs into a preserve-first diagnostic sequence for competing latency, throughput, error, saturation, and generator hypotheses.

Official references and version notes

Apache JMeter downloads — JMeter 5.6.3 and Java 8+ requirement.
Dashboard Report — CSV requirements, percentile settings, response-time/throughput/error/thread graphs and estimator caveat.
Properties Reference — result-save fields, thread counts and report-generator properties.
JMeter Glossary — elapsed, latency and connect-time definitions.

Version and compatibility note

Rechecked against current primary documentation on 2026-09-05. Mandatory runtime: Apache JMeter 5.6.3, Java 17, no third-party plugin. Dashboard percentiles default to 90/95/99 and are configurable. JMeter warns dashboard percentile estimates can differ from Aggregate Report, especially for small/wide samples; therefore this chapter audits p50/p90/p95/p99 from raw CSV JTL with a documented nearest-rank method and retains the dashboard as corroborating evidence. The lab uses a standard closed Thread Group, so “offered load” means configured concurrency plus pacing pressure, not an independent fixed open-arrival RPS.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20

0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.