Chapter 29Lesson 05~320 minutes

Checkpoint Lab — Latency Percentiles, Throughput, Errors, Saturation, and Bottleneck Diagnosis

The checkpoint produces a defensible bottleneck analysis: controlled degradation, two competing hypotheses, a second metric, one-variable validation rerun and remaining uncertainty.

CheckpointHypothesesQueue telemetryOne-variable rerunUncertainty

Learning objectives

Complete the chapter checkpoint for Latency Percentiles, Throughput, Errors, Saturation, and Bottleneck Diagnosis as one reviewable, bounded experiment.
State the workload, predictions, acceptance criteria, authorization boundary, and abort conditions before execution.
Reconcile configured versus achieved work with JTL, jmeter.log, target evidence, and generator validity before making a conclusion.
Produce an evidence packet that records the exact inputs, results, diagnosis or gate outcome, and any material limitations.
Perform cleanup or rollback and explain how the checkpoint evidence hands off to the next chapter or operating practice.

1. Exact assumptions and ceilings

Item	Checkpoint
Runtime	JMeter5.6.3 / Java17 / no plugin; Python3 stdlib fixture/analyzer.
Target	127.0.0.1:8029 only.
Constrained SUT	capacity2/service80/queue-timeout180.
Validation SUT	capacity6; service/timeout unchanged.
Profiles	2×30=60, 4×30=120, 8×30=240; 50ms timer.
Primary metrics	p50/p90/p95/p99, total/success throughput, errors, active threads.
Second metric	queue wait/max waiting + in-service/capacity.
Generator validity	CPU/heap/GC/socket evidence during high profile.
Evidence	JTL + jmeter.log + dashboard + summary/timeline + target JSONL + generator snapshot.

Abort: non-loopback traffic, target event count >configured, generator saturation, fixture crash, missing evidence, unplanned SUT change or >8 threads.

2. Predictions before execution

cap2 low: little queueing/no error knee.
cap2 medium/high: diminishing successful-throughput gain + rising p95/p99/queue wait; high may show bounded 503s.
in-service stays near2 while waiting grows.
If H1 is right, capacity2→6 at identical 8-thread load reduces queue/tail/errors and raises successful RPS.
If H2 is right, generator health saturates and cap6 does not materially fix the pattern.

3. Execute the cap2 sweep

Run/retain the 2/4/8 profiles from Lesson2. Require JTL/target counts=threads×30. Capture generator health during the 8-thread run.

4. State two competing hypotheses

Hypothesis	Support	Falsification
H1 target capacity2	in-service≈2; queue wait tracks tail/errors; generator healthy.	cap6 fails to reduce queue/tail or improve success RPS.
H2 generator	JMeter CPU/GC/network saturated; target below capacity.	Generator healthy and cap6 materially improves results.

5. Use a second metric before changing anything

Align p29-high/timeline.csv with generator snapshots. Tail/error seconds that coincide with high queue wait/max waiting and in-service pinned at2 support H1, but correlation is not final proof.

6. Controlled validation experiment

Preserve cap2 evidence, stop fixture, restart with only capacity6. Keep service80/timeout180/JMX/8 threads/30 loops/pacing50/result schema/analyzer unchanged.

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n -t .\plans\bottleneck.jmx -q .\config\bottleneck.properties `
  -Jthreads=8 -Jloops=30 -Jrun.id=p29-check-cap6 `
  -l .\results\p29-check-cap6\results.jtl -j .\results\p29-check-cap6\jmeter.log `
  -e -o .\results\p29-check-cap6\dashboard

7. Compare the rerun

Signal	cap2 high	cap6 high	Supports H1 if
p95/p99	High	Lower	Tail falls materially.
Successful RPS	Plateaued/lower	Higher	More useful work completes.
Errors	May include503	Lower/zero	Queue timeout pressure eases.
p95 queue wait	High	Lower	Waiting mechanism responds.
Max in-service	≈2	Can reach toward6	Changed resource is used.
Generator	Healthy	May rise modestly	Still not saturated.
service_ms	80	80	Service work unchanged.

8. Remaining uncertainty

closed-model arrival feedback/coordinated omission;
short run/sample-size percentile sensitivity;
single-machine loopback versus distributed topology;
synthetic semaphore versus real DB/thread pool;
JVM/OS warm-up/noise;
no soak/leak behavior.

These limitations bridge directly to Chapter30.

9. Evidence packet

Artifact	Required
Workload profile	2/4/8 threads, loops30, timer50, configured/achieved counts.
Percentiles	p50/p90/p95/p99 by Work label + nearest-rank formula.
Throughput/errors	All/success RPS + status/error rate.
Active threads	JTL thread-count evidence.
Generator	CPU/heap/GC/socket snapshot.
SUT	capacity, queue wait/max waiting, in-service, service_ms.
Timeline	per-second JTL+queue correlation.
Hypotheses	H1/H2 + predicted falsification.
Rerun	same 8-thread load, capacity2→6 only.
Uncertainty	explicit limitations after result.

10. Example conclusion form

Example: “At configured 2→4→8 closed-model threads, successful-throughput gains diminished while Work p95/p99 and target queue wait increased. At 8 threads, in-service was pinned near capacity2 and waiting grew while JMeter CPU/GC/network remained within the chosen validity limits. H1 (SUT worker contention) and H2 (generator saturation) were considered. Changing only SUT capacity2→6 reduced queue/tail/errors and increased successful throughput while service_ms remained80. This supports H1 for this controlled fixture, but does not establish an open-arrival production capacity; coordinated omission, warm-up, short-run noise and loopback topology remain limitations.”

11. Verification checklist

Only127.0.0.1:8029.
JMeter5.6.3/Java17/no plugin.
60/120/240 configured and matching achieved counts.
Same JMX/properties/formula.
p50/p90/p95/p99 + all/success RPS + errors + active threads.
Generator health + target queue/in-service evidence.
Two hypotheses before rerun.
Only capacity2→6 changes.
Remaining uncertainty documented.

12. Cleanup / rollback

Stop fixture and confirm port8029 free.
Keep cap2/cap6 evidence until review.
Rollback is restart with --capacity 2; no global OS/JVM change.
Delete disposable results only after retention policy.
No production/shared target, credential, remote engine or cloud resource was modified.

13. Operating-model addition and Chapter30 bridge

Chapter29 adds a bottleneck-diagnosis contract: exact workload/scope, label-filtered percentile formula/window, total/success throughput, errors, active concurrency, generator health, correlated SUT saturation metrics, competing hypotheses, one-variable validation, configured-versus-achieved evidence and remaining uncertainty.

Chapter30 is Test Validity, Coordinated Omission, Warm-Up, Noise, and Experimental Design, where the assumptions behind this closed-model sweep and causal comparison are tested more rigorously.

Knowledge check

Why require two hypotheses?

What second metric separates H1 from generator saturation?

What is the only planned SUT change?

If cap6 helps but JMeter becomes saturated, can you claim cap6 production capacity?

What Chapter30 question follows naturally?

Next chapter

Test Validity, Coordinated Omission, Warm-Up, Noise, and Experimental Design

Chapter 30 tests whether the experiment itself is valid by covering coordinated omission, warm-up, noise, repeatability, and controlled comparison.

Official references and version notes

Apache JMeter downloads — JMeter 5.6.3 and Java 8+ requirement.
Dashboard Report — CSV requirements, percentile settings, response-time/throughput/error/thread graphs and estimator caveat.
Properties Reference — result-save fields, thread counts and report-generator properties.
JMeter Glossary — elapsed, latency and connect-time definitions.

Version and compatibility note

Rechecked against current primary documentation on 2026-09-05. Mandatory runtime: Apache JMeter 5.6.3, Java 17, no third-party plugin. Dashboard percentiles default to 90/95/99 and are configurable. JMeter warns dashboard percentile estimates can differ from Aggregate Report, especially for small/wide samples; therefore this chapter audits p50/p90/p95/p99 from raw CSV JTL with a documented nearest-rank method and retains the dashboard as corroborating evidence. The lab uses a standard closed Thread Group, so “offered load” means configured concurrency plus pacing pressure, not an independent fixed open-arrival RPS.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20

0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.