Latency Percentiles, Throughput, Errors, Saturation, and Bottleneck Diagnosis: Guided Hands-On Workflow

1. Safety envelope

2. Local bottleneck fixture

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlparse, parse_qs
import argparse, json, re, threading, time
FIXTURE_VERSION='prompt29-bottleneck-fixture-v1'
SAFE=re.compile(r'^[A-Za-z0-9_.-]{1,64}$')
lock=threading.Lock(); event_log=None; capacity=2; service_ms=80; queue_timeout_ms=180; sem=None
state={'requests':0,'successes':0,'errors':0,'waiting':0,'in_service':0,'max_waiting':0,'max_in_service':0}
def now_ms(): return int(time.time()*1000)
def snapshot():
    with lock: return {'fixture_version':FIXTURE_VERSION,'capacity':capacity,'service_ms':service_ms,'queue_timeout_ms':queue_timeout_ms,**state}
def write_event(e):
    if event_log is None: return
    with lock:
        with event_log.open('a',encoding='utf-8') as h: h.write(json.dumps(e,sort_keys=True)+'\n')
class Handler(BaseHTTPRequestHandler):
    protocol_version='HTTP/1.1'
    def send_json(self,status,payload):
        raw=json.dumps(payload,sort_keys=True).encode(); self.send_response(status); self.send_header('Content-Type','application/json'); self.send_header('Content-Length',str(len(raw))); self.send_header('X-Fixture-Version',FIXTURE_VERSION); self.end_headers(); self.wfile.write(raw)
    def do_GET(self):
        started=now_ms(); p=urlparse(self.path)
        if p.path=='/health' or p.path=='/stats': self.send_json(200,{'status':'ok','state':snapshot()}); return
        if p.path!='/work': self.send_json(404,{'status':'not_found'}); return
        q=parse_qs(p.query); run_id=q.get('run_id',[''])[0]; thread_id=q.get('thread',[''])[0]; seq_raw=q.get('seq',[''])[0]
        if not SAFE.fullmatch(run_id) or not SAFE.fullmatch(thread_id): self.send_json(400,{'status':'invalid_metadata'}); return
        try: seq=int(seq_raw)
        except ValueError: self.send_json(400,{'status':'invalid_seq'}); return
        with lock:
            state['requests']+=1; state['waiting']+=1; state['max_waiting']=max(state['max_waiting'],state['waiting']); waiting_at_arrival=state['waiting']
        t0=time.perf_counter(); acquired=sem.acquire(timeout=queue_timeout_ms/1000.0); queue_wait_ms=int(round((time.perf_counter()-t0)*1000))
        if not acquired:
            with lock: state['waiting']-=1; state['errors']+=1; current=dict(state)
            self.send_json(503,{'status':'queue_timeout','run_id':run_id,'thread':thread_id,'seq':seq,'queue_wait_ms':queue_wait_ms,'capacity':capacity})
            write_event({'ts_ms':now_ms(),'operation':'work','status':503,'run_id':run_id,'thread':thread_id,'seq':seq,'capacity':capacity,'service_ms':service_ms,'queue_timeout_ms':queue_timeout_ms,'queue_wait_ms':queue_wait_ms,'service_wall_ms':0,'waiting_at_arrival':waiting_at_arrival,'in_service_at_start':current['in_service']}); return
        try:
            with lock:
                state['waiting']-=1; state['in_service']+=1; state['max_in_service']=max(state['max_in_service'],state['in_service']); in_service_at_start=state['in_service']
            time.sleep(service_ms/1000.0)
            with lock: state['successes']+=1
            self.send_json(200,{'status':'ok','run_id':run_id,'thread':thread_id,'seq':seq,'queue_wait_ms':queue_wait_ms,'service_ms':service_ms,'capacity':capacity})
            write_event({'ts_ms':now_ms(),'operation':'work','status':200,'run_id':run_id,'thread':thread_id,'seq':seq,'capacity':capacity,'service_ms':service_ms,'queue_timeout_ms':queue_timeout_ms,'queue_wait_ms':queue_wait_ms,'service_wall_ms':now_ms()-started,'waiting_at_arrival':waiting_at_arrival,'in_service_at_start':in_service_at_start})
        finally:
            with lock: state['in_service']-=1
            sem.release()
    def log_message(self,format,*args): return
def main():
    p=argparse.ArgumentParser(); p.add_argument('--host',default='127.0.0.1'); p.add_argument('--port',type=int,default=8029); p.add_argument('--capacity',type=int,default=2); p.add_argument('--service-ms',type=int,default=80); p.add_argument('--queue-timeout-ms',type=int,default=180); p.add_argument('--log',required=True); a=p.parse_args()
    global event_log,capacity,service_ms,queue_timeout_ms,sem
    capacity=a.capacity; service_ms=a.service_ms; queue_timeout_ms=a.queue_timeout_ms; sem=threading.BoundedSemaphore(capacity); event_log=Path(a.log).resolve(); event_log.parent.mkdir(parents=True,exist_ok=True); event_log.write_text('',encoding='utf-8')
    print(f'fixture_version={FIXTURE_VERSION}',flush=True); print(f'listen=http://{a.host}:{a.port}',flush=True); print(f'capacity={capacity} service_ms={service_ms} queue_timeout_ms={queue_timeout_ms}',flush=True); ThreadingHTTPServer((a.host,a.port),Handler).serve_forever()
if __name__=='__main__': main()

capacity is the resource under test. JSONL records queue wait, waiting-at-arrival, in-service count, status and service timing.

3. Start capacity=2

python .\fixtures\bottleneck_fixture.py `\n --host 127.0.0.1 --port 8029 `\n --capacity 2 --service-ms 80 --queue-timeout-ms 180 `\n --log .\results\target-cap2-events.jsonl

Preflight /health must report capacity2/service80/timeout180.

4. JMX/result contract

target.host=127.0.0.1
target.port=8029
threads=2
loops=30
pacing.ms=50
connect.timeout.ms=500
response.timeout.ms=2000
jmeter.httpsampler=HttpClient4
httpclient4.retrycount=0
jmeter.save.saveservice.output_format=csv
jmeter.save.saveservice.print_field_names=true
jmeter.save.saveservice.timestamp_format=ms
jmeter.save.saveservice.time=true
jmeter.save.saveservice.label=true
jmeter.save.saveservice.response_code=true
jmeter.save.saveservice.response_message=true
jmeter.save.saveservice.thread_name=true
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.bytes=true
jmeter.save.saveservice.sent_bytes=true
jmeter.save.saveservice.thread_counts=true
jmeter.save.saveservice.latency=true
jmeter.save.saveservice.connect_time=true
jmeter.save.saveservice.assertion_results_failure_message=true
jmeter.save.saveservice.response_data=false
jmeter.save.saveservice.response_data.on_error=false
jmeter.save.saveservice.samplerData=false
jmeter.save.saveservice.responseHeaders=false
jmeter.save.saveservice.requestHeaders=false
jmeter.save.saveservice.url=false
jmeter.reportgenerator.overall_granularity=2000
jmeter.reportgenerator.aggregate_rpt_pct1=90
jmeter.reportgenerator.aggregate_rpt_pct2=95
jmeter.reportgenerator.aggregate_rpt_pct3=99

Test Plan
├── HTTP Request Defaults -> ${__P(target.host,127.0.0.1)}:${__P(target.port,8029)}, HttpClient4
└── Thread Group -> threads=${__P(threads,2)}, loops=${__P(loops,30)}, Continue on error
    ├── Counter -> SEQ (per user)
    └── HTTP Request — Work
        GET /work?run_id=${__P(run.id,p29)}&thread=T${__threadNum}&seq=${SEQ}
        Use KeepAlive=checked
        ├── Constant Timer ${__P(pacing.ms,50)} ms
        └── Response Assertion: response code = 200

The dashboard keeps standard graphs; the audit script computes label-filtered p50/p90/p95/p99 and combines them with target queue telemetry.

5. Analyzer

import argparse,csv,json,math
from collections import Counter,defaultdict
from pathlib import Path
def nr(v,p):
    d=sorted(v)
    return 0 if not d else d[max(1,math.ceil(len(d)*p/100.0))-1]
def main():
    p=argparse.ArgumentParser(); p.add_argument('--jtl',required=True); p.add_argument('--target-events',required=True); p.add_argument('--run-id',required=True); p.add_argument('--threads',type=int,required=True); p.add_argument('--loops',type=int,required=True); p.add_argument('--out',required=True); p.add_argument('--timeline',required=True); a=p.parse_args(); expected=a.threads*a.loops
    rows=[r for r in csv.DictReader(Path(a.jtl).open(newline='',encoding='utf-8')) if r.get('label')=='Work']; elapsed=[int(float(r['elapsed'])) for r in rows]; failures=sum(r.get('success','').lower()!='true' for r in rows)
    events=[json.loads(x) for x in Path(a.target_events).read_text(encoding='utf-8').splitlines() if x.strip()]; target=[e for e in events if e.get('operation')=='work' and e.get('run_id')==a.run_id]
    if rows:
        start=min(int(r['timeStamp']) for r in rows); end=max(int(r['timeStamp'])+int(float(r['elapsed'])) for r in rows); span=max((end-start)/1000.0,.001)
    else: start=0; span=0
    active=[int(float(r.get('allThreads') or r.get('grpThreads'))) for r in rows if (r.get('allThreads') or r.get('grpThreads')) not in (None,'')]; qw=[int(e.get('queue_wait_ms',0)) for e in target]
    s={'run_id':a.run_id,'configured':{'threads':a.threads,'loops':a.loops,'expected_samples':expected},'achieved':{'jtl_samples':len(rows),'target_events':len(target),'failures':failures,'error_rate_pct':round(100*failures/len(rows),3) if rows else 100,'throughput_all_samples_rps':round(len(rows)/span,3) if span else 0,'throughput_success_rps':round((len(rows)-failures)/span,3) if span else 0,'max_active_threads_observed':max(active) if active else None},'latency_ms':{'p50':nr(elapsed,50),'p90':nr(elapsed,90),'p95':nr(elapsed,95),'p99':nr(elapsed,99),'max':max(elapsed) if elapsed else 0},'target':{'statuses':dict(Counter(int(e.get('status',0)) for e in target)),'capacity_values':sorted({int(e.get('capacity',0)) for e in target}),'service_ms_values':sorted({int(e.get('service_ms',0)) for e in target}),'avg_queue_wait_ms':round(sum(qw)/len(qw),3) if qw else 0,'p95_queue_wait_ms':nr(qw,95),'max_waiting_at_arrival':max((int(e.get('waiting_at_arrival',0)) for e in target),default=0),'max_in_service_at_start':max((int(e.get('in_service_at_start',0)) for e in target),default=0)},'valid':len(rows)==expected and len(target)==expected}; Path(a.out).write_text(json.dumps(s,indent=2),encoding='utf-8')
    buckets=defaultdict(lambda:{'e':[],'samples':0,'fail':0,'active':[],'q':[],'w':[],'ins':[]})
    for r in rows:
        sec=max(0,(int(r['timeStamp'])-start)//1000); b=buckets[sec]; b['samples']+=1; b['e'].append(int(float(r['elapsed']))); b['fail']+=r.get('success','').lower()!='true'; av=r.get('allThreads') or r.get('grpThreads'); b['active'] += [int(float(av))] if av not in (None,'') else []
    for e in target:
        sec=max(0,(int(e['ts_ms'])-start)//1000); b=buckets[sec]; b['q'].append(int(e.get('queue_wait_ms',0))); b['w'].append(int(e.get('waiting_at_arrival',0))); b['ins'].append(int(e.get('in_service_at_start',0)))
    with Path(a.timeline).open('w',newline='',encoding='utf-8') as h:
        w=csv.writer(h); w.writerow(['second','samples','failures','p95_elapsed_ms','max_active_threads','avg_queue_wait_ms','max_waiting','max_in_service'])
        for sec in sorted(buckets):
            b=buckets[sec]; w.writerow([sec,b['samples'],b['fail'],nr(b['e'],95),max(b['active']) if b['active'] else '',round(sum(b['q'])/len(b['q']),3) if b['q'] else '',max(b['w']) if b['w'] else '',max(b['ins']) if b['ins'] else ''])
    print(json.dumps(s,indent=2)); raise SystemExit(0 if s['valid'] else 3)
if __name__=='__main__': main()

It emits configured/achieved counts, total/success RPS, errors, active threads, percentiles, queue metrics and a per-second timeline.

6. Run 2/4/8-thread profiles

Low:

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n -t .\plans\bottleneck.jmx -q .\config\bottleneck.properties `
  -Jthreads=2 -Jloops=30 -Jrun.id=p29-low `
  -l .\results\p29-low\results.jtl -j .\results\p29-low\jmeter.log `
  -e -o .\results\p29-low\dashboard

Medium:

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n -t .\plans\bottleneck.jmx -q .\config\bottleneck.properties `
  -Jthreads=4 -Jloops=30 -Jrun.id=p29-medium `
  -l .\results\p29-medium\results.jtl -j .\results\p29-medium\jmeter.log `
  -e -o .\results\p29-medium\dashboard

High:

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n -t .\plans\bottleneck.jmx -q .\config\bottleneck.properties `
  -Jthreads=8 -Jloops=30 -Jrun.id=p29-high `
  -l .\results\p29-high\results.jtl -j .\results\p29-high\jmeter.log `
  -e -o .\results\p29-high\dashboard

Analyze each with --threads 2|4|8 --loops 30 against target-cap2-events.jsonl. Expected qualitatively: low has little queueing; medium adds tail/queue; high shows diminishing successful-throughput gains, large tail/queue and possibly bounded 503s. Record actual numbers.

7. Capture generator health during high run

Windows:

$Proc = Get-CimInstance Win32_Process | Where-Object { $_.Name -eq 'java.exe' -and $_.CommandLine -like '*ApacheJMeter.jar*' } | Select-Object -First 1
$Pid = $Proc.ProcessId
Get-Process -Id $Pid | Select-Object Id,CPU,WorkingSet64,PrivateMemorySize64,Handles,Threads
jcmd $Pid GC.heap_info
jstat -gcutil $Pid 1000 6
Get-NetTCPConnection -OwningProcess $Pid -ErrorAction SilentlyContinue | Group-Object State | Select-Object Name,Count

Linux:

PID=$(jps -lv | awk '/ApacheJMeter.jar/ {print $1; exit}')
ps -p "$PID" -o pid,pcpu,pmem,rss,vsz,nlwp,etime,cmd
jcmd "$PID" GC.heap_info
jstat -gcutil "$PID" 1000 6
ss -tanp

If JMeter CPU/GC/network/disk saturates first, stop and report the run invalid for target-bottleneck attribution.

8. Competing hypotheses

9. One server-side change

Preserve cap2 evidence, stop fixture, restart with only --capacity 6:

python .\fixtures\bottleneck_fixture.py `\n --host 127.0.0.1 --port 8029 `\n --capacity 6 --service-ms 80 --queue-timeout-ms 180 `\n --log .\results\target-cap6-events.jsonl

Do not alter service delay, timeout, threads, pacing, JMeter heap, retries or analysis formula.

10. Rerun eight threads

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n -t .\plans\bottleneck.jmx -q .\config\bottleneck.properties `
  -Jthreads=8 -Jloops=30 -Jrun.id=p29-high-cap6 `
  -l .\results\p29-high-cap6\results.jtl -j .\results\p29-high-cap6\jmeter.log `
  -e -o .\results\p29-high-cap6\dashboard

If H1 is correct, queue/tail/errors fall and successful RPS rises while service_ms stays80.

11. Challenge

p95=360 ms, JMeter CPU=18%, target max_waiting=6 and max_in_service=2. Double heap or capacity only?

Capacity only. Heap has no supporting evidence; target queue/in-service directly supports H1.

Knowledge check

Official references and version notes

• Apache JMeter downloads — JMeter 5.6.3 and Java 8+ requirement.
• Dashboard Report — CSV requirements, percentile settings, response-time/throughput/error/thread graphs and estimator caveat.
• Properties Reference — result-save fields, thread counts and report-generator properties.
• JMeter Glossary — elapsed, latency and connect-time definitions.