Checkpoint Lab — Backend Listener, InfluxDB/Graphite, Grafana, and Real-Time Telemetry

1. Exact assumptions and hard ceilings

2. Setup, authorization and preflight

1. Start pinned InfluxDB/Grafana stack (or exact-version native fallback).
2. Start the fixture with a fresh JSONL event log.
3. Verify InfluxDB/Grafana/fixture health and record UTC epoch/clock.
4. Record JMeter/Java/InfluxDB/Grafana versions and JMX/property hashes.
5. Verify Backend Listener queue=500, sampler regex=^Work$, low-cardinality tags, fake token only.
6. Verify target starts at baseline delay=40 ms and prior run IDs are absent from the current query window.

3. Predictions before execution

P1 — target/SUT metric: p24_sut.configured_delay_ms will show 40→180→40, with target JSONL control events at the same transitions.

P2 — JMeter latency: Work average/p95 will rise during degraded phase and fall during recovery; raw JTL/target service time will show the same direction.

P3 — throughput: hit/completion rate may decrease during degraded phase because the fixed number of closed threads spend longer waiting for each Work response.

P4 — active threads: mean active threads should stay near the configured two while completion rate changes, demonstrating that users and requests/second are different quantities.

P5 — overhead: Backend Listener-enabled run will add measurable generator/backend/network work relative to the no-backend variant, but the size of the overhead is machine-dependent and must not be pre-invented.

4. Start the independent SUT telemetry publisher

python .\tools\publish_sut_metric.py `
  --token p24-local-token-0123456789-do-not-reuse `
  --run-id p24-check `
  --duration 25 `
  --interval 1

This writes one server-state point/second. If it fails, stop and repair telemetry before the checkpoint; do not continue with a “Grafana maybe catches up” assumption.

5. Start the reversible control timeline

python .\tools\control_timeline.py `
  --run-id p24-check `
  --degrade-after 4 `
  --degraded-for 6

Preserve its printed epoch responses. The target JSONL also records the same control events.

6. Execute JMeter in CLI mode

New-Item -ItemType Directory -Force .\results\p24-check | Out-Null

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n -t .\plans\telemetry-local.jmx `
  -q .\config\local.properties `
  -Jrun.id=p24-check `
  -Jinflux.token=p24-local-token-0123456789-do-not-reuse `
  -l .\results\p24-check\results.jtl `
  -j .\results\p24-check\jmeter.log

Capture JMeter CPU/working set/heap during the run and docker stats --no-stream p24-influxdb p24-grafana during each phase if practical. Keep the terminal/clock order documented.

7. Observe the four Grafana panels

Panel A — JMeter Work avg + p95:
from(bucket: "jmeter")
  |> range(start: -30m)
  |> filter(fn: (r) => r._measurement == "jmeter")
  |> filter(fn: (r) => r.application == "p24-lab")
  |> filter(fn: (r) => r.run_id == "p24-check")
  |> filter(fn: (r) => r.transaction == "Work" and r.statut == "all")
  |> filter(fn: (r) => r._field == "avg" or r._field == "pct95.0")

Panel B — JMeter aggregate hits/window:
from(bucket: "jmeter")
  |> range(start: -30m)
  |> filter(fn: (r) => r._measurement == "jmeter")
  |> filter(fn: (r) => r.application == "p24-lab")
  |> filter(fn: (r) => r.run_id == "p24-check")
  |> filter(fn: (r) => r.transaction == "all" and r.statut == "all")
  |> filter(fn: (r) => r._field == "hit")

Panel C — JMeter mean active threads:
from(bucket: "jmeter")
  |> range(start: -30m)
  |> filter(fn: (r) => r._measurement == "jmeter")
  |> filter(fn: (r) => r.application == "p24-lab")
  |> filter(fn: (r) => r.run_id == "p24-check")
  |> filter(fn: (r) => r.transaction == "internal")
  |> filter(fn: (r) => r._field == "meanAT")

Panel D — independent SUT configured delay:
from(bucket: "jmeter")
  |> range(start: -30m)
  |> filter(fn: (r) => r._measurement == "p24_sut")
  |> filter(fn: (r) => r.run_id == "p24-check")
  |> filter(fn: (r) => r._field == "configured_delay_ms" or r._field == "inflight")

Use the same absolute time range and no per-panel hidden offset. Save/export the dashboard definition or at least record the exact Flux queries and datasource configuration in the evidence packet.

8. Verify raw phase evidence

python tools/correlate_timeline.py   results/p24-check/results.jtl   results/server-events.jsonl   p24-check

Require three phases and the expected delay values. JTL row count and target work count should agree within the run boundaries. If the run completes before recovery produces samples, increase neither threads nor target load; modestly extend only the bounded loop count/duration after documenting the first run.

9. Verify the time-series store directly

Use query_influx.py to inspect both measurements:

python .\tools\query_influx.py `
  --token p24-local-token-0123456789-do-not-reuse `
  --flux 'from(bucket:"jmeter") |> range(start:-30m) |> filter(fn:(r)=> r.run_id=="p24-check") |> keep(columns:["_time","_measurement","transaction","statut","_field","_value","mode","run_id"])'

Confirm JMeter metrics and p24_sut points exist before trusting Grafana. The Grafana dashboard can be deleted/recreated without changing the stored/raw evidence.

10. State the diagnosis causally

Accept this causal interpretation only if:

• the control event and p24_sut.configured_delay_ms independently show the server delay change;
• target service_wall_ms rises during that same phase;
• raw JTL elapsed rises in the same phase;
• Backend Listener Work avg/p95 changes in the same direction;
• generator CPU/GC/network is not saturated enough to explain the effect;
• workload/thread/pacing and label/tag configuration are unchanged.

Grafana correlation alone is insufficient; the controlled target mechanism and raw evidence establish the diagnosis.

11. Overhead comparison

Restore target baseline. Create two separate run IDs with 2 threads ×40 loops ×100 ms:

• p24-overhead-off — identical JMX with Backend Listener disabled;
• p24-overhead-on — Backend Listener enabled with the same queue/send/tags.

For each preserve:

Report the observed deltas and environment. Do not generalize the percentage to another machine, remote backend or test scale.

12. Retention and cleanup note

The local run ID tag is useful for querying but increases series history. After review:

• export/retain JTL, jmeter.log, workload/backend config, key queries and interpretation note as the long-lived evidence;
• delete disposable Influx/Grafana volumes if no longer needed;
• in a real platform, configure backend retention to match operational history requirements and cardinality budget;
• never preserve real tokens in dashboard exports, JMX or command history.

13. Required evidence packet

14. Validity statement

15. Verification checklist

• Only loopback target/backend/dashboard ports.
• Exact JMeter 5.6.3 / Java 17 / InfluxDB 2.9.1 / Grafana 13.2.1 recorded.
• No third-party JMeter plugin required.
• Queue/send/window/tag/sampler-regex configuration preserved.
• p24_sut shows 40→180→40; control timestamps preserved.
• Raw JTL and target events agree on achieved work.
• Live JMeter latency changes align with raw/target timing.
• Active-thread interpretation remains separate from request rate.
• Generator/backend overhead measured and limitations stated.
• No sensitive/high-cardinality tags; cleanup/retention documented.

16. Cleanup / rollback

1. POST fixture back to mode=baseline if it is still running.
2. Stop JMeter/publisher/control processes and fixture.
3. Preserve required evidence until review completes.
4. Stop/delete local stack when finished: docker compose down; add -v only after confirming disposable Influx/Grafana data may be deleted.
5. Remove fake local credentials from shell history if desired; no real credential was used.
6. No public/production target, recorder CA, remote RMI engine, managed telemetry, paid CI, database/message service, OS/JVM tuning or external failure injection was changed.

17. What Chapter 24 adds to the operating model

The production performance-testing operating model now has a real-time telemetry contract: Backend Listener implementation/queue/send interval/window/percentiles, sampler/transaction allow-list, tag/cardinality policy, backend protocol/version, datasource/dashboard queries, run/time/clock identity, SUT metric ownership, retention/secret policy, generator/backend overhead, direct-store verification, JTL/target cross-check and cleanup are required before live graphs support a performance diagnosis.

Chapter 25 moves to Distributed Testing, Remote Engines, Network Topology, and Synchronization. It extends this contract across multiple injectors where Backend Listener instances, JTL/sample transfer, clocks, files, properties, RMI security and network paths must all be synchronized deliberately.

Knowledge check

Official references and version notes

• JMeter Component Reference — Backend Listener — asynchronous listener queue, Graphite and InfluxDB clients, parameters, custom tags, raw-client resource warning.
• JMeter User Manual — Real-time Results — built-in InfluxDB/Graphite paths, live metrics, thread/response metrics, InfluxDB v2 setup and Grafana flow.
• JMeter Properties Reference — Backend Listener — send intervals, timeouts, metrics window and percentile estimator.
• InfluxDB OSS v2 — Write API — /api/v2/write, bucket/org/token and line protocol.
• InfluxDB OSS v2 release notes — 2.9.x compatibility baseline.
• Grafana InfluxDB data source — current product/version/query-language support.
• Grafana OSS downloads — current stable Grafana version.
• Apache JMeter downloads — current stable release and Java requirement.