Checkpoint Lab — CLI Mode, Headless Execution, Result Files, and Reproducible Runs

1. Exact assumptions and hard ceilings

2. Setup / preflight

1. Create fixture, property file, JMX, launcher, gate/manifest/analyzer scripts from Lesson 2.
2. Start fixture on 127.0.0.1:8021 with a fresh JSONL event log.
3. GET /health//stats and record baseline work count.
4. Record JMeter/Java version and SHA-256 of JMX/properties.
5. Verify results/p21-check-a does not exist.

3. Predictions before the successful run

Prediction A — artifacts: launcher creates one new run directory containing command/version/pre-manifest, JTL, jmeter.log, engine/gate exit files, HTML report and final manifest.

Prediction B — workload: exactly eight successful Work samples appear in JTL and exactly eight matching target events occur: four for T1 and four for T2.

Prediction C — status: engine exit=0 and gate exit=0 when row count, failures and lab p95 are valid.

4. Successful run A

PowerShell:

Set-Location "F:\Labs\p21-cli-lab"
.\tools\run-local.ps1 -RunId p21-check-a
$LASTEXITCODE

Or Bash:

cd /path/to/p21-cli-lab
./tools/run-local.sh p21-check-a
printf 'wrapper_exit=%s
' "$?"

5. Verify run A

• engine-exit.txt = 0;
• gate-exit.txt = 0;
• gate.json = PASS, samples=8, failures=0, p95≤500;
• JTL has header + 8 sample data rows;
• jmeter.log has no unexplained ERROR/FATAL;
• html-report/index.html exists;
• run-manifest.json contains exact command, versions, hashes and paths.

python tools/analyze_events.py results/server-events.jsonl p21-check-a

Require eight target work events with four per thread.

6. Predict the intentional missing-file failure

Prediction D: directing raw JMeter at plans/missing-plan.jmx produces non-success engine/process evidence before the normal eight-sample workload. The target should receive zero work events for the missing-run ID, and the safe launcher's historical run A must remain untouched.

7. Execute the missing-JMX diagnostic

PowerShell:

$ProjectRoot = "F:\Labs\p21-cli-lab"
$FailDir = Join-Path $ProjectRoot "results\p21-missing"
if (Test-Path $FailDir) { throw "Failure evidence directory already exists." }
New-Item -ItemType Directory $FailDir | Out-Null

$Command = @(
  "$env:JMETER_HOME\bin\jmeter.bat","-n",
  "-t",(Join-Path $ProjectRoot "plans\missing-plan.jmx"),
  "-q",(Join-Path $ProjectRoot "config\local.properties"),
  "-Jrun.id=p21-missing",
  "-l",(Join-Path $FailDir "results.jtl"),
  "-j",(Join-Path $FailDir "jmeter.log")
)
$Command -join " " | Set-Content (Join-Path $FailDir "command.txt")

& $Command[0] $Command[1..($Command.Count-1)]
$EngineExit = $LASTEXITCODE
$EngineExit | Set-Content (Join-Path $FailDir "engine-exit.txt")

Bash:

PROJECT_ROOT=/path/to/p21-cli-lab
FAIL_DIR="$PROJECT_ROOT/results/p21-missing"
[[ ! -e "$FAIL_DIR" ]] || { echo "Failure dir exists" >&2; exit 2; }
mkdir -p "$FAIL_DIR"

"$JMETER_HOME/bin/jmeter"   -n   -t "$PROJECT_ROOT/plans/missing-plan.jmx"   -q "$PROJECT_ROOT/config/local.properties"   -Jrun.id=p21-missing   -l "$FAIL_DIR/results.jtl"   -j "$FAIL_DIR/jmeter.log"
ENGINE_EXIT=$?
printf '%s
' "$ENGINE_EXIT" >"$FAIL_DIR/engine-exit.txt"

Do not use -f. Do not delete the failure directory after observing the expected error.

8. Verify non-success evidence

Require:

• non-zero engine/process exit;
• jmeter.log or console evidence identifies the missing/unloadable JMX path;
• no valid eight-row workload for p21-missing;
• python tools/analyze_events.py results/server-events.jsonl p21-missing reports zero work events;
• run A artifacts/hashes are unchanged.

This is an engine/input failure, not an SLO failure.

9. Restore with the validated launcher from a clean shell

PowerShell:

Set-Location "$env:TEMP"
& "F:\Labs\p21-cli-lab\tools\run-local.ps1" -RunId p21-check-b
$LASTEXITCODE

Run B must create a new directory and use the real cli-local.jmx without touching p21-missing.

10. Compare successful run A and B

Verify in both manifests:

• same JMX SHA-256;
• same property-file SHA-256;
• same JMeter/Java expected baseline;
• same configured 2×4/20 ms load and eight achieved successful samples;
• same target operation distribution;
• different run IDs/output directories and possibly caller cwd;
• both reports reconstruct from their retained JTL.

11. Prove report reproducibility from raw JTL

Generate a second report from run A:

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -g "F:\Labs\p21-cli-lab\results\p21-check-a\results.jtl" `
  -o "F:\Labs\p21-cli-lab\results\p21-check-a\html-report-rebuilt"

The rebuilt report is derived from preserved raw results; no target traffic is generated.

12. Generator/measurement validity

Record generator CPU/memory/network/disk during one successful run. The workload is intentionally tiny; the expected conclusion is only that the CLI/evidence mechanics are valid under local headroom.

Configured load = 8 work samples. Achieved load = 8 successful JTL samples and 8 target events. Never turn this localhost number into a production capacity statement.

13. Required evidence packet

14. Validity statement

15. Verification checklist

• Target exactly 127.0.0.1:8021.
• JMeter 5.6.3 / Java 17 / no plugins recorded.
• No load run in GUI.
• Safe wrapper never uses -f and refuses existing run IDs.
• JMX/property SHA-256 recorded.
• Valid runs have engine exit=0 and independent gate=0.
• Each valid run has exactly eight JTL samples and eight target work events.
• Missing-JMX run has non-success engine evidence and zero target work.
• HTML report exists and can be regenerated later from JTL.
• No secrets appear in commands, properties, logs or manifests.

16. Cleanup / rollback

1. Stop the localhost fixture after target evidence is captured.
2. Keep JTL/jmeter.log/reports/manifests/version/hash/failure evidence until review completes.
3. Delete only disposable local run directories after review; never delete historical evidence as part of rerun startup.
4. No public/production API, credential, recorder certificate, remote RMI engine, container, database/message service, paid platform, CI secret or OS/JVM global tuning was changed.

17. What Chapter 21 adds to the operating model

The production performance-testing operating model now has a headless execution/evidence contract: approved JMeter/Java versions; explicit JMX/property/data hashes; safe CLI flags; immutable run directory; raw JTL; run-specific jmeter.log; optional reproducible dashboard; exact command; process exit; independent correctness/performance gate; generator resource evidence; target operation evidence; and retention/cleanup rules are required before a run becomes CI/scheduled release evidence.

Chapter 22 moves to Listeners, Result Collection, Memory Cost, and Safe Debugging. It deepens the result-side contract by showing which listeners are safe during authoring/load, what they retain in memory, how result fields/save settings change cost, and how to debug without invalidating the generator.

Knowledge check

Official references and version notes

• JMeter Getting Started — CLI mode — -n, -t, -l, -j, -g, -e, -o, -J, -L, remote flags, and CLI/load guidance.
• JMeter Listeners / Result files — CSV versus XML, save-service defaults, CLI -l listener, result fields, and memory guidance.
• JMeter Generating Dashboard Report — dashboard-required CSV fields, -g, -e -o, output-folder rules, graphs, and report properties.
• JMeter Best Practices — GUI authoring/debugging and CLI execution for load.
• Apache JMeter downloads — current stable release and Java requirement.