CLI Mode, Headless Execution, Result Files, and Reproducible Runs: Core Concepts and Mental Model

1. The practical problem: “I ran the same JMX” is not reproducibility

Two engineers can load the same JMX and still run different experiments: one uses GUI, one CLI; one has 20 threads from a property file, one silently defaults to 1; one overwrites yesterday's JTL; one enables verbose logs; one runs JMeter 5.6.3/Java 17 while another has a floating installation. If only the HTML dashboard survives, the original execution cannot be reconstructed.

2. Mental model: explicit inputs → one engine process → raw artifacts → independent gate

flowchart TD
I[Versioned JMX + property files + deterministic data] --> C[Explicit jmeter -n CLI command]
C --> J[JVM / JMeter engine]
J --> T[Bounded thread execution]
T --> S[Authorized local target]
J --> R[JTL sample results]
J --> L[jmeter.log engine diagnostics]
J --> O[Process exit status]
R --> H[Optional HTML dashboard]
R --> G[Post-run sample/SLO gate]
L --> G
O --> G
M[Run manifest: command + versions + hashes + paths] --> G
S --> E[Target event telemetry]
E --> G

The JMX/property/data files are immutable inputs. The launcher constructs one exact CLI command and starts one JMeter JVM/engine. The engine creates thread activity against the target. Raw sample outcomes go to JTL; engine/classpath/configuration messages go to jmeter.log; the shell receives a process exit status. The HTML dashboard is derived evidence, not the raw source. A post-run validator then checks sample count, failures, latency and artifact integrity. The run manifest binds command, versions, hashes and paths to those outputs, while target telemetry independently confirms what work reached the service.

3. The core CLI contract

4. GUI versus CLI

Use GUI to build, inspect scope, run tiny validation/debug checks, and temporarily inspect a few samples. Current JMeter guidance explicitly says not to run load tests in GUI mode. GUI components/renderers consume generator resources and a GUI click is harder to automate faithfully.

Use CLI for meaningful load because it gives explicit files, command-line properties, predictable log/result destinations, shell exit handling and automation hooks.

5. JTL is raw sample evidence

The -l file records the top-level test results using save-service settings. Current CSV defaults can include timestamp, elapsed, label, response code/message, thread name, success/failure message, bytes sent/received and active-thread counts. CSV is much smaller than XML and cannot store response bodies, which makes it a safer default for repeated load samples.

JTL is not just “data for the dashboard.” Your post-run gate should read it directly before derived reports are accepted.

6. jmeter.log is engine evidence

JTL answers “what happened to samples?” jmeter.log answers “what happened inside JMeter?” It captures plan loading, component/plugin/classpath messages, script exceptions, remote/engine problems and other diagnostics that may not appear as ordinary target samples.

Always use -j to put it inside the run directory. Do not rely on whatever default jmeter.log happens to exist in the launch directory.

7. HTML report is derived evidence

-e -o generates the dashboard after a successful load execution path; -g results.jtl -o report can generate it later from an existing compatible CSV log. The dashboard is useful for analysis but should be reproducible from the retained raw JTL/properties.

The output directory must be new/empty. A safe launcher creates the parent run directory but deliberately does not pre-create the report directory.

8. Process exit is not the performance gate

An engine/process exit code answers whether the JMeter invocation itself completed according to its process-level rules. It does not replace sample assertions, error-rate checks, expected-row counts or latency thresholds. A plan can finish as a process while some HTTP samples are unsuccessful.

Therefore the launcher records two statuses: engine_exit_code and a separate post-run gate_exit_code. CI should fail if either is non-zero, while preserving which layer failed.

9. Run manifest binds inputs to outputs

A manifest should record at least:

• run ID and creation time;
• caller cwd and resolved project root;
• exact command tokens;
• JMX/property-file absolute paths and SHA-256 hashes;
• JMeter and Java version output;
• JTL/log/report paths;
• engine exit and gate exit;
• gate metrics/sample counts.

Do not put passwords/tokens into command manifests.

10. State checklist before execution

11. Read-only preflight

PowerShell:

& "$env:JMETER_HOME\bin\jmeter.bat" -v
java -version
Get-FileHash .\plans\cli-local.jmx -Algorithm SHA256
Get-FileHash .\config\local.properties -Algorithm SHA256
Get-Content .\config\local.properties
Test-Path .\results\planned-run-id
curl.exe --fail --silent http://127.0.0.1:8021/health
curl.exe --fail --silent http://127.0.0.1:8021/stats

Nothing above generates load. It proves current tool/input/path/target state first.

12. Why the safe launcher does not use -f

Current JMeter exposes -f/--forceDeleteResultFile to delete an existing result file and report folder before starting. That is convenient for disposable automation but dangerous with a misresolved/user-controlled path because it intentionally destroys evidence.

This chapter instead creates a unique run directory and refuses if it already exists. Historical evidence is never deleted as part of “run again.”

13. DevOps connection

A performance test becomes automatable when the same explicit input/command/output contract works in an interactive shell and an unattended runner. The runner should not need GUI state or undocumented local files; it should only need the versioned project, pinned JMeter/Java runtime, authorized target and sufficient generator resources.

Knowledge check

Official references and version notes

• JMeter Getting Started — CLI mode — -n, -t, -l, -j, -g, -e, -o, -J, -L, remote flags, and CLI/load guidance.
• JMeter Listeners / Result files — CSV versus XML, save-service defaults, CLI -l listener, result fields, and memory guidance.
• JMeter Generating Dashboard Report — dashboard-required CSV fields, -g, -e -o, output-folder rules, graphs, and report properties.
• JMeter Best Practices — GUI authoring/debugging and CLI execution for load.
• Apache JMeter downloads — current stable release and Java requirement.