CLI Mode, Headless Execution, Result Files, and Reproducible Runs: Diagnostics, Failure Modes, and Production Practices

1. Preserve first-failure evidence

2. Diagnostic sequence

flowchart TD
E[Preserve command + JTL + jmeter.log + manifest + target evidence] --> V[Confirm JMeter / Java / plugin / tool versions]
V --> C[Confirm exact JMX / data / properties / CLI + authorized target]
C --> S[Validate tree scope + resolved variables/properties + expected sample count]
S --> P[Inspect protocol/session/data state]
P --> G[Inspect generator JVM / CPU / GC / disk / network]
G --> T[Inspect SUT telemetry / received work]
T --> X[Inspect CI/container/remote workspace/result-transfer state if relevant]
X --> F[Least destructive correction]
F --> R[Small controlled rerun]

3. Failure mode: running meaningful load in GUI

Symptoms: higher generator CPU/memory, UI freezes, retained listener data and harder-to-reproduce results. Repair: use GUI only to author/validate tiny cases; save the JMX and run the bounded load through the recorded CLI contract. Do not conclude the server is slow from a saturated GUI injector.

4. Failure mode: overwriting prior evidence

Broken:

jmeter -n -t plan.jmx -l results/results.jtl -j results/jmeter.log
# repeated tomorrow with the same paths

Depending on file/report behavior and surrounding scripts, the second invocation can mix/replace evidence or fail ambiguously. Repair: allocate a new run ID directory and refuse reuse.

5. Failure mode: unsafe -f

Broken automation constructs output paths from an unchecked variable and adds -f “so stale files never block the job.” Current JMeter defines -f to force-delete existing result files and web report folders.

Repair: remove -f from the general launcher. Use immutable run directories. If a specialized disposable workspace ever uses -f, verify the resolved path is a dedicated generated child of a known workspace before invoking JMeter.

6. Failure mode: embedding credentials in CLI

Commands can be visible in process listings, shell history, CI logs and manifests. Current JMeter docs explicitly warn that proxy credentials on the command line may be visible to other system users.

Repair: keep non-secret workload knobs in -J; inject real secrets using an approved secret mechanism that is not echoed/manifested, and redact diagnostic output. The mandatory fixture uses no authentication.

7. Failure mode: ignoring jmeter.log

A JTL may be empty or shorter than expected because JMX parsing, script/class/plugin initialization, report generation or engine shutdown had a problem. If the wrapper only parses JTL and discards jmeter.log, the root cause is lost.

Always retain -j run/jmeter.log and search ERROR/FATAL/WARN context before changing the SUT.

8. Failure mode: process exit success interpreted as SLO success

A JMeter invocation can complete at the process layer while JTL contains HTTP/assertion failures or an unacceptable p95. That is why the Chapter 21 gate parses result rows independently.

Never rewrite the engine exit code to “failed because p95” without also preserving the original engine status. Store both.

9. Failure mode: floating JMeter/Java versions

Two scheduled runners pick different JMeter or Java versions from PATH. Behavior/performance changes can then be attributed incorrectly to the SUT.

Repair: standardize/pin an approved runtime, capture jmeter -v and java -version every run, and reject/flag unexpected versions before a regression comparison.

10. Intentionally broken example: missing JMX file

Create a dedicated failure directory, then intentionally invoke a nonexistent plan directly so the JMeter process—not the safe wrapper's preflight—owns the failure:

$FailDir = "F:\Labs\p21-cli-lab\results\p21-missing-jmx"
New-Item -ItemType Directory -Force $FailDir | Out-Null

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n `
  -t "F:\Labs\p21-cli-lab\plans\missing-plan.jmx" `
  -q "F:\Labs\p21-cli-lab\config\local.properties" `
  -l "$FailDir\results.jtl" `
  -j "$FailDir\jmeter.log"

$EngineExit = $LASTEXITCODE
$EngineExit | Set-Content "$FailDir\engine-exit.txt"

Expected: non-success process/engine evidence, jmeter.log indicating plan/file load failure, no valid 8-sample JTL workload, and no matching target work events. Preserve all files. The target is healthy; this failure occurred before workload execution.

Repair: do not copy a random replacement plan or switch target. Use the existing hashed plans/cli-local.jmx through the safe launcher with a new run ID.

11. Failure mode: report output directory already contains files

-o requires a new/empty destination. A fixed report-output path from yesterday can make report generation fail even when JTL is valid.

Repair: generate inside a unique run directory or use a new directory for later -g. Preserve the raw JTL; report generation can be retried without rerunning target load.

12. Failure mode: global DEBUG under load

-LDEBUG can dramatically enlarge jmeter.log and consume generator CPU/disk. Use a narrow category only in a small reproduction. Record any logging override because the generator workload changed.

13. Causal symptom table

14. Troubleshooting shortcuts to reject

• Do not add blanket retries or arbitrary long sleeps.
• Do not raise heap before identifying avoidable listeners/log/result cost.
• Do not mass-disable listeners without evidence and then lose required diagnostics.
• Do not use global property hacks to force missing runtime state.
• Do not disable TLS/RMI verification.
• Do not test a production endpoint because localhost/CI paths failed.
• Do not increase threads/sample count while process/artifact validity is unresolved.
• Do not delete failed JTL/log/manifests to free space before diagnosis.

Knowledge check

Official references and version notes

• JMeter Getting Started — CLI mode — -n, -t, -l, -j, -g, -e, -o, -J, -L, remote flags, and CLI/load guidance.
• JMeter Listeners / Result files — CSV versus XML, save-service defaults, CLI -l listener, result fields, and memory guidance.
• JMeter Generating Dashboard Report — dashboard-required CSV fields, -g, -e -o, output-folder rules, graphs, and report properties.
• JMeter Best Practices — GUI authoring/debugging and CLI execution for load.
• Apache JMeter downloads — current stable release and Java requirement.