Troubleshooting Out-of-Memory, Socket, SSL, DNS, and Distributed Failures: Diagnostics, Failure Modes, and Production Practices

1. Required diagnostic sequence

flowchart TD
P[Preserve first-failure JTL / jmeter.log / engine / target evidence] --> V[Confirm JMeter / Java / plugin / tool versions]
V --> I[Confirm JMX / data / properties / CLI / authorized target]
I --> S[Validate tree scope + resolved variables/properties]
S --> D[Inspect protocol/session/data]
D --> G[Inspect generator JVM/OS/network]
G --> T[Inspect SUT telemetry]
T --> R[Inspect remote/CI/container state]
R --> F[Least destructive correction]
F --> M[Smallest controlled rerun]
M --> O[Original workload verification]

2. Blind heap increases

An OOM occurs with a verbose listener, retained response data or a pathological script. Jumping from1GiB to8GiB may only delay the same retention problem and change GC behavior. Inspect listener/result settings, live-set trend and generator headroom first. Increase heap only when sizing evidence proves legitimate demand and host memory supports it.

3. Blanket retries

Connection refused, UnknownHost, PKIX and missing files do not become correct after five attempts. Retries add load and time. Keep httpclient4.retrycount=0 for diagnostics; model legitimate application retry behavior separately and explicitly.

4. Intentionally broken TLS example

# anti-patterns — do not use
server.rmi.ssl.disable=true
Trust All Certificates = true
# or weakening global Java trust/security settings for one lab certificate

If jmeter.log shows PKIX or hostname failure, inspect certificate/trust/hostname. Lesson2 repairs only the scoped test truststore. Current remote transport uses SSL by default; configure RMI keystore/network rather than disabling it.

5. Global hosts/DNS edit without rollback

A forgotten hosts entry can redirect browsers, CI and future JMeter runs. Prefer JMeter DNS Cache Manager for test-scoped HttpClient4 mappings. If an OS/DNS change is genuinely required, record exact before/after state, authorization and rollback.

6. Deleting logs before analysis

Deleting or overwriting results/ removes the exception/timing evidence that distinguishes the layer. Use a new directory for each rerun, hash the first failure, and clean up only after retention policy.

7. Blaming the SUT for generator socket exhaustion

Ephemeral-port/file-descriptor pressure, connection churn or TLS CPU can limit the injector while target telemetry stays quiet. Inspect generator socket counts/CPU/GC and achieved RPS before scaling the target.

8. Fixing one engine but not the fleet

Manually copying a CSV/JAR to Engine B while A/C/D remain different creates a non-reproducible fleet. Provision exact JMeter/Java/data/plugin/property hashes and make parity a preflight.

9. Controller/result-transfer pressure

Remote result transfer can overload the controller/network. That is generator/control-plane overhead, not automatically SUT latency. Preserve controller/engine logs and resource state. Change sender/result strategy only after evidence identifies the bottleneck.

10. CI/container startup versus SUT failure

A service container may not be ready, DNS/service names may differ, mounts can miss CSV/JARs, or the runner may have less heap/socket capacity than a laptop. Capture image/runner identity, service logs, mounted-file inventory and preflight. Pipeline timeout alone does not prove server slowness.

11. Causal symptom table

12. Shortcuts explicitly rejected

• Do not add blanket retries or arbitrary long sleeps.
• Do not assign giant heaps without evidence.
• Do not mass-disable listeners/evidence without measured cost.
• Do not use global property hacks.
• Do not disable TLS/RMI verification.
• Do not edit production/public DNS/hosts for an experiment.
• Do not increase workload while the layer is unresolved.
• Do not delete result files before preserving first-failure evidence.

13. Security-sensitive evidence

Heap dumps/JFR/debug logs can retain credentials or bodies; cert private keys/truststores are identity material; environment/property files may contain secrets; RMI/packet capture broadens network exposure. This chapter uses synthetic data, one-day local certificates and no real RMI traffic.

Knowledge check

Official references and version notes

• Apache JMeter downloads — JMeter 5.6.3 and Java 8+.
• JMeter changes — Java 17+ recommended for 5.6.x.
• Getting Started — CLI, -l, -j, -L, JVM startup settings.
• Best Practices — CLI load execution, listener/memory cost, JSR223 Groovy.
• DNS Cache Manager — scoped HttpClient4 DNS caching/static mapping.
• Remote Testing — same JMeter versions, Java parity, data files, RMI SSL.
• JDK 17 jcmd — JVM diagnostics and command impact.
• JDK 17 memory troubleshooting — heap-dump/OOM evidence.
• JDK 17 networking properties — positive/negative DNS cache behavior.