Chapter 33Lesson 01~195 minutes

Troubleshooting Out-of-Memory, Socket, SSL, DNS, and Distributed Failures: Core Concepts and Mental Model

Chapter 32 showed that classpath and extension drift can look like a target failure. Chapter 33 generalizes the same preserve-first discipline across the JVM, sockets, DNS, TLS, remote engines, CI/container state, and the system under test. A red sample is only a symptom; recovery starts by proving which layer owns that symptom.

Preserve firstOOMSocket/DNSTLSRemote parity

Learning objectives

Use symptom → preserve evidence → classify layer → minimal reproduction → one-factor correction → verification.
Distinguish plan/config defects, generator exhaustion, network/DNS/TLS failures, remote-engine mismatch, and SUT failure.
Define generator/thread/property/session/result/trust state before changing it.
Use non-destructive inspection before logging/tuning changes.
Return to the original configured and achieved load before capacity/regression conclusions.

1. The practical problem

A run fails and several people “fix” it simultaneously: heap is increased, retries are enabled, certificate checks are bypassed, hosts files are edited, and the target is restarted. The next run is green, but the original cause is gone. Fast troubleshooting is the opposite: preserve the first JTL and jmeter.log, classify the owner layer, reduce to the smallest reproduction, change one state, and verify the original workload afterward.

Mandatory safety boundary: target traffic stays on learner-controlled loopback. HTTP=127.0.0.1:8033; HTTPS=localhost:8443. Baseline=2 threads×5 loops=10 target samples. Negative network/DNS/TLS tests=1×1. Global fixture ceiling=40. The OOM case uses a separate 1×1 JMeter JVM with -Xmx64m and no HTTP sampler. No public/production target, real credential, RMI traffic, global DNS edit, trust-all, or system-wide JVM tuning is required.

2. Mental model

Preserve-first troubleshooting flow

The failure message is a symptom. Preserve the first state, locate the owning layer, then make one falsifiable correction before rerunning.

flowchart TD
A[Failure symptom] --> B[Preserve JTL + jmeter.log + process/engine + target evidence]
B --> C{Classify owner}
C --> P[Plan/config/scope/data]
C --> G[Generator JVM/CPU/heap/socket]
C --> N[DNS/network/connect]
C --> T[TLS trust/identity]
C --> R[Remote engine/JAR/data/version]
C --> S[SUT/dependency]
P --> I[Inspect smallest relevant state]
G --> I
N --> I
T --> I
R --> I
S --> I
I --> M[Minimal authorized reproduction]
M --> F[Change one factor]
F --> V[Verify original workload + generator validity]

JTL records sampler outcomes; jmeter.log records engine/component exceptions; JVM/OS state explains generator resource failures; resolver/socket/certificate state explains connection establishment; per-engine manifests explain remote-only failures; target logs/telemetry explain requests that reached the SUT. The arrow from “one factor” to “verify original workload” is critical: a one-sample fix is not yet a valid load-test recovery.

3. Core failure terms

Term	Meaning
Generator exhaustion	JMeter's JVM/OS/network cannot sustain the intended generator work.
Connection refused	Host/address resolved, but nothing accepts TCP at the selected endpoint/port.
UnknownHost	Name resolution failed before a target connection existed.
TLS trust/identity failure	TCP can connect, but certificate chain/hostname/trust cannot authenticate the TLS peer.
Remote mismatch	Controller/engine files, JMeter/Java/JAR/property/network state differ.
First-failure bundle	Original JTL, jmeter.log, engine/process output, target evidence, exact inputs and hashes.
Minimal reproduction	Smallest authorized test that preserves the same failure class.

4. State inventory

Boundary	Inspect
Generator	JMeter/Java version, JVM command line, heap/GC/CPU, sockets/file descriptors, listener/result cost.
Thread/arrival	Threads, loops, timers, retries, configured versus achieved samples.
Tree/scope	HTTP Defaults, host/port/protocol, DNS Cache Manager, assertions, scripts/listeners.
Variables/properties/data	Resolved target, property files, CSV/JAR/file presence on each engine.
Protocol/session	HttpClient4, connection reuse, DNS resolver/cache, TLS truststore, cookies/tokens.
Target	Listener ports, health, certificate, request count, logs, dependencies.
Artifacts	JTL, jmeter.log, target events, diagnostic command output, optional heap dump.
Trust boundary	Certificates/keystores, plugin/JMX code, environment/system properties, RMI identities.
Validity	Did DEBUG, heap, retry, DNS/trust, engine-count or listener changes alter measurement?

5. Read-only inspection first

& "$env:JMETER_HOME\bin\jmeter.bat" -v
java -version
jcmd -l

Get-Content .\config\troubleshooting.properties
Get-FileHash .\plans\*.jmx, .\config\*.properties -Algorithm SHA256

Get-NetTCPConnection -State Listen -ErrorAction SilentlyContinue |
  Where-Object LocalPort -in 8033,8443,6553

Resolve-DnsName localhost
Resolve-DnsName p33.invalid -ErrorAction SilentlyContinue

Get-Content .\results\*\jmeter.log -Tail 80 -ErrorAction SilentlyContinue

Do not edit hosts, truststores, heap, retries, listeners, or target configuration yet. First prove what failed.

6. Narrow debug windows

JMeter 5.6.3 supports -L[category]=level. Use category DEBUG only for a short minimal reproduction; root DEBUG can add disk/CPU/privacy cost and itself distort the generator.

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -Lorg.apache.jmeter.protocol.http=DEBUG `
  -n -t .\plans\minimal-repro.jmx `
  -l .\results\minimal\results.jtl `
  -j .\results\minimal\jmeter.log

7. JVM evidence before “more heap”

JDK17 jcmd offers process and heap inspection. GC.heap_info is lower-impact than a full heap dump. Dumps can pause the JVM, consume disk, and contain response bodies/credentials, so they are sensitive evidence. The lab's optional OOM case is intentionally tiny/synthetic to bound this risk.

8. DNS is cached state

Java17 caches successful and failed lookups; the documented default negative TTL is 10 seconds. JMeter DNS Cache Manager has its own per-thread cache for HttpClient4 and supports a scoped static mapping. That gives a reproducible fix for p33.invalid without a global hosts-file hack.

9. Distributed state is engine-local

Remote JMeter sends the plan, but each server runs the complete plan. Current documentation requires exactly the same JMeter version and recommends the same Java version; data files are not automatically copied. Chapter32 adds JAR/plugin parity to that inventory.

10. DevOps connection

Recovery becomes reliable when evidence survives restarts and fixes are reviewable. First-failure bundles, minimal reproductions, parity checks, and one-factor corrections can become CI preflights and runbooks instead of tribal “restart until green” knowledge.

Knowledge check

Why preserve JTL and jmeter.log before retrying?

What distinguishes DNS failure from socket refusal?

Why is trust-all not a TLS fix?

Why inspect every remote engine?

When is recovery valid for performance conclusions?

Next lesson

Inject five bounded failure classes

Lesson2 runs controlled OOM, socket, DNS, TLS and simulated engine-parity failures and preserves exact before/after evidence.

Official references and version notes

Apache JMeter downloads — JMeter 5.6.3 and Java 8+.
JMeter changes — Java 17+ recommended for 5.6.x.
Getting Started — CLI, -l, -j, -L, JVM startup settings.
Best Practices — CLI load execution, listener/memory cost, JSR223 Groovy.
DNS Cache Manager — scoped HttpClient4 DNS caching/static mapping.
Remote Testing — same JMeter versions, Java parity, data files, RMI SSL.
JDK 17 jcmd — JVM diagnostics and command impact.
JDK 17 memory troubleshooting — heap-dump/OOM evidence.
JDK 17 networking properties — positive/negative DNS cache behavior.

Version and compatibility note

Checked against current primary documentation on 2026-09-05. Mandatory runtime: Apache JMeter 5.6.3, Java 17, no third-party plugin. Meaningful runs use CLI with raw CSV JTL and a matching jmeter.log. Temporary logging uses category-specific -L...=DEBUG only for minimal reproductions. The bounded OOM case overrides JVM heap only for one disposable process. DNS Cache Manager is the scoped fallback for the p33.invalid exercise; Java 17 negative DNS cache defaults to 10 seconds. Remote JMeter runs the complete plan on each engine; data files are not automatically copied. RMI uses SSL by default and is not disabled in this chapter.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20

0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.