Troubleshooting Out-of-Memory, Socket, SSL, DNS, and Distributed Failures: Guided Hands-On Workflow

1. Lab layout and ceilings

p33-troubleshooting/
├── fixtures/troubleshooting_fixture.py
├── plans/{baseline,socket-refused,dns-negative,dns-fixed,tls-localhost,heap-pressure}.jmx
├── config/troubleshooting.properties
├── tls/
├── simulated-engines/
├── tools/{collect_diagnostics,check_engine_parity,failure_matrix}.py
└── results/

2. Local HTTP/HTTPS fixture

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlparse, parse_qs
import argparse, json, ssl, threading, time

FIXTURE_VERSION = "prompt33-troubleshooting-fixture-v1"
lock = threading.Lock()
event_log = None
max_requests = 40
body_kb = 4
state = {"requests": 0, "successes": 0, "rejections": 0}

def snapshot():
    with lock:
        return {"fixture_version": FIXTURE_VERSION, "max_requests": max_requests, "body_kb": body_kb, **state}

def write_event(event):
    with lock:
        with event_log.open("a", encoding="utf-8") as h:
            h.write(json.dumps(event, sort_keys=True) + "\n")

class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def send_json(self, status, payload):
        raw = json.dumps(payload, sort_keys=True).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.send_header("X-Fixture-Version", FIXTURE_VERSION)
        self.end_headers()
        self.wfile.write(raw)

    def do_GET(self):
        parsed = urlparse(self.path)
        if parsed.path == "/health":
            self.send_json(200, {"status": "ok", "state": snapshot()}); return
        if parsed.path == "/stats":
            self.send_json(200, {"status": "ok", "state": snapshot()}); return
        if parsed.path != "/work":
            self.send_json(404, {"status": "not_found"}); return
        q = parse_qs(parsed.query)
        run_id = q.get("run_id", [""])[0]
        seq = q.get("seq", [""])[0]
        with lock:
            state["requests"] += 1
            n = state["requests"]
        if n > max_requests:
            with lock: state["rejections"] += 1
            self.send_json(429, {"status": "sample_ceiling"}); return
        time.sleep(0.020)
        payload = "x" * (body_kb * 1024)
        with lock: state["successes"] += 1
        ended = int(time.time() * 1000)
        self.send_json(200, {"status": "ok", "run_id": run_id, "seq": seq, "payload": payload})
        write_event({"ts_ms": ended, "operation": "work", "status": 200, "run_id": run_id, "seq": seq, "body_kb": body_kb, "service_ms": 20})

    def log_message(self, format, *args):
        return

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--host", default="127.0.0.1")
    p.add_argument("--port", type=int, default=8033)
    p.add_argument("--max-requests", type=int, default=40)
    p.add_argument("--body-kb", type=int, default=4)
    p.add_argument("--log", required=True)
    p.add_argument("--tls-cert")
    p.add_argument("--tls-key")
    args = p.parse_args()
    if args.host != "127.0.0.1":
        raise SystemExit("fixture may bind only to 127.0.0.1")
    if not 1 <= args.max_requests <= 40:
        raise SystemExit("max-requests must be 1..40")
    if not 1 <= args.body_kb <= 64:
        raise SystemExit("body-kb must be 1..64")
    if bool(args.tls_cert) != bool(args.tls_key):
        raise SystemExit("provide both --tls-cert and --tls-key")
    global event_log, max_requests, body_kb
    max_requests, body_kb = args.max_requests, args.body_kb
    event_log = Path(args.log).resolve()
    event_log.parent.mkdir(parents=True, exist_ok=True)
    event_log.write_text("", encoding="utf-8")
    server = ThreadingHTTPServer((args.host, args.port), Handler)
    scheme = "http"
    if args.tls_cert:
        ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
        ctx.load_cert_chain(args.tls_cert, args.tls_key)
        server.socket = ctx.wrap_socket(server.socket, server_side=True)
        scheme = "https"
    print(f"fixture_version={FIXTURE_VERSION}", flush=True)
    print(f"listen={scheme}://{args.host}:{args.port}", flush=True)
    print(f"max_requests={max_requests} body_kb={body_kb}", flush=True)
    server.serve_forever()

if __name__ == "__main__":
    main()

Run HTTP on8033 for baseline/socket/DNS. The same script can wrap TLS on8443 for the certificate exercise. Target JSONL independently proves whether the failed case reached the service.

3. Known-good baseline

config/troubleshooting.properties:

target.host=127.0.0.1
target.port=8033
threads=2
loops=5
pacing.ms=150
connect.timeout.ms=500
response.timeout.ms=1500
jmeter.httpsampler=HttpClient4
httpclient4.retrycount=0
jmeter.save.saveservice.output_format=csv
jmeter.save.saveservice.print_field_names=true
jmeter.save.saveservice.timestamp_format=ms
jmeter.save.saveservice.time=true
jmeter.save.saveservice.label=true
jmeter.save.saveservice.response_code=true
jmeter.save.saveservice.response_message=true
jmeter.save.saveservice.thread_name=true
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.bytes=true
jmeter.save.saveservice.sent_bytes=true
jmeter.save.saveservice.thread_counts=true
jmeter.save.saveservice.latency=true
jmeter.save.saveservice.connect_time=true
jmeter.save.saveservice.assertion_results_failure_message=true
jmeter.save.saveservice.response_data=false
jmeter.save.saveservice.response_data.on_error=false
jmeter.save.saveservice.samplerData=false
jmeter.save.saveservice.responseHeaders=false
jmeter.save.saveservice.requestHeaders=false
jmeter.save.saveservice.url=false

JMX tree:

Test Plan
├── HTTP Request Defaults
│   host=${__P(target.host,127.0.0.1)}
│   port=${__P(target.port,8033)}
│   implementation=HttpClient4
│   connect timeout=${__P(connect.timeout.ms,500)}
│   response timeout=${__P(response.timeout.ms,1500)}
└── Thread Group — baseline
    threads=${__P(threads,2)}
    loops=${__P(loops,5)}
    Action after Sampler error=Stop Thread
    ├── Counter -> SEQ (per user)
    └── HTTP Request — Work
        GET /work?run_id=${__P(run.id,p33)}&seq=T${__threadNum}:${SEQ}
        ├── Constant Timer ${__P(pacing.ms,150)} ms
        └── Response Assertion: HTTP code = 200

Expected baseline:
- 2×5 = 10 configured HTTP samples
- 10 target events
- CLI JTL + matching jmeter.log
- no retry behavior

Start the fixture with max40/body4KiB. Run CLI mode into results/baseline/. Require10 Work rows,10 target events, JMeter5.6.3/Java17 and a matching jmeter.log.

4. First-failure bundle helper

import argparse, hashlib, json, platform, subprocess
from pathlib import Path

def sha256(path):
    h = hashlib.sha256()
    with Path(path).open("rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""): h.update(chunk)
    return h.hexdigest()

def maybe_run(cmd):
    try:
        cp = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
        return {"returncode": cp.returncode, "stdout": cp.stdout, "stderr": cp.stderr}
    except Exception as exc:
        return {"error": str(exc)}

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--case", required=True)
    p.add_argument("--jtl")
    p.add_argument("--jmeter-log")
    p.add_argument("--out", required=True)
    args = p.parse_args()
    result = {
        "case": args.case,
        "platform": platform.platform(),
        "python": platform.python_version(),
        "commands": {"java_version": maybe_run(["java", "-version"]), "jcmd_list": maybe_run(["jcmd", "-l"])},
        "files": {}
    }
    for label, value in (("jtl", args.jtl), ("jmeter_log", args.jmeter_log)):
        if value:
            pth = Path(value).resolve()
            result["files"][label] = {"path": str(pth), "exists": pth.exists(), "bytes": pth.stat().st_size if pth.exists() else None, "sha256": sha256(pth) if pth.exists() else None}
    Path(args.out).write_text(json.dumps(result, indent=2), encoding="utf-8")
    print(json.dumps(result, indent=2))

if __name__ == "__main__":
    main()

It hashes JTL/logs and captures Java/jcmd process evidence without changing the target or JVM.

5. Failure A — refused local port

Negative copy of baseline:
  host=127.0.0.1
  port=6553
  threads=1
  loops=1
  connect timeout=500ms

Expected:
  Non HTTP response code: java.net.ConnectException
  Connection refused wording may differ by OS.
  Target events remain unchanged.

Get-NetTCPConnection -State Listen -ErrorAction SilentlyContinue |
  Where-Object LocalPort -in 6553,8033
Test-NetConnection 127.0.0.1 -Port 6553
Test-NetConnection 127.0.0.1 -Port 8033

Fix only the port back to8033 and rerun one sample. A retry repeats the wrong endpoint; it is not the correction.

6. Failure B — name resolution

Negative:
  host=p33.invalid
  port=8033
  implementation=HttpClient4
  threads=1, loops=1

Expected:
  java.net.UnknownHostException
  target receives zero requests.

Scoped fix at Test Plan/root:
  DNS Cache Manager
    Use custom DNS resolver = enabled
    Static Host Table:
      p33.invalid -> 127.0.0.1

Then use the same HTTP Request with host=p33.invalid.
Run the fixed case in a fresh JMeter process.

.invalid is deliberately non-public. Preserve the first UnknownHost JTL/log. Do not edit the OS hosts file or global Java DNS security properties.

7. Failure C — local TLS trust

New-Item -ItemType Directory -Force .\tls | Out-Null

openssl req -x509 -newkey rsa:2048 -nodes `
  -keyout .\tls\localhost.key `
  -out .\tls\localhost.crt `
  -days 1 `
  -subj "/CN=localhost" `
  -addext "subjectAltName=DNS:localhost"

python .\fixtures\troubleshooting_fixture.py `
  --host 127.0.0.1 --port 8443 `
  --max-requests 10 --body-kb 4 `
  --tls-cert .\tls\localhost.crt `
  --tls-key .\tls\localhost.key `
  --log .\results\tls-target-events.jsonl

Run one https://localhost:8443/work sample without a custom truststore. Expect SSLHandshakeException/PKIX path building failed or equivalent trust failure. Inspect the cert:

openssl x509 -in .\tls\localhost.crt -noout -subject -issuer -dates -ext subjectAltName
keytool -printcert -file .\tls\localhost.crt

Fix with a scoped truststore:

keytool -importcert -noprompt `
  -alias p33-localhost `
  -file .\tls\localhost.crt `
  -keystore .\tls\p33-truststore.p12 `
  -storetype PKCS12 `
  -storepass changeit

$TrustPath=(Resolve-Path .\tls\p33-truststore.p12).Path.Replace("\","/")
@"
javax.net.ssl.trustStore=$TrustPath
javax.net.ssl.trustStorePassword=changeit
javax.net.ssl.trustStoreType=PKCS12
"@ | Set-Content .\tls\tls-system.properties

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -S .\tls\tls-system.properties `
  -n -t .\plans\tls-localhost.jmx `
  -l .\results\tls-fixed\results.jtl `
  -j .\results\tls-fixed\jmeter.log

The password is fake local training data. Verification remains enabled. Do not use Trust All, global trust changes, or server.rmi.ssl.disable=true.

8. Failure D — bounded JMeter OOM

The plan is one JSR223 Groovy sampler, one thread, one loop, no HTTP:

// Intentionally destructive only to this one tiny disposable JMeter JVM.
// Run exactly once with 1 thread, 1 loop and Xmx=64m.
def blocks = []
96.times {
    blocks.add(new byte[1024 * 1024])
}
SampleResult.setSuccessful(true)
SampleResult.setResponseMessage("unexpected: allocation completed")

New-Item -ItemType Directory -Force .\results\heap | Out-Null
$HeapDump=(Resolve-Path .\results\heap).Path
$OldJvmArgs=$env:JVM_ARGS
$env:JVM_ARGS="-Xms32m -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$HeapDump"

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n -t .\plans\heap-pressure.jmx `
  -l .\results\heap\results.jtl `
  -j .\results\heap\jmeter.log

$env:JVM_ARGS=$OldJvmArgs

Expected: java.lang.OutOfMemoryError: Java heap space; target request count does not change. Preserve log/optional HPROF first. The repair is to remove the synthetic retention and restore normal JVM settings—not blindly assign a giant heap.

9. Lightweight JVM observation

jcmd -l
jcmd <PID> VM.command_line
jcmd <PID> GC.heap_info
jcmd <PID> Thread.print

A full heap dump is higher-impact and can contain sensitive data. Use it only when retained-object evidence is actually needed.

10. Failure E — simulated engine data/JAR mismatch

New-Item -ItemType Directory -Force `
  .\simulated-engines\engine-a\data, `
  .\simulated-engines\engine-a\extensions, `
  .\simulated-engines\engine-b\data, `
  .\simulated-engines\engine-b\extensions | Out-Null

"username`nsynthetic-user-01" | Set-Content .\simulated-engines\engine-a\data\users.csv
"SIMULATED-EXTENSION-1.0.0" | Set-Content .\simulated-engines\engine-a\extensions\p33-extension-1.0.0.jar

tools/check_engine_parity.py:

import argparse, hashlib, json
from pathlib import Path

def sha256(path):
    h = hashlib.sha256()
    with Path(path).open("rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""): h.update(chunk)
    return h.hexdigest()

def inventory(root):
    root = Path(root).resolve()
    result = {}
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = str(path.relative_to(root)).replace("\\", "/")
        result[rel] = {"bytes": path.stat().st_size, "sha256": sha256(path)}
    return result

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--engine", action="append", required=True, help="NAME=DIR")
    p.add_argument("--required", action="append", default=[])
    p.add_argument("--out", required=True)
    args = p.parse_args()
    engines = {}
    for spec in args.engine:
        name, directory = spec.split("=", 1)
        engines[name] = inventory(directory)
    names = sorted(engines)
    mismatches = []
    for required in args.required:
        for name in names:
            if required not in engines[name]:
                mismatches.append({"engine": name, "type": "missing_required_file", "path": required})
    all_paths = sorted({p for inv in engines.values() for p in inv})
    for path in all_paths:
        values = {name: engines[name].get(path) for name in names}
        hashes = {v["sha256"] for v in values.values() if v}
        if len(hashes) > 1:
            mismatches.append({"type": "hash_mismatch", "path": path, "engines": values})
    result = {"status": "PASS" if not mismatches else "FAIL", "engines": engines, "mismatches": mismatches, "note": "Simulation only: no RMI traffic is generated."}
    Path(args.out).write_text(json.dumps(result, indent=2), encoding="utf-8")
    print(json.dumps(result, indent=2))
    raise SystemExit(0 if not mismatches else 3)

if __name__ == "__main__":
    main()

Require both files on A/B. First run must FAIL for Engine B. Copy the exact files/hashes, rerun and require PASS. This simulates remote preflight only; no RMI traffic occurs.

11. Failure matrix

import argparse, json
from pathlib import Path
def main():
    p = argparse.ArgumentParser(); p.add_argument("--out", required=True); args = p.parse_args()
    rows = [
        {"case":"socket-refused","symptom":"java.net.ConnectException / Connection refused","layer":"network/socket endpoint","first_fix":"correct local port; do not retry","verification":"same sampler succeeds on 127.0.0.1:8033"},
        {"case":"dns-unresolved","symptom":"java.net.UnknownHostException for p33.invalid","layer":"name resolution","first_fix":"JMeter DNS Cache Manager static mapping","verification":"fresh process resolves p33.invalid -> 127.0.0.1"},
        {"case":"tls-untrusted","symptom":"SSLHandshakeException / PKIX path building failed","layer":"TLS trust/identity","first_fix":"scoped test truststore","verification":"same HTTPS sampler succeeds with verification enabled"},
        {"case":"heap-pressure","symptom":"java.lang.OutOfMemoryError: Java heap space","layer":"generator/JVM memory","first_fix":"remove synthetic retention; restore normal JVM settings","verification":"minimal/baseline plan runs normally"},
        {"case":"engine-parity","symptom":"missing CSV/JAR on one simulated engine","layer":"distributed engine filesystem/classpath","first_fix":"copy exact locked artifacts","verification":"parity report PASS before RMI"}
    ]
    Path(args.out).write_text(json.dumps({"cases":rows}, indent=2), encoding="utf-8")
    print(json.dumps({"cases":rows}, indent=2))
if __name__ == "__main__": main()

Write actual OS/JVM exception text into the evidence packet rather than assuming every platform prints identical wording.

12. Challenge

Engine B alone reports FileNotFoundException: data/users.csv; the SUT has no errors. Should response timeout be increased?

No. It is engine-local file state. Compare manifests, provision the exact file/JAR state, and re-run parity before any load.

Knowledge check

Official references and version notes

• Apache JMeter downloads — JMeter 5.6.3 and Java 8+.
• JMeter changes — Java 17+ recommended for 5.6.x.
• Getting Started — CLI, -l, -j, -L, JVM startup settings.
• Best Practices — CLI load execution, listener/memory cost, JSR223 Groovy.
• DNS Cache Manager — scoped HttpClient4 DNS caching/static mapping.
• Remote Testing — same JMeter versions, Java parity, data files, RMI SSL.
• JDK 17 jcmd — JVM diagnostics and command impact.
• JDK 17 memory troubleshooting — heap-dump/OOM evidence.
• JDK 17 networking properties — positive/negative DNS cache behavior.