Chapter 33Lesson 05~340 minutes

Checkpoint Lab — Troubleshooting Out-of-Memory, Socket, SSL, DNS, and Distributed Failures

The checkpoint is a recovery drill. Diagnose at least three failures from evidence rather than from the injection instructions. Recommended: socket refusal, DNS failure, and TLS trust. OOM and engine parity are optional additions. Never overwrite a first-failure directory.

CheckpointDecision tree3+ failuresLeast-invasive fixPost-fix baseline

Learning objectives

Complete the chapter checkpoint for Troubleshooting Out-of-Memory, Socket, SSL, DNS, and Distributed Failures as one reviewable, bounded experiment.
State the workload, predictions, acceptance criteria, authorization boundary, and abort conditions before execution.
Reconcile configured versus achieved work with JTL, jmeter.log, target evidence, and generator validity before making a conclusion.
Produce an evidence packet that records the exact inputs, results, diagnosis or gate outcome, and any material limitations.
Perform cleanup or rollback and explain how the checkpoint evidence hands off to the next chapter or operating practice.

1. Exact assumptions and ceilings

Item	Checkpoint
Runtime	Apache JMeter5.6.3 / Java17 / no third-party plugin.
HTTP target	127.0.0.1:8033 learner-owned fixture.
HTTPS target	localhost:8443 one-day self-signed SAN=localhost cert.
Baseline	2 threads×5 loops=10 HTTP samples; 150ms pacing.
Negative network cases	1 thread×1 loop each.
Fixture ceiling	40 requests before restart.
OOM optional	Separate JMeter process,1×1 Groovy,Xmx64MiB,no HTTP.
Distributed optional	Filesystem parity simulation only; no RMI.
Evidence	per-case JTL+jmeter.log, JVM/socket/DNS/TLS/engine diagnostics, target events, exact fix, post-fix comparison.

Abort: any non-loopback traffic, >40 target requests, sensitive data, heap-dump disk pressure, uncontrolled debug volume, RMI/public DNS/global trust/hosts changes, or generator saturation that prevents a valid recovery comparison.

2. Write the decision tree before injection

1. Did JMeter/JVM start?
   NO -> startup/classpath/JVM/OOM.
2. Did hostname resolve?
   NO -> DNS/resolver/cache/scope.
3. Did TCP connect?
   NO -> host/port/listener/firewall/socket/generator socket state.
4. Did TLS handshake succeed?
   NO -> certificate hostname/chain/trust/protocol.
5. Did request reach target?
   NO -> client/proxy/session/protocol.
6. Did target return expected status/business result?
   NO -> SUT/dependency/assertion/data.
7. Is failure remote-only?
   YES -> per-engine JMeter/Java/JAR/data/property/network parity.
8. After fix, does original configured AND achieved load match baseline?
   NO -> recovery changed the experiment.

3. Predictions

P1: socket case creates one failed ConnectException JTL and zero target events; changing only6553→8033 makes the one-sample repro pass.

P2: DNS case fails before target traffic; adding only scoped DNS Cache Manager mapping makes a fresh one-sample process pass.

P3: TLS case fails after TCP connect due to untrusted local cert; adding only the scoped truststore makes the same HTTPS request pass while verification remains enabled.

P4 optional: OOM affects only generator/JVM and target count remains unchanged.

4. Known-good baseline

Start HTTP fixture on127.0.0.1:8033 with max40/body4KiB.
Verify JMeter5.6.3/Java17 and plan/property hashes.
Run2×5 into results/baseline/.
Require10 JTL Work rows,10 target events, no rejection.
Preserve baseline JTL + matching jmeter.log.

5. Case 1 — socket refusal

Run one sample to port6553 into socket-first/. Before fixing, save the JTL/log, listener/Test-NetConnection evidence, and unchanged target count. Classify the layer. Then change only the port to8033 and rerun into socket-fixed/.

6. Case 2 — DNS

Run one sample against p33.invalid:8033 into dns-first/. Preserve UnknownHost evidence and target count. Add only the root/Test Plan DNS Cache Manager mapping p33.invalid→127.0.0.1, use a fresh JMeter process, and rerun into dns-fixed/. Record that OS hosts/global Java DNS settings were unchanged.

7. Case 3 — TLS trust

Start the local HTTPS fixture/cert. Run one sample without the truststore into tls-first/. Preserve JTL/log and cert subject/SAN/issuer/dates. Create the scoped PKCS12 truststore and rerun into tls-fixed/. Prove no Trust All or RMI SSL-disable property was used.

8. Optional OOM / engine parity

OOM: run the one-process64MiB plan, preserve log/HPROF and target-count proof, then restore normal JVM settings. Engine parity: run the local parity helper with Engine B missing files, preserve FAIL JSON, copy exact hashes, and require PASS. Neither optional case adds target traffic.

9. Complete the observed matrix

Case	Actual first symptom	Owning layer	Preserved evidence	One change	Post-fix proof
Socket
DNS
TLS
OOM optional
Engine parity optional

Use the actual exception text emitted by your OS/JVM; do not rewrite it to match the lesson.

10. Re-run the original workload

After repairs, restart/reset the HTTP fixture if needed and run the original2×5 baseline with normal INFO/default logging and normal JVM settings. Require10/10 configured/achieved target samples, matching JTL/target events and acceptable generator state. The one-sample fix is necessary but not sufficient.

11. Evidence packet

Artifact	Required
Failure matrix	Symptom→layer→least-invasive fix→verification.
Per-case JTL/log	First and fixed runs in separate directories.
JVM/resource	Java/JMeter version, jcmd/process state; HPROF only if OOM option used.
Socket	Listener/Test-NetConnection evidence for6553 and8033.
DNS	UnknownHost first run + scoped DNS Manager fix; no global hosts edit.
TLS	Cert SAN/issuer/dates, PKIX log, scoped truststore fixed log.
Engine parity	If optional: both manifests + PASS after exact-hash fix.
Target	Event/request counts proving which failures reached SUT.
Post-fix	Original2×5 configured/achieved baseline + generator validity.

12. Example conclusion form

Example form: “JMeter5.6.3/Java17 baseline produced10/10 loopback target samples. Socket refusal was isolated by ConnectException/listener state and fixed only by restoring port8033; DNS failure was isolated before target traffic and fixed only with a scoped DNS Cache Manager mapping; the untrusted localhost certificate produced a PKIX handshake failure and was fixed only by a process-scoped truststore while verification remained enabled. Every first-failure JTL/jmeter.log was retained separately. The final original2×5 run again achieved10 target requests under normal logging/JVM settings. This validates the troubleshooting procedure for the controlled local environment; it does not establish production network/TLS/DNS capacity or authorize remote/production testing.”

13. Cleanup / rollback

Stop HTTP/HTTPS fixtures and confirm ports8033/8443 close.
Restore/remove temporary JVM_ARGS after OOM option.
Delete disposable TLS private key/cert/truststore after evidence review.
Verify no OS hosts file, global Java trust/security property, or RMI SSL setting changed.
Retain first/fixed evidence per policy, then delete appropriately.

14. Production operating-model addition and Chapter34 bridge

Chapter33 adds a preserve-first troubleshooting contract: first-failure JTL/log/engine/target evidence, layer classification, exact runtime/input state, narrow diagnostics, minimal reproduction, one-factor repair, engine parity, security-preserving DNS/TLS behavior, configured-versus-achieved recovery verification, and diagnostic-overhead/validity notes are required before declaring an incident resolved.

Chapter34 moves to Performance Governance, Baselines, SLOs, Regression Budgets, and Reporting. Troubleshooting evidence becomes governance input: a failed gate needs a classified cause, an invalid-run reason, an approved baseline decision, and reporting that separates product regression from generator/network/environment failure.

Knowledge check

Which three recommended checkpoint failures are required?

Why re-run the original2×5 workload?

What proves DNS failed before target traffic?

A TLS test passes only after Trust All. Fixed?

One engine misses a CSV. Is SUT tuning relevant?

Next chapter

Performance Governance, Baselines, SLOs, Regression Budgets, and Reporting

Chapter34 converts valid measurements and classified failures into reviewable performance policy.

Official references and version notes

Apache JMeter downloads — JMeter 5.6.3 and Java 8+.
JMeter changes — Java 17+ recommended for 5.6.x.
Getting Started — CLI, -l, -j, -L, JVM startup settings.
Best Practices — CLI load execution, listener/memory cost, JSR223 Groovy.
DNS Cache Manager — scoped HttpClient4 DNS caching/static mapping.
Remote Testing — same JMeter versions, Java parity, data files, RMI SSL.
JDK 17 jcmd — JVM diagnostics and command impact.
JDK 17 memory troubleshooting — heap-dump/OOM evidence.
JDK 17 networking properties — positive/negative DNS cache behavior.

Version and compatibility note

Checked against current primary documentation on 2026-09-05. Mandatory runtime: Apache JMeter 5.6.3, Java 17, no third-party plugin. Meaningful runs use CLI with raw CSV JTL and a matching jmeter.log. Temporary logging uses category-specific -L...=DEBUG only for minimal reproductions. The bounded OOM case overrides JVM heap only for one disposable process. DNS Cache Manager is the scoped fallback for the p33.invalid exercise; Java 17 negative DNS cache defaults to 10 seconds. Remote JMeter runs the complete plan on each engine; data files are not automatically copied. RMI uses SSL by default and is not disabled in this chapter.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20

0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.