Containers, Kubernetes, Ephemeral Injectors, and Infrastructure Automation: Diagnostics, Failure Modes, and Production Practices

1. Preserve first-failure evidence

2. Diagnostic sequence

flowchart TD
E[Preserve JTL + jmeter.log + image/mount/runtime + target evidence] --> V[Confirm JMeter / Java / plugin / Docker-kubectl-kind versions]
V --> C[Confirm image ID + JMX/data/properties/CLI + authorized target]
C --> S[Validate tree scope / resolved properties / injector-count load math]
S --> P[Inspect HTTP session / target DNS / mounted data state]
P --> G[Inspect JVM + container CPU/memory/PID/filesystem/network state]
G --> T[Inspect SUT events/telemetry]
T --> O[Inspect Job/Pod/container exit/OOM/restart/CI state]
O --> F[Least-destructive correction]
F --> R[Small controlled rerun]

3. Failure mode: assuming an unofficial image is Apache-maintained

An image named someone/jmeter can be useful, but the namespace/name does not establish Apache provenance. Inspect its source/Dockerfile, base image, JMeter download verification, plugin contents, update process and digest; run jmeter -v/java -version.

Repair: adopt the verified custom build or an organization-vetted community image pinned by digest. Document the maintainer accurately.

4. Failure mode: floating latest

Yesterday's latest and today's latest can resolve to different content. A regression then mixes application change with injector/runtime change.

Repair: use version-specific tags plus resolved digest/image ID in evidence; rebuild/upgrade deliberately and re-baseline.

5. Failure mode: baking a secret into the image

Broken Dockerfile:

ARG API_TOKEN
ENV API_TOKEN=$API_TOKEN
COPY secret.properties /opt/jmeter/bin/user.properties

Build args/environment/layers/history/caches can leak credential material. Never put a reusable secret into JMX, image layers or Git.

Repair: remove it from build context/Dockerfile, rotate the real credential if exposure occurred, rebuild cleanly, and inject the secret at runtime through an approved secret mechanism with redacted JTL/logs. Mandatory lab uses no credentials at all.

6. Failure mode: JTL written only inside the ephemeral container

Broken run writes -l /work/results.jtl with no artifact mount; CI then removes the container. Exit code is known, but raw samples are gone.

Repair: preserve the failed container if still present and docker cp the files out; then change future runs to /artifacts bind/PVC collection. Do not delete the only surviving container before copying evidence.

7. Intentionally broken example: wrong localhost

Run the JMeter container on p27-net but pass:

-Jtarget.host=127.0.0.1
-Jtarget.port=8027

Expected evidence:

• JTL samples fail with connection errors because nothing listens on 8027 inside the JMeter container;
• jmeter.log records the connection failure;
• p27-fixture target event count remains zero;
• Docker network inspection shows both containers attached, so the topology exists but target identity is wrong.

Repair: preserve failed JTL/log/inspect evidence, change only -Jtarget.host=p27-fixture, rerun the same 1×5 smoke before the full 5×20 test. Do not increase timeouts/retries.

8. Failure mode: unrestricted Kubernetes scaling/retries

A Job is changed from parallelism 2 to 20 while each Pod still runs 100 samples: configured total becomes 2,000 samples. If backoffLimit allows retries, failures can add more execution attempts.

Repair: explicit target authorization, total-load equation, conservative parallelism/completions, backoffLimit: 0 for this teaching workload, active deadline, resource limits and target abort/count gate.

9. Failure mode: non-root bind-mount permission denied

The JMeter image runs UID 10001. On a Linux host, /artifacts bind mount is owned by another user and denies writes. JMeter may fail before producing JTL.

Diagnosis:

• container exit/log error;
• docker inspect confirms mount is writable in Docker metadata;
• host filesystem ownership/mode denies UID 10001.

Repair one dedicated result directory: pre-create/chown it intentionally, or on local Linux run the container with the current host UID/GID when compatible. Do not switch the whole pattern to root or chmod 777 as a universal fix. Docker Desktop filesystem semantics differ; record the platform.

10. Failure mode: container OOM mistaken for JMeter/test failure

If memory limit is below effective Java/native requirement, Docker may report OOMKilled=true or Kubernetes Pod reason OOMKilled. JTL can be truncated. This is generator infrastructure saturation, not a target regression.

Repair from Chapter 26 evidence: reduce plan allocation/threads or assign measured memory/heap headroom. Keep -Xmx below the container limit.

11. Causal symptom table

12. Security-sensitive boundaries

Image build contexts/layers, runtime env vars, Docker socket, registry credentials, Kubernetes Secrets/service accounts, CI secrets, mounted files, target credentials, recorder certs/RMI keys and privileged/root settings are sensitive. Do not mount the host Docker socket into a JMeter container; it grants broad daemon/host control and is unnecessary here.

13. Troubleshooting shortcuts to reject

• Do not add blanket retries or arbitrary long sleeps.
• Do not assign giant heaps to outrun a memory quota.
• Do not mass-disable evidence/listeners without measuring their cost.
• Do not use global property hacks to hide namespace/config mistakes.
• Do not disable TLS/RMI verification.
• Do not test container fixes against production/public targets.
• Do not increase Pod parallelism while validity is unresolved.
• Do not delete failed containers/Pods/JTL/logs before evidence collection.

Knowledge check

Official references and version notes

• Apache JMeter downloads — JMeter 5.6.3, Java requirement, SHA-512/PGP integrity verification.
• Apache-published JMeter 5.6.3 SHA-512 — checksum used by the lab Dockerfile.
• Eclipse Temurin Docker Official Image — version-specific Java 17.0.20+8 JRE base used by the custom injector image.
• Docker bind mounts — read-only input mounts and writable host artifact mounts.
• Docker run reference — read-only root filesystem, CPU/memory limits, mounts, networks and exit-state inspection.
• Kubernetes Jobs — run-to-completion semantics, parallelism/completions, restartPolicy, backoff and active deadlines.
• kind Quick Start — local cluster lifecycle and loading locally built images.