Checkpoint Lab — Distributed Testing, Remote Engines, Network Topology, and Synchronization

1. Exact assumptions and hard ceilings

2. Record the topology before execution

Controller:
  127.0.0.1
  callbacks 5000-5002
  JMX + remote-global.properties + aggregate JTL

Engine A:
  registry 1099
  server engine 4000
  engine.id=engine-a
  data.file=.../data/engine-a.csv

Engine B:
  registry 1100
  server engine 4100
  engine.id=engine-b
  data.file=.../data/engine-b.csv

Target:
  127.0.0.1:8025

3. Engine/controller inventory and file manifest

For every node record:

• jmeter -v output = 5.6.3;
• java -version = approved Java 17 build;
• JMX SHA-256;
• engine shard SHA-256;
• third-party lib/ext plugin/JAR list (expected none);
• RMI registry/server/callback ports and SSL keystore identity;
• UTC epoch/time and generator CPU/heap/network baseline.

4. Predictions before the valid run

P1 — workload: engine-a=6, engine-b=6, aggregate JTL=12, target events=12.

P2 — data: engine-a uses exactly a001-a006; engine-b uses exactly b001-b006; 12 unique accounts; no duplicates.

P3 — properties: -Gthreads/-Gloops/-Grun.id/-Gtarget.* are identical on both engines; engine.id/data.file stay distinct because they were set at each server startup.

P4 — failure run: if engine-b is deliberately restarted with engine-a.csv, its six requests should fail 409 wrong_shard while RMI/controller and engine-a remain healthy.

5. Target authorization/preflight

curl.exe --fail --silent http://127.0.0.1:8025/health
curl.exe --fail --silent -X POST http://127.0.0.1:8025/reset
curl.exe --fail --silent http://127.0.0.1:8025/stats

State must show zero requests/errors before each run.

6. Execute the valid two-engine run

Start the two engines exactly as in Lesson 2, then run:

& "$env:JMETER_HOME\bin\jmeter.bat" `
  -n `
  -t .\plans\distributed-local.jmx `
  -q .\config\remote-results.properties `
  -R127.0.0.1:1099,127.0.0.1:1100 `
  -G.\config\remote-global.properties `
  -l .\results\p25-check-valid\aggregate.jtl `
  -j .\results\p25-check-valid\controller-jmeter.log

If RMI is not feasible on the workstation, run the two independent workers described in Lesson 2 with run ID p25-check-valid, merge their JTLs, and clearly label the evidence simulation—not RMI validation.

7. Verify valid run independently

python tools/verify_distributed.py   results/p25-check-valid/aggregate.jtl   results/server-events.jsonl   p25-check-valid   6

Require PASS. Also inspect controller/engine logs for RMI/CSV/script errors and record generator/network headroom. A 12-row central file alone is insufficient if, for example, all 12 somehow came from one engine.

8. Deliberate configuration mismatch: wrong engine-b shard

1. Preserve the entire valid run.
2. Stop only engine-b.
3. Restart engine-b with engine.id=engine-b but data.file=.../engine-a.csv.
4. Reset target state.
5. Use new run ID p25-check-broken and run the same 2×3 plan.

Expected:

• 12 offered samples still possible;
• engine-a six successes;
• engine-b six 409 wrong_shard failures;
• aggregate JTL identifies ENGINE_ID=engine-b with a-prefixed ACCOUNT_ID;
• target JSONL reason=wrong_shard for engine-b;
• controller/RMI logs need not show transport failure, proving this is an engine data-state error.

9. Diagnose before repairing

Preserve:

• broken aggregate JTL;
• controller log;
• engine-a/engine-b logs;
• engine-b startup command/property manifest;
• target event log/stats;
• file hashes.

The causal chain is: engine-b startup property → wrong local CSV → ACCOUNT_ID values → HTTP query → target shard validation → 409. Do not disable assertions or change the target.

10. Least-destructive repair

1. Stop only engine-b.
2. Restore data.file=.../engine-b.csv.
3. Restart engine-b with same secured RMI ports/version.
4. Reset target state.
5. Run new ID p25-check-repaired.
6. Require verifier PASS (6+6=12, 12 unique accounts, zero failures).

11. Aggregate/result evidence requirements

12. Configured-versus-achieved validity statement

13. Network/security verification

• RMI SSL enabled; no server.rmi.ssl.disable=true.
• Only approved private/loopback registry/server/callback ports reachable.
• Controller callbacks 5000–5002 accounted for.
• No public port-forward/NAT exposure.
• Lab keystore treated as disposable and not reused as production PKI.
• Controller/engine clocks recorded/synchronized.

14. Cleanup / rollback

1. Stop both remote engines gracefully after evidence capture; -X is an optional CLI remote-exit mechanism when intentionally included in the run contract.
2. Stop the localhost fixture.
3. Preserve valid/broken/repaired JTL/log/manifests/events until review.
4. Delete the disposable RMI keystore only after no lab engine/controller needs it.
5. Remove temporary CSV shards/simulation files after the retention window.
6. No production/public target, real credential, remote public RMI service, recorder CA, managed cloud, paid CI, database/message system or OS/JVM global tuning was changed.

15. What Chapter 25 adds to the operating model

The production performance-testing operating model now has a distributed generator contract: controller/engine inventory, exact JMeter/Java/plugin versions, engine-local properties/files, total-load equation, secured/fixed RMI topology, remote-global property manifest, data-shard ownership/hashes, sender/result strategy, central/per-engine logs/results, clock synchronization, controller/engine resource evidence, missing-engine policy, target per-engine counts and cleanup are required before distributed capacity results are trusted.

Chapter 26 moves to Load Generator Sizing, JVM Tuning, OS Limits, and Network Capacity. It measures how many threads/sockets/bytes/requests each generator can sustain before the generator—not the SUT—becomes the limiting system.

Knowledge check

Official references and version notes

• JMeter User Manual — Remote (Distributed) Testing — full-plan replication, same-version guidance, data-file behavior, RMI SSL, remote ports, CLI remote execution, and sample sender modes.
• JMeter Getting Started — -r, -R, -G, -X, CLI/server mode, property semantics.
• JMeter Properties Reference — remote hosts, controller/server RMI ports, SSL keystore/truststore settings, client failure policy and result properties.
• Component Reference — CSV Data Set Config — distributed CSV file placement and relative/absolute path behavior.
• JMeter Listeners / Result files — CSV result fields, sample variables and host attribution.
• Apache JMeter downloads — current stable release and Java requirement.