Distributed Testing, Remote Engines, Network Topology, and Synchronization: Core Concepts and Mental Model

1. The practical problem: scaling the injector can silently change the experiment

A local plan is configured for 1,000 threads. An engineer adds six remote engines expecting about 167 threads per engine. JMeter actually starts the full 1,000-thread plan on each engine: 6,000 configured threads. If the data file contains only 1,000 unique users or the SUT was authorized for 1,000 users, the mistake is both a validity and a safety failure.

2. Mental model: controller sends commands; engines generate load

flowchart TD
C[Controller/client
JMX + remote command + global properties] -->|RMI over SSL| A[Engine A
full JMX]
C -->|RMI over SSL| B[Engine B
full JMX]
A -->|reads local shard A| DA[data/engine-a.csv]
B -->|reads local shard B| DB[data/engine-b.csv]
A -->|HTTP target traffic| T[Authorized SUT]
B -->|HTTP target traffic| T
A -->|remote sample results| C
B -->|remote sample results| C
C --> J[Central aggregate JTL + controller jmeter.log]
A --> LA[engine-a jmeter.log + generator state]
B --> LB[engine-b jmeter.log + generator state]
T --> E[target telemetry/events]

The controller holds the JMX and starts remote engines using secured RMI. It sends the same plan to both engines. Engine A and Engine B each instantiate their own Thread Groups, variables, protocol connections and local file handles; therefore each produces its own configured load. External CSV files are not part of the JMX transfer, so each engine reads an explicitly staged shard. Sample results flow back through the RMI result path and the controller can write one aggregate JTL. Engine logs/resource state remain engine-local. The target sees the sum of both engines.

3. Distributed load math

For a closed Thread Group with one sampler:

per_engine_threads = Thread Group threads
per_engine_samples = threads × loops × samplers_per_loop
total_threads = sum(per_engine_threads across engines)
total_samples = sum(per_engine_samples across engines)

Example:
2 engines × (2 threads × 3 loops × 1 sampler) = 12 total samples
6 engines × 1000 configured threads = 6000 configured threads

Do the total-load calculation before starting any server. If engines intentionally use different properties, calculate each term separately rather than multiplying by engine count.

4. Controller/client versus remote engine

5. Version and plugin identity is part of correctness

Current JMeter remote guidance requires exactly the same JMeter version on client and servers and discourages different Java versions. The same principle applies to third-party plugins, user JARs, Groovy dependencies and certificates: a remote engine can only instantiate classes that exist on that engine.

The mandatory lab uses JMeter 5.6.3, Java 17 and no third-party plugins, so the plugin manifest is intentionally empty. Production should compare file names/hashes for all non-core additions.

6. What is transferred—and what is not

The controller sends the test plan to servers automatically. It does not send CSV data files. A CSV Data Set relative path must resolve on each server in the correct server-side directory; absolute paths work only when that engine actually has the corresponding path.

Do not assume JMX transfer means “the whole project directory was deployed.” Data, plugins, user JARs, certificates and other files require an explicit deployment/manifest step.

7. Local, remote-global and engine-local properties

• -Jname=value changes a property in the process where the command runs—the controller.
• -Gname=value sends a JMeter property to all remote servers.
• Engine-specific properties such as engine.id or data.file should be set on each server before/startup, not with one global -G value.
• Thread-local variables still belong to each engine/thread; they are not shared across JVMs.

In the lab, both engines receive threads=2/loops=3 through -G, but engine-a starts with its own engine.id/data.file and engine-b starts with different values.

8. RMI network paths and SSL

JMeter remote mode uses RMI. A controller connects to each server registry (default 1099 unless changed), then also needs reverse connections for returned sample/thread events. Since JMeter 4.0 the default RMI transport is SSL.

For firewall-friendly private environments, explicitly set:

• one registry port per server instance;
• server.rmi.localport for the server engine;
• client.rmi.localport on the controller for up to three consecutive callback ports;
• the advertised RMI hostname/IP reachable only inside the authorized network;
• a controlled keystore/truststore shared according to your trust design.

9. Result transfer can become the bottleneck

Remote sample delivery consumes engine/controller/network resources. Current default sender mode is StrippedBatch, which strips successful response bodies and batches results. Standard sends each sample synchronously; Hold retains all samples in memory until the end and is discouraged. Asynch uses a queue that can eventually fill and block sampler threads.

A slow controller or return network can therefore reduce achieved load even when the SUT is healthy.

10. Distributed state checklist

11. Read-only inspection before starting remote servers

PowerShell:

& "$env:JMETER_HOME\bin\jmeter.bat" -v
java -version

Get-FileHash .\plans\distributed-local.jmx -Algorithm SHA256
Get-FileHash .\data\engine-a.csv,.\data\engine-b.csv -Algorithm SHA256

Get-Content .\config\remote-global.properties
Get-Content .\config\remote-results.properties

Get-NetTCPConnection -State Listen |
  Where-Object LocalPort -in 1099,1100,4000,4100,5000,5001,5002

curl.exe --fail --silent http://127.0.0.1:8025/health

Do not start remote engines if ports are unexpectedly occupied, versions differ, files/hashes are missing, or total-load math is not approved.

12. DevOps connection

Distributed load generation is infrastructure. It needs inventory, version pinning, file deployment, secrets/certificates, ports/firewalls, capacity monitoring, health checks, logs, immutable manifests and cleanup just like any other service fleet. You cannot safely scale the SUT workload until you can operate the generators themselves.

Knowledge check

Official references and version notes

• JMeter User Manual — Remote (Distributed) Testing — full-plan replication, same-version guidance, data-file behavior, RMI SSL, remote ports, CLI remote execution, and sample sender modes.
• JMeter Getting Started — -r, -R, -G, -X, CLI/server mode, property semantics.
• JMeter Properties Reference — remote hosts, controller/server RMI ports, SSL keystore/truststore settings, client failure policy and result properties.
• Component Reference — CSV Data Set Config — distributed CSV file placement and relative/absolute path behavior.
• JMeter Listeners / Result files — CSV result fields, sample variables and host attribution.
• Apache JMeter downloads — current stable release and Java requirement.