Parallel Execution with Pabot, Sharding, and Resource Contention: Core Concepts and Mental Model

1. The practical problem: serial correctness does not imply parallel correctness

A Robot suite can be perfectly deterministic when one process owns all resources and still become flaky the moment two processes run simultaneously. The usual cause is not “Pabot is unstable.” It is that serial execution concealed an ownership assumption: two tests use the same filename, row, user account, TCP port, browser profile, approval queue, or cleanup directory.

Parallelism therefore starts as a state-design problem, not a command-line optimization. Pabot can schedule work faster, but it cannot infer which external resources are safe to share.

2. Read-only inspection before adding concurrency

python --version
python -m robot --version
pabot --version

# Read the suite inventory without mutating external state.
python -m robot --dryrun --outputdir evidence/dryrun suites

# Inspect available CPU only as one input to a capacity decision.
python -c "import os; print('cpu_count=', os.cpu_count())"

These checks establish interpreter ownership, Robot/Pabot versions, parse/import health, suite names, and a rough CPU ceiling. The dry run does not prove that runtime resources are isolated; it only proves that Robot can build and resolve the selected model.

3. Mental model: three concurrency layers, several state stores

flowchart TD
A[Suite / test inventory] --> B[CI shard selection]
B --> C[Pabot scheduler in shard]
C --> D1[Robot process 1]
C --> D2[Robot process 2]
D1 --> E1[Owned files / data / ports]
D2 --> E2[Owned files / data / ports]
D1 --> F1[Partial output.xml + stdout/stderr]
D2 --> F2[Partial output.xml + stdout/stderr]
F1 --> G[Local Pabot / Rebot aggregation]
F2 --> G
G --> H[Shard output.xml / log / report]
H --> I[Cross-shard aggregation / CI evidence]
J[PabotLib] -. narrow locks or value sets .-> D1
J -. narrow locks or value sets .-> D2

The first scheduler may be CI: two jobs receive disjoint shards. Inside each job, Pabot may launch multiple Robot processes. Each Robot process has its own Python interpreter state, Robot variables, library instances, and output directory. External systems do not automatically become independent; you must allocate them.

PabotLib is a coordination service. Its locks can serialize a truly singular fixture, and its value sets can distribute a finite pool of credentials or endpoints. Neither changes the fact that independent resources are preferable because locking reduces throughput and can create deadlock/liveness concerns.

4. State inventory: what must have an owner?

5. Pabot execution identity is diagnostic metadata

Pabot injects variables such as ${PABOTQUEUEINDEX}, ${PABOTEXECUTIONPOOLID}, ${PABOTNUMBEROFPROCESSES}, ${PABOTLIBURI}, and ${CALLER_ID}. Queue indexes start from zero and identify scheduled execution items; pool IDs identify executor slots. These are useful for worker-specific artifact paths and traces.

*** Keywords ***
Resolve Execution Identity
    ${queue}=    Get Variable Value    \${PABOTQUEUEINDEX}    serial
    ${pool}=     Get Variable Value    \${PABOTEXECUTIONPOOLID}    serial
    ${pid}=      Evaluate    __import__("os").getpid()
    Log    queue=${queue} pool=${pool} pid=${pid}
    RETURN    ${queue}    ${pool}    ${pid}

The escaped variable names are deliberate: a normal serial robot run does not define Pabot variables, so Get Variable Value must receive the variable name as data and provide a fallback. Do not store business records under a Pabot queue index and expect that identity to survive a different scheduling plan or rerun.

6. Suite-level versus test-level splitting

Choose the coarsest split that meets the feedback-time goal. More granular splitting is not automatically better; it increases scheduling overhead and makes hidden suite-level coupling more visible.

7. Result topology: preserve subprocess evidence before judging the merge

A current Pabot run writes normal final Robot artifacts and a pabot_results/ tree containing per-queue-index partial output.xml, robot_stdout.out, robot_stderr.out, robot_argfile.txt, and pabot_manager.log. The manager log is especially useful for reconstructing process start/finish and timeouts.

8. Capacity model: the smallest ceiling wins

A safe process count is constrained by more than cores. Suppose one worker needs about 180 MiB of available memory, the test host has 900 MiB reserved for this job, the local service permits four concurrent clients, and only three isolated synthetic accounts exist. The memory ceiling is 5, service ceiling 4, account ceiling 3; even on an eight-core host, the safe budget is at most 3.

safe_processes = min(
    CPU budget,
    memory budget / memory per worker,
    external-service concurrent capacity,
    number of isolated accounts/data slots,
    number of safe port/browser/profile slots
)

Measure, then increase gradually. The objective is throughput with deterministic evidence—not maximum process count.

9. DevOps connection: parallelism is an operating contract

Once parallel execution enters CI, the run contract must name process count, shard count, data-allocation strategy, per-worker artifact path, external capacity, lock use, merge strategy, and failure-artifact retention. This makes a parallel failure explainable instead of “rerun it; it is probably timing.”

Knowledge check

10. Summary and bridge

Pabot gives Robot Framework a scheduler and multiple worker processes; it does not grant extra capacity to the system under test. Correct parallelism comes from explicit resource ownership, bounded concurrency, per-process evidence, and disciplined aggregation. Lesson 2 turns this model into a local experiment with measurable serial and parallel runs.

References and version anchors

• Robot Framework PyPI — current stable/pre-release and Python support
• Robot Framework 7.4.2 User Guide — execution, variables, result files, Rebot, and core semantics
• robotframework-pabot PyPI — stable 5.2.2 and prerelease status
• Pabot repository README — current CLI, PabotLib, ordering, global variables, sharding, and output/artifact behavior
• PabotLib keyword documentation — locks and shared resource distribution