Parallel Execution with Pabot, Sharding, and Resource Contention: Diagnostics, Failure Modes, and Production Practices

1. Diagnostic sequence for a parallel failure

1. Preserve the failed output directory before any rerun.
2. Record python --version, python -m robot --version, and pabot --version.
3. Record exact selection, split mode, process count, shard index/total, argument files, and output directory.
4. Validate parse/import state with a dry run if the failure could be model-related.
5. Inspect the first failing keyword/test and its worker’s robot_stdout.out, robot_stderr.out, partial output.xml, and marker identities.
6. Identify the external state touched by that worker: file, port, account, row, queue, browser profile, service rate limit.
7. Inspect pabot_manager.log for overlap, process termination, or scheduling clues.
8. If CI is involved, multiply job/shard concurrency by local Pabot processes and compare with capacity.
9. Apply the least destructive correction: unique ownership, smaller process count, explicit pool, or smallest possible lock.
10. Rerun the smallest controlled slice into a new output directory.

2. Failure taxonomy

3. Hidden multiplication: Pabot × CI

A common production mistake is independently tuning each layer. A team decides that eight Pabot processes are safe on a laptop; later a CI matrix launches six jobs against the same shared test environment. The effective concurrency can become 48. If the environment supports ten clients, failures may look like random API or database instability.

4. Artifact collisions and missing evidence

Pabot separates its subprocess output directories and can collect configured artifact extensions. Problems reappear when test code overrides that design with one hard-coded shared path. The most reliable evidence layout includes run ID, shard ID if any, and execution/test identity.

evidence/
└── run-20260831-001/
    ├── shard-1/
    │   ├── output.xml
    │   └── pabot_results/...
    ├── shard-2/
    │   ├── output.xml
    │   └── pabot_results/...
    └── aggregate/
        ├── output.xml
        ├── log.html
        └── report.html

5. Lock deadlocks: correctness without liveness is still failure

Consider worker A acquiring account then database, while worker B acquires database then account. Both can wait forever. Prevent this by defining a single global lock order, avoiding nested locks where possible, and keeping critical sections short. A lock is not a reason to use giant process timeouts.

Bad:
Worker A: lock(account) -> lock(database)
Worker B: lock(database) -> lock(account)

Safer:
All workers: lock(account) -> lock(database)
Better when possible: allocate independent account + database namespace and use no lock.

6. Stale workers and process timeout

--processtimeout is a Pabot guardrail for a worker process that fails to finish. It is not the same as Robot test/keyword timeout and not the same as an external HTTP/database timeout. If it fires, inspect why the process is stuck before increasing it. Current Pabot manager logging records process kills/timeouts, which should be part of the evidence packet.

7. Intentionally broken example: a shared filename race

*** Settings ***
Library    OperatingSystem

*** Test Cases ***
Unsafe Parallel Artifact
    # BROKEN ON PURPOSE: all workers share one filename.
    Create Directory    ${LAB_ROOT}${/}shared
    Create File    ${LAB_ROOT}${/}shared${/}result.txt    ${TEST_NAME}
    Sleep    0.5s
    ${actual}=    Get File    ${LAB_ROOT}${/}shared${/}result.txt
    Should Be Equal    ${actual}    ${TEST_NAME}

Run two copies/tests at test-level parallelism. Both write the same path; whichever worker writes last can cause the other worker’s assertion to observe the wrong test name. The parser is fine, the Robot assertion is doing exactly what it should, and the operating system is not “random.” The ownership model is wrong.

Repair: derive an owned path

*** Settings ***
Library    OperatingSystem

*** Test Cases ***
Owned Parallel Artifact
    ${queue}=    Get Variable Value    \${PABOTQUEUEINDEX}    serial
    ${owned}=    Set Variable    ${LAB_ROOT}${/}workers${/}${queue}
    Create Directory    ${owned}
    Create File    ${owned}${/}${TEST_NAME}.txt    ${TEST_NAME}
    ${actual}=    Get File    ${owned}${/}${TEST_NAME}.txt
    Should Be Equal    ${actual}    ${TEST_NAME}

The repair changes the resource identity rather than timing. Because queue index is diagnostic scheduler identity, production code should prefer a domain/test-specific unique ID when the artifact must remain meaningful across reruns; for this disposable worker-evidence lab, queue ownership is appropriate.

8. Merged report mismatch

If the final test count differs from expectation, do not start by rerunning. Compare the selected inventory, .pabotsuitenames/ordering inputs, per-worker partial outputs, shard commands, and final Rebot inputs. A merge cannot create a test that was never selected or whose shard output was never collected.

9. Performance diagnosis by layer

Knowledge check

10. Summary and bridge

Parallel failures become tractable when every worker, resource, and result artifact has an identity. The repair hierarchy is: preserve evidence, classify the layer, isolate state, bound capacity, then use locks only when sharing is unavoidable. Lesson 5 combines these principles into a checkpoint lab and produces a reviewable concurrency budget.

References and version anchors

• Robot Framework PyPI — current stable/pre-release and Python support
• Robot Framework 7.4.2 User Guide — execution, variables, result files, Rebot, and core semantics
• robotframework-pabot PyPI — stable 5.2.2 and prerelease status
• Pabot repository README — current CLI, PabotLib, ordering, global variables, sharding, and output/artifact behavior
• PabotLib keyword documentation — locks and shared resource distribution