Parallel Execution with Pabot, Sharding, and Resource Contention: Guided Hands-On Workflow

1. Scenario and preflight

The lab has four ordinary synthetic tests. Each test writes a marker containing its case ID, OS process ID, Pabot queue index, and Pabot executor pool ID, then waits briefly. The work is intentionally slow enough that parallel execution is measurable but harmless.

rf24-parallel-lab/
├── libraries/
│   └── lease_helper.py
├── resources/
│   └── parallel.resource
├── suites/
│   ├── alpha.robot
│   ├── beta.robot
│   ├── collision.robot
│   └── locked.robot
└── evidence/

python -m venv .venv
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# Bash/zsh: source .venv/bin/activate
python -m pip install "robotframework==7.4.2" "robotframework-pabot==5.2.2"
python -m robot --version
pabot --version

# Use a unique run id so evidence never aliases a previous attempt.
# PowerShell:
$env:RF24_RUN_ID = "run-001"
# Bash/zsh:
export RF24_RUN_ID="run-001"

The environment variable is not a secret. It is an operator-selected namespace used to create an isolated lab root under the system temp directory.

2. Shared resource: identify the execution without assuming Pabot

*** Settings ***
Library    OperatingSystem

*** Keywords ***
Resolve Execution Identity
    ${queue}=    Get Variable Value    \${PABOTQUEUEINDEX}    serial
    ${pool}=     Get Variable Value    \${PABOTEXECUTIONPOOLID}    serial
    ${pid}=      Evaluate    __import__("os").getpid()
    RETURN    ${queue}    ${pool}    ${pid}

Write Worker Marker
    [Arguments]    ${case_id}    ${delay}=0.35s
    ${queue}    ${pool}    ${pid}=    Resolve Execution Identity
    ${worker_dir}=    Set Variable    ${LAB_ROOT}${/}workers${/}${queue}
    Create Directory    ${worker_dir}
    ${marker}=    Set Variable    ${worker_dir}${/}${case_id}.txt
    Create File    ${marker}    case=${case_id}${\n}pid=${pid}${\n}queue=${queue}${\n}pool=${pool}${\n}
    Sleep    ${delay}
    File Should Exist    ${marker}

${LAB_ROOT} will be supplied from the command line. The keyword gets Pabot variables by escaped name, so the same suites also run serially. A queue-index directory prevents two scheduled execution items from writing the same marker directory.

3. Two deterministic suites

suites/alpha.robot

*** Settings ***
Resource    ../resources/parallel.resource

*** Test Cases ***
Alpha One
    Write Worker Marker    alpha-1

Alpha Two
    Write Worker Marker    alpha-2

suites/beta.robot

*** Settings ***
Resource    ../resources/parallel.resource

*** Test Cases ***
Beta One
    Write Worker Marker    beta-1

Beta Two
    Write Worker Marker    beta-2

Each test owns a different marker filename. There are no shared accounts, ports, mutable database rows, or production endpoints. This is the control workload.

4. Measure the serial baseline first

# PowerShell
$root = Join-Path $env:TEMP "rf24-parallel-lab-$env:RF24_RUN_ID"
Measure-Command { python -m robot --outputdir evidence/serial --variable "LAB_ROOT:$root" suites/alpha.robot suites/beta.robot }

# Bash/zsh
root="${TMPDIR:-/tmp}/rf24-parallel-lab-${RF24_RUN_ID}"
time python -m robot --outputdir evidence/serial --variable "LAB_ROOT:$root" suites/alpha.robot suites/beta.robot

Record wall-clock time, test count, result status, and the marker tree. A typical serial result should show one OS PID and queue=serial in every marker because one Robot process executed all tests.

5. Run controlled suite-level parallelism

# Same LAB_ROOT and same suites; only the scheduler changes.
pabot --processes 2 --outputdir evidence/pabot-suite   --variable "LAB_ROOT:${root}" suites/alpha.robot suites/beta.robot

# Windows PowerShell (one line is also fine):
pabot --processes 2 --outputdir evidence/pabot-suite `
  --variable "LAB_ROOT:$root" suites/alpha.robot suites/beta.robot

With two suite files, Pabot can schedule each suite into a separate execution. Inspect:

evidence/pabot-suite/
├── output.xml
├── log.html
├── report.html
└── pabot_results/
    ├── 0/ ... partial output/stdout/stderr ...
    ├── 1/ ... partial output/stdout/stderr ...
    └── pabot_manager.log

The exact queue indexes are scheduling metadata, not a promise that “queue 0 means alpha forever.” The evidence should show multiple OS PIDs when actual parallel processes were used.

6. Compare test-level splitting

pabot --testlevelsplit --processes 2 --outputdir evidence/pabot-test   --variable "LAB_ROOT:${root}" suites/alpha.robot suites/beta.robot

Now individual tests are scheduling units. Compare total time, number of pabot_results execution directories, and worker markers. For tiny tests, scheduler/process overhead can erase the expected speedup. This is why measurement comes before policy.

7. Intentionally broken case: one exclusive lease, two parallel tests

The helper below uses Python file mode x, which atomically fails if the file already exists. That models a singular external resource such as one test account or one fixed port without touching a real system.

from pathlib import Path


def acquire_exclusive_lease(path: str) -> str:
    p = Path(path)
    p.parent.mkdir(parents=True, exist_ok=True)
    # mode='x' is atomic: it fails if another execution already owns the lease.
    with p.open("x", encoding="utf-8") as stream:
        stream.write("owned\n")
    return str(p)


def release_exclusive_lease(path: str) -> None:
    p = Path(path)
    if p.exists():
        p.unlink()

*** Settings ***
Library    OperatingSystem
Library    ../libraries/lease_helper.py

*** Test Cases ***
Compete For Shared Lease A
    ${lease}=    Set Variable    ${LAB_ROOT}${/}lease${/}shared.lock
    Acquire Exclusive Lease    ${lease}
    TRY
        Sleep    0.8s
    FINALLY
        Release Exclusive Lease    ${lease}
    END

Compete For Shared Lease B
    ${lease}=    Set Variable    ${LAB_ROOT}${/}lease${/}shared.lock
    Acquire Exclusive Lease    ${lease}
    TRY
        Sleep    0.8s
    FINALLY
        Release Exclusive Lease    ${lease}
    END

# Serial should pass because each test releases the lease before the next starts.
python -m robot --outputdir evidence/collision-serial --variable "LAB_ROOT:${root}" suites/collision.robot

# Test-level parallelism should expose contention; preserve this failed evidence.
pabot --testlevelsplit --processes 2 --outputdir evidence/collision-fail   --variable "LAB_ROOT:${root}" suites/collision.robot

The important evidence is the first FileExistsError/keyword failure plus Pabot subprocess logs. Do not add retries or a larger sleep. The failure correctly reveals that two scheduling units believe they own the same resource.

8. Repair choice: isolate first; lock only if the fixture is truly singular

Preferred repair: make the resource name include a unique work-item identity, for example lease-${PABOTQUEUEINDEX}.lock, or allocate distinct synthetic accounts/ports. Then both tests can make progress simultaneously.

Alternative for a genuinely singular fixture: serialize only the critical section with PabotLib:

*** Settings ***
Library    OperatingSystem
Library    ../libraries/lease_helper.py
Library    pabot.PabotLib

*** Test Cases ***
Use Singular Fixture A
    Use Shared Lease Safely    locked-a

Use Singular Fixture B
    Use Shared Lease Safely    locked-b

*** Keywords ***
Use Shared Lease Safely
    [Arguments]    ${case_id}
    ${lease}=    Set Variable    ${LAB_ROOT}${/}lease${/}shared.lock
    Acquire Lock    rf24-demo-shared-lease
    TRY
        Acquire Exclusive Lease    ${lease}
        TRY
            Log    ${case_id} owns the singular synthetic fixture.
            Sleep    0.35s
        FINALLY
            Release Exclusive Lease    ${lease}
        END
    FINALLY
        Release Lock    rf24-demo-shared-lease
    END

pabot --testlevelsplit --processes 2 --outputdir evidence/locked-pass   --variable "LAB_ROOT:${root}" suites/locked.robot

The lock makes correctness explicit but removes parallelism inside that critical section. A large locked region can make a four-worker run behave almost serially.

9. Artifact rules

Pabot 5.2.2 copies configured artifact extensions from subprocess output directories and rewrites relative links in the final log. The default artifact extension is PNG. If a library writes text evidence inside each subprocess output directory, you can request TXT collection explicitly; the separate marker tree under ${TEMPDIR} in this lab is inspected directly and is not implicitly collected by Pabot:

pabot --processes 2 --artifacts png,txt --artifactsinsubfolders   --outputdir evidence/artifacts --variable "LAB_ROOT:${root}" suites/alpha.robot suites/beta.robot

Do not force every worker to a single shared screenshot or trace directory. Pabot’s own subprocess directories are designed to separate concurrent outputs; manually collapsing them defeats that isolation.

10. Challenge: choose the layer

You have four parallel tests but only two synthetic user accounts. Which layer owns the fix?

1. Increase --processes to 8.
2. Add a Robot suite variable that alternates usernames.
3. Use PabotLib value-set/resource allocation or redesign fixtures so a worker receives a unique account.
4. Add Sleep 5s before login.

The correct answer is 3. Account availability is an external resource-pool problem. It must be allocated, not guessed from timing or hidden in a suite variable.

Knowledge check

11. Summary and bridge

You now have a measured serial baseline, two parallel scheduling modes, worker/process evidence, a reproducible collision, and two distinct repairs. Lesson 3 turns these observations into design rules for process counts, sharding, ordering, PabotLib, setup duplication, and artifact strategy.

References and version anchors

• Robot Framework PyPI — current stable/pre-release and Python support
• Robot Framework 7.4.2 User Guide — execution, variables, result files, Rebot, and core semantics
• robotframework-pabot PyPI — stable 5.2.2 and prerelease status
• Pabot repository README — current CLI, PabotLib, ordering, global variables, sharding, and output/artifact behavior
• PabotLib keyword documentation — locks and shared resource distribution