Checkpoint Lab — Performance, Large Suites, Output Management, and Execution Optimization

1. Checkpoint scenario

Your team owns a growing Robot Framework regression estate. CI feedback is approaching its service-level budget and result artifacts are increasing. You are asked to improve the suite, but the acceptance criteria prohibit deleting assertions, sharing mutable state across tests, hiding first failures, or adding uncontrolled concurrency.

The checkpoint uses the disposable project from Lesson 2 or an equivalent local synthetic suite. You will create a baseline dossier, identify the dominant cost, apply Optimization A, re-measure, then apply Optimization B, re-measure, and finally define a budget. Never apply A and B before measuring A alone.

2. Evidence architecture

flowchart TD
S[Fixed source + pinned environment] --> B[Baseline: 3+ samples]
B --> R[Preserve raw output]
R --> A[Optimization A: one variable]
A --> VA[Verify selection/status/evidence]
VA --> C[Optimization B: one variable]
C --> VC[Verify selection/status/evidence]
VC --> D[Dossier + budgets + rollback triggers]
D --> N[Chapter 29 troubleshooting baseline]

3. Preflight

• Robot Framework 7.4.2 is installed in an isolated virtual environment.
• Pabot 5.2.2 is installed only if you use the parallel option.
• No 7.5 beta feature is required.
• The target is local/synthetic; no production service or credential is involved.
• Runner hardware/class and OS are recorded.
• The exact test selection, variables, data seed, and process count are fixed.
• results/ is empty or moved aside so old artifacts cannot contaminate size measurements.

python --version
python -m robot --version
pabot --version
python -m robot --dryrun -d results/preflight tests

4. Required predictions before execution

Write down at least two predictions before changing anything. For example:

5. Build the baseline dossier

Run the unoptimized command at least three times. Preserve the exact command and raw output from each sample or at minimum the representative median sample plus the benchmark ledger.

python bench.py base-1 -- python -m robot -d results/base-1 tests
python bench.py base-2 -- python -m robot -d results/base-2 tests
python bench.py base-3 -- python -m robot -d results/base-3 tests

Record: median wall time, minimum/maximum, selected test count, PASS/FAIL/SKIP counts, output.xml bytes, log.html bytes, report.html bytes, process count, retry count, Python/Robot/Pabot versions, OS/runner class, and any notable cold-start behavior.

6. Identify the top cost with evidence

Classify the dominant cost rather than selecting an optimization you already prefer.

7. Optimization A — one variable

Choose exactly one evidence-supported change. A safe default for the synthetic lab is deferring HTML rendering:

python bench.py opt-a-1 -- python -m robot -d results/opt-a-1 --log NONE --report NONE tests
python bench.py opt-a-2 -- python -m robot -d results/opt-a-2 --log NONE --report NONE tests
python bench.py opt-a-3 -- python -m robot -d results/opt-a-3 --log NONE --report NONE tests
python -m robot.rebot --outputdir results/opt-a-view results/opt-a-2/output.xml

Verify identical selected-test count/statuses and the same assertions. Confirm the raw output remains sufficient to generate the human view. Record Rebot time separately if report generation moves to a later pipeline stage.

8. Optimization B — one additional variable

Only after Optimization A has its own evidence, choose a second change. For an independent synthetic workload, a controlled Pabot comparison is appropriate:

python bench.py opt-b-p1 -- pabot --testlevelsplit --processes 1 -d results/opt-b-p1 --log NONE --report NONE tests
python bench.py opt-b-p2 -- pabot --testlevelsplit --processes 2 -d results/opt-b-p2 --log NONE --report NONE tests

If the suite uses real browser/API/database state, do not copy this blindly. First provide unique users/records/ports/files per worker and model target capacity. If two workers offer no meaningful benefit, keep one; the purpose is to optimize, not to maximize process count.

9. Functional-equivalence gate

10. Optional derived-output budget

If result volume is a measured problem, create a compact derivative from the preserved raw result and state exactly what detail is lost.

python -m robot.rebot --flattenkeywords FOR \
  --output results/derived-compact.xml \
  --log results/derived-compact-log.html \
  --report results/derived-compact-report.html \
  results/opt-a-2/output.xml

Do not delete results/opt-a-2/output.xml as part of the same experiment. Retention policy can expire raw artifacts later according to governance, but performance tuning should not erase its own proof.

11. Set budgets and regression thresholds

Derive values from your measured baseline; the table below is a template, not a universal standard.

A CI gate should allow normal variance. Avoid failing on a single noisy sample unless the workload is extremely deterministic; use repeated or rolling measurements where feasible.

12. Required evidence packet

• Manifest: date, source revision, Python/Robot/Pabot versions, OS/runner class, CPU/memory context if available.
• Commands: exact baseline and optimization commands.
• Selection: suite/test/task count, tags/variables/data seed.
• Timing: all samples, median, range, and cold/warm note.
• Artifacts: raw output/log/report sizes, derived sizes, Rebot timing if separated.
• Concurrency: process count and external capacity assumptions.
• Correctness: statuses, assertion equivalence, exit status, isolation verification.
• Decision: accepted optimizations, rejected experiments, budget, threshold, and rollback trigger.

13. Verification and cleanup

1. Open at least one baseline log/report and one optimized/derived view.
2. Confirm the same tests and statuses.
3. Confirm no output paths escaped the disposable lab.
4. Archive the dossier if desired.
5. Delete only generated results/ data after the evidence is reviewed.
6. Deactivate and optionally remove the lab virtual environment.

python -c "from pathlib import Path; assert Path('tests').is_dir(); assert Path('results').is_dir(); print('cleanup guard OK')"

Knowledge check

Summary and bridge to Chapter 29

Chapter 28 turns speed and result size into governed engineering properties. You can now establish a reproducible baseline, find the dominant cost, optimize one variable at a time, preserve correctness/evidence, model concurrency capacity, and enforce budgets. Chapter 29 uses this disciplined baseline to troubleshoot imports, scope, timing, encoding, and flaky automation—especially the regressions that appear only under real suite scale.

Further reading

• Robot Framework User Guide — execution, output files, log levels, Rebot, keyword removal/flattening, and library scope.
• Robot Framework releases and Robot Framework on PyPI — verify the stable/pre-release boundary before reproducing measurements.
• Pabot documentation and Pabot releases — process count, suite/test splitting, chunking, ordering, PabotLib, and output handling.
• Robot Framework documentation portal — current ecosystem guidance and examples.

Version-sensitive commands in this chapter were authored against the stable course baseline recorded above. Re-check current primary documentation when upgrading Robot Framework, Pabot, Python, external libraries, or CI runners.