Listeners, PreRunModifiers, Visitors, and Programmatic Execution: Guided Hands-On Workflow

1. Disposable local scenario

rf22-extension-lab/
├─ suites/extension_demo.robot
├─ extensions/__init__.py
├─ extensions/audit_listener.py
├─ extensions/select_and_mark.py
├─ tools/inspect_model.py
├─ tools/result_report.py
├─ tools/run_suite.py
└─ evidence/

The lab touches only synthetic files inside its own project directory. No network endpoint, browser, API, database, SSH server, production service, credential, container runtime, or paid product is required.

2. Preflight and version pin

python -m venv .venv
. .venv/bin/activate
python -m pip install "robotframework==7.4.2"
python --version
robot --version
python -c "import robot; print(robot.__version__)"

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install "robotframework==7.4.2"
python --version
robot --version
python -c "import robot; print(robot.__version__)"

3. Build the suite and extension pack

*** Settings ***
Documentation    Synthetic Chapter 22 suite. No production systems or secrets.

*** Test Cases ***
Candidate Alpha
    [Tags]    candidate    smoke
    Log    alpha executed
    Should Be Equal    alpha    alpha

Candidate Beta
    [Tags]    candidate    regression
    Log    beta executed
    Should Be Equal As Integers    2    2

Control Not Selected
    [Tags]    control
    Log    control executes only without the Chapter 22 filter

from __future__ import annotations
import json
from pathlib import Path
from robot.api.interfaces import ListenerV3

class AuditListener(ListenerV3):
    ROBOT_LISTENER_API_VERSION = 3

    def __init__(self, evidence_file: str):
        self.path = Path(evidence_file)
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def _write(self, item: dict):
        with self.path.open("a", encoding="utf-8") as stream:
            stream.write(json.dumps(item, sort_keys=True) + "\n")

    def start_suite(self, data, result):
        self._write({"event": "suite-start", "id": data.id, "name": data.name})

    def start_test(self, data, result):
        self._write({"event": "test-start", "id": data.id, "name": data.name})

    def end_test(self, data, result):
        self._write({"event": "test-end", "id": data.id, "name": data.name,
                     "status": result.status})

    def end_suite(self, data, result):
        self._write({"event": "suite-end", "id": data.id, "status": result.status})

from __future__ import annotations
import json
from pathlib import Path
from robot.api import SuiteVisitor

class SelectAndMark(SuiteVisitor):
    def __init__(self, required_tag="candidate", evidence_file="evidence/modifier.json"):
        self.required_tag = required_tag
        self.evidence_file = Path(evidence_file)

    def start_suite(self, suite):
        before = [test.name for test in suite.tests]
        kept = [test for test in suite.tests if self.required_tag in test.tags]
        for test in kept:
            test.tags.add("selected-by-rf22")
        suite.tests = kept
        self.evidence_file.parent.mkdir(parents=True, exist_ok=True)
        self.evidence_file.write_text(json.dumps({
            "suite": suite.name,
            "required_tag": self.required_tag,
            "before": before,
            "after": [test.name for test in kept],
            "added_tag": "selected-by-rf22"
        }, indent=2, sort_keys=True), encoding="utf-8")

    def end_suite(self, suite):
        suite.suites = [child for child in suite.suites if child.test_count > 0]

import json
import sys
from robot.api import SuiteVisitor, TestSuiteBuilder

class Inventory(SuiteVisitor):
    def __init__(self):
        self.suites = []
        self.tests = []

    def start_suite(self, suite):
        self.suites.append({"id": suite.id, "name": suite.name,
                            "test_count": len(suite.tests)})

    def visit_test(self, test):
        self.tests.append({"id": test.id, "name": test.name,
                           "tags": list(test.tags)})

def main(path):
    suite = TestSuiteBuilder().build(path)
    visitor = Inventory()
    suite.visit(visitor)
    print(json.dumps({"suites": visitor.suites, "tests": visitor.tests},
                     indent=2, sort_keys=True))
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1]))

import json
import sys
from pathlib import Path
from robot.api import ExecutionResult, ResultVisitor

class EvidenceVisitor(ResultVisitor):
    def __init__(self):
        self.tests = []

    def visit_test(self, test):
        self.tests.append({"id": test.id, "name": test.name,
                           "status": test.status})

def main(output_xml, destination):
    result = ExecutionResult(output_xml)
    visitor = EvidenceVisitor()
    result.visit(visitor)
    Path(destination).write_text(
        json.dumps({"tests": visitor.tests}, indent=2, sort_keys=True),
        encoding="utf-8")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1], sys.argv[2]))

from pathlib import Path
from robot import run
from extensions.audit_listener import AuditListener
from extensions.select_and_mark import SelectAndMark

def main():
    root = Path(__file__).resolve().parents[1]
    evidence = root / "evidence" / "programmatic"
    evidence.mkdir(parents=True, exist_ok=True)
    rc = run(
        str(root / "suites" / "extension_demo.robot"),
        outputdir=str(evidence),
        listener=AuditListener(str(evidence / "listener.jsonl")),
        prerunmodifier=SelectAndMark("candidate", str(evidence / "modifier.json")),
    )
    (evidence / "return-code.txt").write_text(str(rc), encoding="utf-8")
    return rc

if __name__ == "__main__":
    raise SystemExit(main())

4. Inspect the model before mutation

python tools/inspect_model.py suites/extension_demo.robot > evidence/source-model.json
cat evidence/source-model.json

Expected: three tests; two tagged candidate; no selected-by-rf22 tag. This is the baseline against which the pre-run model mutation will be compared.

5. Run through the normal CLI

robot --pythonpath . \
  --listener extensions.audit_listener.AuditListener:evidence/cli/listener.jsonl \
  --prerunmodifier extensions.select_and_mark.SelectAndMark:candidate:evidence/cli/modifier.json \
  --outputdir evidence/cli \
  suites/extension_demo.robot

robot --pythonpath . `
  --listener extensions.audit_listener.AuditListener:evidence/cli/listener.jsonl `
  --prerunmodifier extensions.select_and_mark.SelectAndMark:candidate:evidence/cli/modifier.json `
  --outputdir evidence/cli `
  suites/extension_demo.robot

Expected: two candidate tests execute and pass. The source still contains the control test; the modifier manifest explains why it was not in the execution model.

6. Inspect independent evidence

cat evidence/cli/modifier.json
cat evidence/cli/listener.jsonl
python tools/result_report.py evidence/cli/output.xml evidence/cli/result-summary.json
cat evidence/cli/result-summary.json

7. Run the same suite programmatically

python tools/run_suite.py
rc=$?
echo "programmatic process rc=$rc"
cat evidence/programmatic/return-code.txt
python tools/result_report.py evidence/programmatic/output.xml evidence/programmatic/result-summary.json

python tools/run_suite.py
$rc = $LASTEXITCODE
"programmatic process rc=$rc"
Get-Content evidence/programmatic/return-code.txt
python tools/result_report.py evidence/programmatic/output.xml evidence/programmatic/result-summary.json

Expected: process rc 0, persisted rc 0, and the same two executed tests. The caller explicitly propagates Robot status rather than manufacturing a separate success status.

8. When run_cli is the better wrapper

from robot import run_cli

rc = run_cli([
    "--outputdir", "evidence/run-cli",
    "suites/extension_demo.robot",
], exit=False)
print(f"rc={rc}")
raise SystemExit(rc)

Use run_cli when your Python program intentionally forwards Robot's CLI grammar. If arguments come from untrusted input, allowlist them; an embedded runner is still an execution boundary.

9. Challenge: choose the correct layer

Requirement: add a policy-reviewed tag to candidate tests before --include policy-reviewed is applied. Choose the layer, implement the smallest change, and prove that source did not change while the runtime mutation is visible in evidence.

10. Cleanup

# From the parent directory, only after evidence review:
rm -rf -- rf22-extension-lab

# From the parent directory, only after evidence review:
Remove-Item -LiteralPath .\rf22-extension-lab -Recurse -Force

Never generalize a local cleanup into deletion of arbitrary result roots, home directories, or shared workspaces.

11. Knowledge check

12. Summary and bridge

You now have a complete extension workflow with before/after model evidence and equivalent CLI/programmatic execution. Lesson 3 turns these tools into explicit architecture choices.

References and version anchors

• SuiteVisitor API — visitor traversal and mutation
• robot.api public surface — ExecutionResult and ResultVisitor
• run/run_cli source contract — programmatic options and rc