Listeners, PreRunModifiers, Visitors, and Programmatic Execution: Core Concepts and Mental Model

1. Practical problem: extension hooks can become invisible control planes

Earlier chapters extended Robot Framework by adding reusable keywords and library boundaries. This chapter extends the framework lifecycle itself. You may need organization-wide observation, pre-execution policy transforms, post-run result inspection, or a Python application that invokes Robot directly. These are not ordinary keyword problems.

The risk is leverage: a listener can leak sensitive data, a modifier can silently remove tests, a result processor can rewrite evidence, and a host program can ignore a failed Robot return code. Production design therefore starts with phase, ownership, visibility, and auditability.

2. Mental model: one engine, four extension phases

flowchart TD
A[.robot source] --> B[Running model]
B --> C[Pre-run modifier / SuiteVisitor]
C --> D[Execution engine]
D --> E[Listener v3 events]
D --> F[output.xml / result model]
F --> G[ExecutionResult + ResultVisitor]
H[robot.run / run_cli] --> D
C --> I[Mutation evidence]
E --> J[Observation evidence]
G --> K[Derived report]

The source becomes an executable running model. A pre-run modifier can transform that model. During execution, listeners receive callbacks. Robot writes machine-readable output, which can be loaded as a result model and traversed using ResultVisitor. robot.run and run_cli are alternative entry points into the same engine.

3. Define the new objects before using them

4. Inspect first, before any mutation

python --version
robot --version
python -c "import robot; print(robot.__version__)"
python -c "from robot.api import SuiteVisitor, ResultVisitor, ExecutionResult, TestSuiteBuilder; print('public APIs import OK')"
robot --dryrun --outputdir evidence/preflight suites/extension_demo.robot

Also record the actual executable paths. In PowerShell use Get-Command python and Get-Command robot. A version string from the wrong environment is not useful provenance.

5. Listener v3: observe execution without editing the suite

from __future__ import annotations
import json
from pathlib import Path
from robot.api.interfaces import ListenerV3

class AuditListener(ListenerV3):
    ROBOT_LISTENER_API_VERSION = 3

    def __init__(self, evidence_file: str):
        self.path = Path(evidence_file)
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def _write(self, item: dict):
        with self.path.open("a", encoding="utf-8") as stream:
            stream.write(json.dumps(item, sort_keys=True) + "\n")

    def start_suite(self, data, result):
        self._write({"event": "suite-start", "id": data.id, "name": data.name})

    def start_test(self, data, result):
        self._write({"event": "test-start", "id": data.id, "name": data.name})

    def end_test(self, data, result):
        self._write({"event": "test-end", "id": data.id, "name": data.name,
                     "status": result.status})

    def end_suite(self, data, result):
        self._write({"event": "suite-end", "id": data.id, "status": result.status})

Version 3 callbacks receive Robot Framework model objects. The example deliberately records only synthetic IDs, names, and final status. It does not serialize keyword arguments, variable values, messages, environment variables, or credentials. That is a privacy boundary, not merely a coding style.

6. Pre-run modifier: make mutation explicit and inspectable

from __future__ import annotations
import json
from pathlib import Path
from robot.api import SuiteVisitor

class SelectAndMark(SuiteVisitor):
    def __init__(self, required_tag="candidate", evidence_file="evidence/modifier.json"):
        self.required_tag = required_tag
        self.evidence_file = Path(evidence_file)

    def start_suite(self, suite):
        before = [test.name for test in suite.tests]
        kept = [test for test in suite.tests if self.required_tag in test.tags]
        for test in kept:
            test.tags.add("selected-by-rf22")
        suite.tests = kept
        self.evidence_file.parent.mkdir(parents=True, exist_ok=True)
        self.evidence_file.write_text(json.dumps({
            "suite": suite.name,
            "required_tag": self.required_tag,
            "before": before,
            "after": [test.name for test in kept],
            "added_tag": "selected-by-rf22"
        }, indent=2, sort_keys=True), encoding="utf-8")

    def end_suite(self, suite):
        suite.suites = [child for child in suite.suites if child.test_count > 0]

Robot applies pre-run modifiers before ordinary test selection. The modifier adds a runtime tag and removes non-candidates from the executable model, but it also writes a before/after manifest. The .robot source remains unchanged.

7. SuiteVisitor and ResultVisitor are traversal APIs

import json
import sys
from robot.api import SuiteVisitor, TestSuiteBuilder

class Inventory(SuiteVisitor):
    def __init__(self):
        self.suites = []
        self.tests = []

    def start_suite(self, suite):
        self.suites.append({"id": suite.id, "name": suite.name,
                            "test_count": len(suite.tests)})

    def visit_test(self, test):
        self.tests.append({"id": test.id, "name": test.name,
                           "tags": list(test.tags)})

def main(path):
    suite = TestSuiteBuilder().build(path)
    visitor = Inventory()
    suite.visit(visitor)
    print(json.dumps({"suites": visitor.suites, "tests": visitor.tests},
                     indent=2, sort_keys=True))
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1]))

import json
import sys
from pathlib import Path
from robot.api import ExecutionResult, ResultVisitor

class EvidenceVisitor(ResultVisitor):
    def __init__(self):
        self.tests = []

    def visit_test(self, test):
        self.tests.append({"id": test.id, "name": test.name,
                           "status": test.status})

def main(output_xml, destination):
    result = ExecutionResult(output_xml)
    visitor = EvidenceVisitor()
    result.visit(visitor)
    Path(destination).write_text(
        json.dumps({"tests": visitor.tests}, indent=2, sort_keys=True),
        encoding="utf-8")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1], sys.argv[2]))

Visitors may inspect or mutate. That distinction belongs in the component contract. Import them through robot.api rather than implementation-private packages.

8. Programmatic execution: return code is part of the contract

from pathlib import Path
from robot import run
from extensions.audit_listener import AuditListener
from extensions.select_and_mark import SelectAndMark

def main():
    root = Path(__file__).resolve().parents[1]
    evidence = root / "evidence" / "programmatic"
    evidence.mkdir(parents=True, exist_ok=True)
    rc = run(
        str(root / "suites" / "extension_demo.robot"),
        outputdir=str(evidence),
        listener=AuditListener(str(evidence / "listener.jsonl")),
        prerunmodifier=SelectAndMark("candidate", str(evidence / "modifier.json")),
    )
    (evidence / "return-code.txt").write_text(str(rc), encoding="utf-8")
    return rc

if __name__ == "__main__":
    raise SystemExit(main())

9. DevOps connection: extension code is policy code

Cross-cutting listeners and modifiers can standardize observability and governance across many suites, but every hook adds another failure and trust boundary. Pin versions, version the extension set, minimize callbacks, make mutations visible, keep first-failure evidence immutable, and retain a reproducible path that can run without optional observation plugins.

10. Knowledge check

11. Summary and bridge

The lifecycle is now explicit: source → running model → pre-run transformation → execution/listener observation → output.xml → result traversal. Lesson 2 turns that model into a disposable local workflow.

References and version anchors

• Robot Framework 7.4.2 — Listener interface — listener versions and callbacks
• Robot Framework 7.4.2 — Programmatic modification — pre-run modifier ordering
• Robot Framework 7.4.2 API — public visitors and result APIs
• Robot Framework 7.4.2 run.py — run/run_cli return-code behavior