Listeners, PreRunModifiers, Visitors, and Programmatic Execution: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

1. Preserve the first failing output.xml, log/report, console output, listener files, and modifier manifests.
2. Confirm Python/Robot versions and executable paths.
3. Confirm executed source, selection, output directory, extension import names and constructor arguments.
4. Validate parse/import graph.
5. Confirm listener API version and callback data model.
6. Inspect modifier before/after evidence.
7. Inspect output/result state separately from live callback state.
8. Repair the smallest layer and rerun into a new evidence directory.

2. Broken listener: v3 object treated as v2 dict

# INTENTIONALLY BROKEN: v3 data is treated like a v2 dictionary.
ROBOT_LISTENER_API_VERSION = 3

def end_test(name, attrs):
    print(attrs["status"])

With listener v3, the second argument is a result model object. Treating it as attrs["status"] is a contract mismatch. Repair the data model rather than blindly changing the API version:

ROBOT_LISTENER_API_VERSION = 3

def end_test(data, result):
    print(f"{data.id} {result.status}")

3. Broken modifier: silent test removal

from robot.api import SuiteVisitor

class SilentDrop(SuiteVisitor):
    # INTENTIONALLY BROKEN: removes everything and records no reason/evidence.
    def start_suite(self, suite):
        suite.tests = []

A run with no failed tests is meaningless if a modifier silently removed the intended tests. Repair with explicit criteria, before/after counts or names, policy version, and an expected minimum selected set when appropriate.

4. Broken programmatic gate: ignored return code

from robot import run
run("suites/extension_demo.robot", outputdir="evidence/bad")
print("release gate: allowed")  # Wrong: Robot status was discarded.

from robot import run
rc = run("suites/extension_demo.robot", outputdir="evidence/good")
if rc != 0:
    raise SystemExit(rc)
print("release gate: allowed")

The outermost CI/process status must reflect Robot's status unless the workflow explicitly models an expected diagnostic failure.

5. Private API coupling

Internal model classes may be importable and still be inappropriate external contracts. Prefer robot.api, listener interfaces, and robot.run/run_cli. Private dependencies should be isolated behind a compatibility adapter and version pin.

6. Listener logs recursively or leaks sensitive data

Do not serialize every keyword argument, variable, environment entry, request body, screenshot, or message merely because callbacks expose rich objects. Low-level message callbacks can multiply output and may create recursion/noise if the listener itself logs through observable channels. Collect the minimum telemetry needed.

7. Global mutable listener state breaks isolation

Module-level “current test” lists may appear harmless in serial runs but fail with multiple listener instances, nested suites, programmatic reuse, or future Pabot workers. Keep state instance-local and use run/worker-specific evidence paths. Chapter 24 formalizes parallel ownership.

8. Result mutation destroys audit evidence

evidence/failure-original/output.xml      # immutable first result
evidence/policy-derivative/output.xml     # transformed derivative
evidence/policy-derivative/change.json    # what/why/version

Never save a status-changing post-processor over the only original result. The derivative may be useful; it is not a replacement for the source evidence.

9. Measure the causal layer

10. Controlled broken example

Run the broken listener only against the synthetic candidate tests and write to evidence/broken. Preserve console/Test Execution Errors and any generated output. Repair the callback and rerun to evidence/repaired. Do not overwrite the original failure directory.

11. Production anti-patterns

• Listeners that implement hidden business workflows.
• Modifiers that silently drop tests.
• Broad except Exception: pass around extension failures.
• Blanket retries or giant timeouts for import/callback errors.
• Arbitrary PYTHONPATH changes instead of fixing packaging/imports.
• Ignoring Robot return codes.
• Global mutable listener state.
• Overwriting first-failure output.xml.
• Depending on private model attributes as a public contract.

12. Knowledge check

13. Summary and bridge

Extension failures are diagnosable when each lifecycle phase has separate evidence and ownership. Lesson 5 packages those rules into a checkpoint.

References and version anchors

• Listener interface — callback models and versions
• Errors and warnings during execution — extension error evidence
• run.py source — status propagation