Checkpoint Lab — Listeners, PreRunModifiers, Visitors, and Programmatic Execution

1. Checkpoint mission

flowchart TD
A[Source: 3 synthetic tests] --> B[Model inventory: 3]
B --> C[Modifier: keep 2 candidates + tag]
C --> D[Execution]
D --> E[Listener JSONL]
D --> F[output.xml]
F --> G[ResultVisitor summary]
H[robot.run] --> D
I[Broken listener] --> J[Preserved extension error]
J --> K[Repaired run in new directory]

Build one extension pack containing a read-only listener, transparent modifier, source-model visitor, result visitor, and programmatic runner. Inject one listener contract error, preserve its evidence, repair it, and document exactly when each extension is justified.

2. Exact assumptions and preflight

python --version
robot --version
python -c "from robot.api import SuiteVisitor, ResultVisitor, ExecutionResult, TestSuiteBuilder; print('api imports ok')"
python -c "from robot.api.interfaces import ListenerV3; print('listener v3 base import ok')"

• Python 3.8+ and Robot Framework 7.4.2.
• No third-party Robot libraries.
• No network/browser/API/database/SSH/container/CI/production target.
• All artifacts stay under the owned lab directory.

3. Predict before executing

1. Source inspection will show three tests; the modifier will execute two candidates.
2. selected-by-rf22 will exist only in the runtime model/result, not the source file.
3. Listener events will exist only for executed tests.
4. The passing programmatic run will persist and return rc 0.
5. The broken listener will create extension error evidence; repaired evidence will be separate.

4. Build the extension pack

*** Settings ***
Documentation    Synthetic Chapter 22 suite. No production systems or secrets.

*** Test Cases ***
Candidate Alpha
    [Tags]    candidate    smoke
    Log    alpha executed
    Should Be Equal    alpha    alpha

Candidate Beta
    [Tags]    candidate    regression
    Log    beta executed
    Should Be Equal As Integers    2    2

Control Not Selected
    [Tags]    control
    Log    control executes only without the Chapter 22 filter

from __future__ import annotations
import json
from pathlib import Path
from robot.api.interfaces import ListenerV3

class AuditListener(ListenerV3):
    ROBOT_LISTENER_API_VERSION = 3

    def __init__(self, evidence_file: str):
        self.path = Path(evidence_file)
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def _write(self, item: dict):
        with self.path.open("a", encoding="utf-8") as stream:
            stream.write(json.dumps(item, sort_keys=True) + "\n")

    def start_suite(self, data, result):
        self._write({"event": "suite-start", "id": data.id, "name": data.name})

    def start_test(self, data, result):
        self._write({"event": "test-start", "id": data.id, "name": data.name})

    def end_test(self, data, result):
        self._write({"event": "test-end", "id": data.id, "name": data.name,
                     "status": result.status})

    def end_suite(self, data, result):
        self._write({"event": "suite-end", "id": data.id, "status": result.status})

from __future__ import annotations
import json
from pathlib import Path
from robot.api import SuiteVisitor

class SelectAndMark(SuiteVisitor):
    def __init__(self, required_tag="candidate", evidence_file="evidence/modifier.json"):
        self.required_tag = required_tag
        self.evidence_file = Path(evidence_file)

    def start_suite(self, suite):
        before = [test.name for test in suite.tests]
        kept = [test for test in suite.tests if self.required_tag in test.tags]
        for test in kept:
            test.tags.add("selected-by-rf22")
        suite.tests = kept
        self.evidence_file.parent.mkdir(parents=True, exist_ok=True)
        self.evidence_file.write_text(json.dumps({
            "suite": suite.name,
            "required_tag": self.required_tag,
            "before": before,
            "after": [test.name for test in kept],
            "added_tag": "selected-by-rf22"
        }, indent=2, sort_keys=True), encoding="utf-8")

    def end_suite(self, suite):
        suite.suites = [child for child in suite.suites if child.test_count > 0]

import json
import sys
from robot.api import SuiteVisitor, TestSuiteBuilder

class Inventory(SuiteVisitor):
    def __init__(self):
        self.suites = []
        self.tests = []

    def start_suite(self, suite):
        self.suites.append({"id": suite.id, "name": suite.name,
                            "test_count": len(suite.tests)})

    def visit_test(self, test):
        self.tests.append({"id": test.id, "name": test.name,
                           "tags": list(test.tags)})

def main(path):
    suite = TestSuiteBuilder().build(path)
    visitor = Inventory()
    suite.visit(visitor)
    print(json.dumps({"suites": visitor.suites, "tests": visitor.tests},
                     indent=2, sort_keys=True))
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1]))

import json
import sys
from pathlib import Path
from robot.api import ExecutionResult, ResultVisitor

class EvidenceVisitor(ResultVisitor):
    def __init__(self):
        self.tests = []

    def visit_test(self, test):
        self.tests.append({"id": test.id, "name": test.name,
                           "status": test.status})

def main(output_xml, destination):
    result = ExecutionResult(output_xml)
    visitor = EvidenceVisitor()
    result.visit(visitor)
    Path(destination).write_text(
        json.dumps({"tests": visitor.tests}, indent=2, sort_keys=True),
        encoding="utf-8")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1], sys.argv[2]))

from pathlib import Path
from robot import run
from extensions.audit_listener import AuditListener
from extensions.select_and_mark import SelectAndMark

def main():
    root = Path(__file__).resolve().parents[1]
    evidence = root / "evidence" / "programmatic"
    evidence.mkdir(parents=True, exist_ok=True)
    rc = run(
        str(root / "suites" / "extension_demo.robot"),
        outputdir=str(evidence),
        listener=AuditListener(str(evidence / "listener.jsonl")),
        prerunmodifier=SelectAndMark("candidate", str(evidence / "modifier.json")),
    )
    (evidence / "return-code.txt").write_text(str(rc), encoding="utf-8")
    return rc

if __name__ == "__main__":
    raise SystemExit(main())

5. Baseline inventory and source checksum

python tools/inspect_model.py suites/extension_demo.robot > evidence/source-model.json
python - <<'PY'
from pathlib import Path
import hashlib
p = Path('suites/extension_demo.robot')
print(hashlib.sha256(p.read_bytes()).hexdigest())
PY

Record the checksum in the evidence packet. It must remain unchanged after the modifier run.

6. Governed CLI run

robot --pythonpath . \
  --listener extensions.audit_listener.AuditListener:evidence/cli/listener.jsonl \
  --prerunmodifier extensions.select_and_mark.SelectAndMark:candidate:evidence/cli/modifier.json \
  --outputdir evidence/cli \
  suites/extension_demo.robot
python tools/result_report.py evidence/cli/output.xml evidence/cli/result-summary.json

Verify: before=3, after=2, both executed tests PASS, and the listener IDs correspond to those two result entries.

7. Programmatic run and independent rc verification

python tools/run_suite.py
printf 'shell-rc=%s\n' "$?"
cat evidence/programmatic/return-code.txt
python tools/result_report.py evidence/programmatic/output.xml evidence/programmatic/result-summary.json

The process rc and persisted Robot rc must agree. A mismatch is a runner defect, not a test failure.

8. Inject one extension error

# INTENTIONALLY BROKEN: v3 data is treated like a v2 dictionary.
ROBOT_LISTENER_API_VERSION = 3

def end_test(name, attrs):
    print(attrs["status"])

mkdir -p evidence/broken
robot --pythonpath . \
  --listener extensions.broken_listener \
  --outputdir evidence/broken \
  --include candidate \
  suites/extension_demo.robot > evidence/broken/console.txt 2>&1
printf '%s\n' "$?" > evidence/broken/process-rc.txt

Preserve this directory. Record where Robot reported the listener error and whether execution produced result artifacts. The objective is to diagnose the extension layer without erasing evidence.

9. Repair and rerun separately

ROBOT_LISTENER_API_VERSION = 3

def end_test(data, result):
    print(f"{data.id} {result.status}")

mkdir -p evidence/repaired
robot --pythonpath . \
  --listener extensions.fixed_listener \
  --outputdir evidence/repaired \
  --include candidate \
  suites/extension_demo.robot > evidence/repaired/console.txt 2>&1
printf '%s\n' "$?" > evidence/repaired/process-rc.txt

Compare broken and repaired artifacts. Do not use the repaired output to replace the original failure.

10. API/version manifest and operator runbook

# RF22 Extension Manifest
- Robot Framework: 7.4.2
- Python: <record exact version>
- Listener API: v3; AuditListener is read-only by policy
- Modifier: SelectAndMark(required_tag=candidate)
- Canonical result: evidence/*/output.xml
- Programmatic API: robot.run(), integer rc propagated
- Model/result APIs: imported from robot.api
- Private Robot APIs: none
- Network targets: none
- Secret-bearing fields collected: none

## Operator runbook
1. Record Python/Robot versions and source-model inventory.
2. Review modifier criterion and expected selected count.
3. Run into a fresh evidence directory.
4. Check Robot/process return code.
5. Review output.xml + listener + modifier evidence.
6. Preserve failures before repair or rerun.

11. Verification checklist

• Source model count = 3.
• Modifier before count = 3 and after count = 2.
• Source checksum is unchanged after execution.
• Runtime-selected tests are tagged selected-by-rf22.
• Listener captures only synthetic IDs/names/status.
• CLI and programmatic passing paths return 0.
• Programmatic runner propagates the returned status.
• Broken extension evidence remains intact.
• No private Robot API is used.
• No external/production system was contacted.

12. Cleanup/rollback

cd ..
rm -rf -- rf22-extension-lab

Set-Location ..
Remove-Item -LiteralPath .\rf22-extension-lab -Recurse -Force

Only remove the owned lab root after evidence review. A listener/modifier should never perform broad result-tree cleanup as a hidden side effect.

13. Knowledge check

14. Production operating model and bridge to Chapter 23

Chapter 22 adds a governed extension plane: public APIs, explicit lifecycle phases, transparent mutations, bounded observation, return-code propagation, and immutable original evidence. Chapter 23 applies the same ownership discipline to secrets, credentials, environment isolation, and sensitive artifacts.

References and version anchors

• Listener interface — checkpoint listener contract
• Pre-run modifiers — model mutation contract
• Robot Framework 7.4.2 API — SuiteVisitor, ResultVisitor, ExecutionResult
• run.py source — run/run_cli status semantics