Checkpoint Lab — Screenshots, Logs, Network Evidence, and Failure Diagnostics

1. Checkpoint scenario and safety boundary

The same fixture from Lesson 2 is sufficient. Everything binds to loopback. The only secret string is demo-secret-123, intentionally fake. Each injected case runs in a fresh WebDriver session and produces a unique evidence directory. The raw fixture server log exists only inside the disposable workspace and is removed when you clean up.

2. Setup and exact prerequisites

Prerequisites: Python 3.10+, Selenium Python 4.47.0, a locally installed supported Chromium-family browser, normal Selenium Manager resolution, and the Chapter 17 fixture server on 127.0.0.1:8781.

python -m venv .venv
# Activate the environment, then:
python -m pip install "selenium==4.47.0"
python make_fixture.py
python chapter17_fixture/serve.py

3. Preflight: prove the environment before injecting failure

The following example makes the Preflight: prove the environment before injecting failure behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

import selenium
from selenium import webdriver

probe = webdriver.Chrome()
try:
    probe.get("http://127.0.0.1:8781/?test_id=checkpoint-preflight")
    print("selenium", selenium.__version__)
    print("session", probe.session_id)
    print("browser", probe.capabilities.get("browserName"), probe.capabilities.get("browserVersion"))
    print("platform", probe.capabilities.get("platformName"))
    print("url", probe.current_url)
    print("log_types", probe.log_types)
finally:
    probe.quit()

If the page cannot be reached or the browser cannot start, stop. That is an environment/preflight failure, not one of the three injected application/client/network cases.

4. Predict before running

The following table organizes the key choices and evidence for Predict before running. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

The lab writes these predictions to predictions.json before browser execution. That prevents retrospective “prediction” after seeing the evidence.

5. Run the checkpoint evidence collector

Save as checkpoint_lab.py. It intentionally asserts the healthy oracle against each injected failure so the failure hook captures the original state. It does not retry. For the JavaScript case it polls browser logs with a bounded explicit wait; for all cases it captures before quit().

from pathlib import Path
from datetime import datetime, timezone
from urllib.parse import urlsplit, parse_qsl, urlencode, urlunsplit
import hashlib
import json
import re
import shutil
import selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

BASE_URL = "http://127.0.0.1:8781/"
FIXTURE_RAW = Path("chapter17_fixture/server-raw.jsonl")
OUT = Path("checkpoint-evidence")
SECRET = "demo-secret-123"


def redact_text(value):
    text = str(value).replace(SECRET, "<redacted>")
    text = re.sub(r"(?i)(token=)[^&\\s\"']+", r"\\1<redacted>", text)
    return text


def redact_url(url):
    parts = urlsplit(url)
    query = []
    for key, value in parse_qsl(parts.query, keep_blank_values=True):
        query.append((key, "<redacted>" if key.lower() in {"token", "secret", "key"} else value))
    return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), parts.fragment))


def new_driver():
    options = webdriver.ChromeOptions()
    options.set_capability("goog:loggingPrefs", {"browser": "ALL"})
    return webdriver.Chrome(options=options)


def safe_caps(caps):
    keys = ["browserName", "browserVersion", "platformName", "pageLoadStrategy"]
    return {k: caps.get(k) for k in keys if k in caps}


def pull_logs(driver):
    if "browser" not in set(driver.log_types):
        return []
    return [{k: redact_text(v) for k, v in row.items()} for row in driver.get_log("browser")]


def relevant_server_rows(test_id):
    if not FIXTURE_RAW.exists():
        return []
    rows = []
    for line in FIXTURE_RAW.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        raw = json.loads(line)
        if test_id in json.dumps(raw):
            rows.append(json.loads(redact_text(json.dumps(raw))))
    return rows


def write_json(path, value):
    path.write_text(json.dumps(value, indent=2), encoding="utf-8")


def capture(driver, case_id, first_logs=None):
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S.%fZ")
    root = OUT / case_id / stamp
    root.mkdir(parents=True, exist_ok=False)

    png = root / "viewport.png"
    driver.save_screenshot(str(png))
    status = driver.find_element(By.CSS_SELECTOR, '[data-testid="status"]')
    write_json(root / "dom.json", {
        "state": status.get_attribute("data-state"),
        "text": status.text,
        "correlation": driver.find_element(By.CSS_SELECTOR, '[data-testid="correlation"]').text,
    })
    write_json(root / "metadata.json", {
        "captured_at": stamp,
        "case_id": case_id,
        "selenium": selenium.__version__,
        "session_id": driver.session_id,
        "url": redact_url(driver.current_url),
        "title": driver.title,
        "capabilities": safe_caps(driver.capabilities),
        "log_types": list(driver.log_types),
    })
    logs = list(first_logs or [])
    logs.extend(pull_logs(driver))
    write_json(root / "browser-log.json", logs)
    write_json(root / "server-network.json", relevant_server_rows(case_id))

    resources = driver.execute_script("""
      return performance.getEntriesByType('resource').map(e => ({
        name: e.name, initiatorType: e.initiatorType, duration: e.duration
      }));
    """)
    for row in resources:
        row["name"] = redact_url(row["name"])
    write_json(root / "resource-timing.json", resources)

    return root


def write_manifest(root):
    hashes = {}
    for path in sorted(root.iterdir()):
        if path.is_file() and path.name != "manifest.sha256.json":
            hashes[path.name] = hashlib.sha256(path.read_bytes()).hexdigest()
    write_json(root / "manifest.sha256.json", hashes)


def wait_state(driver, expected):
    return WebDriverWait(driver, 3, poll_frequency=0.05).until(
        lambda d: d.find_element(By.CSS_SELECTOR, '[data-testid="status"]').get_attribute("data-state") == expected
    )


def run_case(case_id, button, expected_state, expected_business="ok"):
    driver = new_driver()
    collected = []
    try:
        driver.get(f"{BASE_URL}?test_id={case_id}")
        before = {
            "session_id": driver.session_id,
            "url": redact_url(driver.current_url),
            "state": driver.find_element(By.CSS_SELECTOR, '[data-testid="status"]').get_attribute("data-state"),
        }
        driver.find_element(By.CSS_SELECTOR, f'[data-testid="{button}"]').click()

        if button == "js-error":
            wait_state(driver, "js-triggered")
            if "browser" in set(driver.log_types):
                def saw_error(d):
                    collected.extend(pull_logs(d))
                    return any("synthetic-client-error" in row.get("message", "") for row in collected)
                WebDriverWait(driver, 3, poll_frequency=0.05).until(saw_error)
        else:
            wait_state(driver, expected_state)

        actual = driver.find_element(By.CSS_SELECTOR, '[data-testid="status"]').get_attribute("data-state")
        try:
            assert actual == expected_business, f"business assertion: expected={expected_business!r} actual={actual!r}"
        except AssertionError as exc:
            root = capture(driver, case_id, collected)
            write_json(root / "failure.json", {"type": type(exc).__name__, "message": str(exc), "before": before, "actual": actual})
            write_manifest(root)
            return root, actual
        raise AssertionError("Injected failure unexpectedly matched the healthy oracle")
    finally:
        driver.quit()


def assert_redacted(root):
    for path in root.rglob("*"):
        if path.is_file() and path.suffix in {".json", ".txt", ".md"}:
            assert SECRET not in path.read_text(encoding="utf-8"), f"secret leaked into {path}"


def main():
    if OUT.exists():
        shutil.rmtree(OUT)
    OUT.mkdir()
    if FIXTURE_RAW.exists():
        FIXTURE_RAW.unlink()

    predictions = {
        "app-error": "DOM state becomes app-error; browser session remains healthy; no transport drop is required.",
        "js-error": "DOM reaches js-triggered and browser console contains a JavaScript exception correlated by test_id.",
        "network-error": "DOM becomes network-error; server log records /api/drop with connection-dropped; browser reports fetch failure.",
    }
    write_json(OUT / "predictions.json", predictions)

    results = {}
    for args in [
        ("app-error", "app-error", "app-error"),
        ("js-error", "js-error", "js-triggered"),
        ("network-error", "network-error", "network-error"),
    ]:
        root, state = run_case(*args)
        results[args[0]] = {"bundle": str(root), "observed_state": state}
        assert_redacted(root)

    conclusions = [
        "# Diagnostic conclusions",
        "",
        "- app-error: DOM business state is explicitly app-error while the WebDriver session remains usable; classify first as an application/business failure.",
        "- js-error: browser-log evidence contains synthetic-client-error while the page records js-triggered; classify as client-side JavaScript failure, not a locator failure.",
        "- network-error: DOM records network-error and server-network evidence shows /api/drop with connection-dropped; classify as a synthetic transport/network boundary failure.",
        "",
        "Screenshots support each conclusion but are not the sole oracle.",
    ]
    (OUT / "conclusions.md").write_text("\n".join(conclusions), encoding="utf-8")
    write_json(OUT / "results.json", results)
    assert SECRET not in (OUT / "conclusions.md").read_text(encoding="utf-8")
    print(json.dumps(results, indent=2))


if __name__ == "__main__":
    main()

6. Execute the three cases

The following example makes the Execute the three cases behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

python checkpoint_lab.py
# Expected: JSON listing a distinct bundle for app-error, js-error, network-error.
# No case should be silently converted to green.

Each case uses a fresh browser session. That isolates log buffers, cookies, page state, and session identity. The evidence directory remains on the test runner after the browser is closed.

7. Required evidence packet

The following table organizes the key choices and evidence for Required evidence packet. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

8. Diagnostic conclusion 1: application error

Expected evidence: dom.json records app-error; the browser session metadata is valid; no dropped request is necessary to explain the business failure. A screenshot may show the message, but the primary classification is the explicit AUT state plus the failed healthy oracle. Do not misclassify a console warning as a JavaScript crash.

9. Diagnostic conclusion 2: client-side JavaScript error

Expected evidence: DOM records js-triggered and the configured browser log contains a correlated synthetic-client-error record. The lab waits for that log evidence without sleeping and preserves the returned records so a later get_log() call cannot erase the observation. This supports a client-runtime classification.

10. Diagnostic conclusion 3: network-like failure

Expected evidence: DOM records network-error; the server evidence shows /api/drop and connection-dropped; the browser may report a fetch failure. Together these establish that the request crossed the controlled network boundary and the connection was intentionally terminated. A screenshot alone could never prove that sequence.

11. Verify privacy controls

The checkpoint calls assert_redacted() on each case. Independently search the packet again if you want a second control. The raw server log may contain the fake token because it simulates an unsafe upstream source, but the persisted evidence packet must not.

from pathlib import Path

secret = "demo-secret-123"
leaks = []
for path in Path("checkpoint-evidence").rglob("*"):
    if path.is_file() and path.suffix in {".json", ".md", ".txt"}:
        if secret in path.read_text(encoding="utf-8"):
            leaks.append(path)
assert not leaks, leaks
print("checkpoint packet is redacted")

12. Verification checklist

• Three case directories exist and have distinct WebDriver session IDs.
• Each bundle was captured before its browser quit.
• Application case shows app-error.
• JavaScript case contains correlated client-error evidence when the configured browser exposes it.
• Network case contains a redacted server row for /api/drop with connection-dropped outcome.
• No textual evidence contains demo-secret-123.
• Manifest hashes exist for bundle files.
• No retry replaced attempt-1 evidence.
• No personal profile, real credential, public Grid, proxy bypass, or production endpoint was used.

13. Cleanup and rollback

The following example makes the Cleanup and rollback behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

# Stop chapter17_fixture/serve.py with Ctrl+C.
python - <<'PY'
from pathlib import Path
import shutil
for p in [Path("chapter17_fixture"), Path("checkpoint-evidence")]:
    if p.exists(): shutil.rmtree(p)
print("checkpoint fixture and artifacts removed")
PY

In CI, evidence retention would normally happen before workspace cleanup. Here the delete step is explicit so the learner controls whether to inspect or discard the synthetic packet.

14. What Chapter 17 adds to a production operating model

Your browser automation platform now has an evidence contract: capture before cleanup, correlate by test/attempt/session/time, preserve first failure, collect proportional layers, redact before persistence, hash/manifest artifacts, separate browser/AUT/network/Grid hypotheses, and define CI retention. That is the foundation for remote triage at scale.

15. Bridge to Chapter 18 — WebDriver BiDi

Chapter 17 intentionally treated BiDi as an evidence source rather than teaching its machinery. Chapter 18 will make the bidirectional connection itself the subject: event subscriptions, browsing contexts, logs, script events, network request/response/auth handling, handler lifecycle, browser support, and the relationship between WebDriver Classic, BiDi, and temporary CDP paths.

Knowledge check

Official references and version notes

• Selenium 4.47 release notes — stable binding/Grid baseline pinned for this chapter.
• Selenium downloads — current stable clients and Selenium Server/Grid.
• Selenium Python 4.47 API — supported Python versions and WebDriver API surface.
• Chromium WebDriver API — get_log, log_types, screenshot APIs, and Chromium-specific capabilities.
• WebDriver BiDi — W3C bidirectional event direction and enablement.
• WebDriver BiDi logging — current console-message and JavaScript-error handlers.
• WebDriver BiDi network — current network handler concepts; Chapter 18 covers these APIs in depth.
• Selenium Grid — remote session and node boundary.
• Grid getting started — local/free Grid execution and version alignment.
• Docker Selenium — official containerized Grid distribution; optional in this chapter.