Checkpoint Lab — Cross-Browser, Responsive, Localization, and Compatibility Testing

1. Checkpoint scenario and production question

Your team is deciding whether a responsive/localized UI change can merge. The fast gate has a strict feedback budget, but one-browser coverage is insufficient. Build a matrix that protects two browser products, two engine families when available, a wide English case, and a compact German case. Preserve evidence that can later explain why a row differs.

Do not add Safari or real mobile unless the required infrastructure actually exists. Instead, document those rows as intentional extended coverage and explain why they are outside the mandatory local fast path.

2. Predict state changes before execution

The following table organizes the key choices and evidence for Predict state changes before execution. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

3. Setup and preflight

Reuse the exact site/index.html, serve.py, and matrix_run.py from Lesson 2. If starting from an empty directory, create them exactly as shown there.

mkdir selenium-ch16-checkpoint
cd selenium-ch16-checkpoint
python -m venv .venv
# PowerShell: .\.venv\Scripts\Activate.ps1
# POSIX: source .venv/bin/activate
python -m pip install "selenium==4.47.0"
mkdir site evidence
# Add site/index.html, serve.py, and matrix_run.py from Lesson 2.
python serve.py

Preflight assumptions: Python 3.10+, Selenium 4.47.0, Selenium Manager for normal driver resolution, at least two locally available supported browsers, and no production endpoint. The loopback fixture uses synthetic data only.

4. Write the matrix decision before running it

The following example makes the Write the matrix decision before running it behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

{
  "fast_ci": {
    "browser_selection": "prefer Firefox/Gecko plus one Chromium browser; otherwise first two available",
    "cases": [
      {"name": "wide-en", "viewport": [1280, 800], "app_locale": "en"},
      {"name": "compact-de", "viewport": [560, 760], "app_locale": "de"}
    ],
    "expected_business_state": "accepted"
  },
  "extended_not_mandatory_here": [
    "Safari/macOS on authorized Apple infrastructure",
    "real iOS/Android device/browser lanes",
    "enterprise-held historical browser versions",
    "additional RTL/locale/timezone combinations"
  ]
}

This file is a policy statement, not a Selenium capability. It explains the intended coverage and prevents the test loop itself from becoming the only documentation of support.

5. checkpoint.py — repeat the fast matrix and preserve the operating rationale

The following example makes the checkpoint.py — repeat the fast matrix and preserve the operating rationale behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

from pathlib import Path
import json
import shutil
import subprocess
import sys

EVIDENCE = Path("evidence")

def run_once(label):
    result = subprocess.run([sys.executable, "matrix_run.py"], text=True, capture_output=True)
    (EVIDENCE / f"{label}.stdout.txt").write_text(result.stdout, encoding="utf-8")
    (EVIDENCE / f"{label}.stderr.txt").write_text(result.stderr, encoding="utf-8")
    if result.returncode != 0:
        raise SystemExit(f"matrix run failed: {label}; inspect evidence/{label}.stderr.txt")
    rows = json.loads((EVIDENCE / "matrix.json").read_text(encoding="utf-8"))
    archive = EVIDENCE / label
    archive.mkdir(exist_ok=True)
    for row in rows:
        src = Path(row["screenshot"])
        dst = archive / src.name
        shutil.copy2(src, dst)
        row["archived_screenshot"] = str(dst)
    preflight = EVIDENCE / "preflight.json"
    if preflight.exists():
        shutil.copy2(preflight, archive / "preflight.json")
    (EVIDENCE / f"{label}.matrix.json").write_text(json.dumps(rows, indent=2), encoding="utf-8")
    return rows

def summarize(rows):
    browsers = sorted({(r["browserName"], r["browserVersion"], r["engine_family"]) for r in rows})
    hashes = {r["screenshot_sha256"] for r in rows}
    return {
        "rows": len(rows),
        "browsers": browsers,
        "all_business_states": sorted({r["business_state"] for r in rows}),
        "distinct_screenshot_hashes": len(hashes),
        "difference_explanation": (
            "Browser identity/version and rendered screenshot bytes differ across matrix rows, "
            "while the normalized business outcome remains 'accepted'. Pixel identity is not the contract."
        ),
        "fast_matrix_excludes": [
            "Safari/macOS unless an authorized Mac lane is available",
            "real iOS/Android device fidelity",
            "enterprise-held historical browser versions",
            "every locale/timezone combination",
        ],
        "exclusion_rationale": (
            "Fast CI protects the highest-risk desktop engines and responsive/localized paths within a bounded feedback budget. "
            "Platform-specific and long-tail rows belong in scheduled or dedicated infrastructure when product risk justifies them."
        ),
    }

def main():
    first = run_once("run-1")
    second = run_once("run-2")
    assert len(first) == len(second)
    summary = summarize(second)
    (EVIDENCE / "matrix-summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
    print(json.dumps(summary, indent=2))

if __name__ == "__main__":
    main()

6. Execute twice

The following example makes the Execute twice behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

# Terminal 1: server already running
# Terminal 2:
python checkpoint.py

python -m json.tool evidence/run-1.matrix.json
python -m json.tool evidence/run-2.matrix.json
python -m json.tool evidence/matrix-summary.json

Expected: both runs contain the same number of matrix rows; every row has business_state=accepted; returned browser identities/versions are recorded; screenshots exist; localized button copy differs by app locale; runtime browser language/timezone are evidence rather than hidden assumptions.

7. Explain one browser difference correctly

The summary deliberately states a safe, observable difference: browser identity/version and rendered screenshot bytes can differ while the normalized business outcome remains accepted. That is a real compatibility observation without inventing a functional defect.

If your screenshots show a visible difference—native control styling, font metrics, line wrapping, scrollbar treatment, or focus outline—record it descriptively. Do not fail the test merely because the pixels differ unless visual identity is an explicit product requirement with a defined oracle.

8. Required evidence packet

The following table organizes the key choices and evidence for Required evidence packet. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

9. Verification checklist

• At least two real local browsers were successfully created; no row is fabricated.
• When possible, the chosen pair includes Gecko plus Chromium engine families.
• Wide-English and compact-German both reach the same normalized accepted state.
• Visible translated text changes without changing the automation identity contract.
• Capabilities, window dimensions, app locale, browser language/timezone, screenshot paths, and hashes are recorded.
• No screenshot equality assertion is used.
• Safari/macOS, real-device, historical-version, and long-tail locale exclusions are explicit—not forgotten.
• Every session is quit and the fixture remains loopback-only.

10. Write the quarantine/coverage escalation rule

Compatibility exclusions must have promotion rules. Example: “If a production incident reproduces only in a currently excluded supported browser/platform, add that exact row to scheduled coverage immediately; promote it to fast CI if recurrence risk and user impact justify the feedback cost.” Likewise, if analytics or support contracts materially change, revise the matrix rather than keeping stale rows forever.

11. Cleanup and rollback

The following example makes the Cleanup and rollback behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

# Stop serve.py with Ctrl+C.
cd ..
# Preserve evidence only if required for review, then delete the disposable lab.
# PowerShell: Remove-Item -Recurse -Force selenium-ch16-checkpoint
# POSIX: rm -rf selenium-ch16-checkpoint

There are no real accounts, production sessions, or external services to revoke. If you adapt this pattern to CI, retention/deletion of screenshots and browser logs must follow your evidence/privacy policy.

12. What Chapter 16 adds to the operating model

The browser-automation platform now has a compatibility policy: select rows from user/support risk, keep matrix inputs explicit, preserve actual browser/platform provenance, distinguish responsive desktop from real mobile, separate app locale from browser locale/timezone, and tier coverage according to feedback/capacity economics. Cross-browser failures are now diagnosable observations instead of mysterious “works on my machine” arguments.

Chapter 17 builds on that evidence discipline by focusing on screenshots, logs, network evidence, and failure diagnostics—how to capture the right artifacts at the right time without leaking sensitive state or drowning CI in noise.

Knowledge check

Official references and version notes

• Selenium 4.47 release notes — stable binding/Grid baseline pinned for this chapter.
• Selenium downloads and supported platforms — current stable releases and browser/platform support links.
• Supported Browsers — browser-specific capability boundaries.
• Working with windows and tabs — standard window-size APIs used for responsive desktop rows.
• Browser Options — standard and browser-specific option/capability model.
• Selenium Grid — remote, parallel, cross-machine and cross-platform execution boundary.
• Getting started with Selenium Grid — local Grid prerequisites and Selenium Manager driver setup.
• WebDriver BiDi — current cross-browser event-stream direction; not required by the mandatory Chapter 16 matrix.
• Docker Selenium Grid — official container/Helm distribution entry point; containerized Grid is optional here.
• Safari specific functionality — Selenium-side SafariDriver setup and options.
• Apple: Enable WebDriver on macOS — Safari remote automation must be enabled on macOS.
• Apple: Testing with WebDriver in Safari — Apple-provided SafariDriver and WebDriver execution model.