Assertions, Failure Semantics, Retries, and Flaky-Test Control: Configuration, Design Patterns, and Trade-Offs

1. Hard failure versus collecting related checks

The following table organizes the key choices and evidence for Hard failure versus collecting related checks. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

The mandatory path remains Python unittest, which uses hard assertions. A hand-written collector should stay small and read-only.

2. Test-level rerun versus step-level retry

The following table organizes the key choices and evidence for Test-level rerun versus step-level retry. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

3. Retry count is also feedback-time and capacity cost

If one browser attempt costs 40 seconds, “retry up to 3 times” can make a single unstable test consume roughly four attempts of browser/Grid capacity. Across a matrix, that can create queue pressure that looks like a second infrastructure problem. Model retry cost as part of capacity planning.

def worst_case_attempts(tests, browsers, initial_plus_retries):
    return tests * browsers * initial_plus_retries

print(worst_case_attempts(120, 3, 1))  # 360 clean attempts
print(worst_case_attempts(120, 3, 3))  # 1080 worst-case attempts

4. Quarantine is a workflow state with an exit condition

The following example makes the Quarantine is a workflow state with an exit condition behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

quarantine record
- test_id: checkout.approval
- reason: timing race after component rerender
- first_seen_revision: abc123
- evidence: ci/run-4812/attempt-1
- owner: checkout-quality
- tracking_issue: QA-184
- gate_behavior: non-blocking but always executed/reported
- review_date: 2026-09-04
- exit_condition: 50 materially equivalent clean first-pass executions after fix
- max_scope: this test only

The dates/counts above are illustrative, not universal Selenium rules. The important properties are visibility, ownership, bounded scope, a root-cause record, continued execution, and a measurable exit condition.

5. Stability thresholds versus delivery velocity

A release-blocking smoke suite may demand stricter first-pass stability than a broad nightly exploratory matrix. Do not hide this difference in runner flags. Name the policy: what blocks, what warns, how pass-after-retry is reported, and which window/revision/browser combination defines the metric.

6. Decision table: what should you do?

The following table organizes the key choices and evidence for Decision table: what should you do?. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

7. Keep configuration layers distinct

• WebDriver: browser session, capabilities, commands, waits.
• Test framework: assertions, setup/cleanup, result statuses.
• Retry/quarantine policy: orchestration/governance outside WebDriver semantics.
• AUT: business idempotency, terminal states, server errors.
• CI/Grid: attempt scheduling, capacity, artifact retention, infrastructure health.

A WebDriver capability cannot make an assertion “soft,” and a CI retry flag cannot fix a wrong locator or missing readiness condition.

8. Worked scenario: approval test intermittently times out

You have 30 controlled runs: 26 clean first-pass, 3 timeout→pass, 1 timeout→timeout. The first three flaky failures all show working at timeout and later become approved; the repeated failure shows error:dependency. Treating all four as “needs longer timeout” would merge two different causes.

1. Fix the timing contract for the three readiness races.
2. Track the dependency error separately as product/environment evidence.
3. Do not keep a blanket retry once the timing contract is corrected.
4. Re-run a controlled stability window and record first-pass results.

9. Security/privacy trade-offs in retry evidence

More attempts generate more screenshots, page sources, logs, downloads, and profiles. Artifact retention therefore expands privacy/security exposure. Sanitize synthetic fixtures, avoid secrets/PII, and configure retention by evidence value. Never solve flakiness by reusing a personal browser profile or dumping full cookies/environment variables.

10. Summary and bridge

Retry scope, assertion style, quarantine, and stability thresholds are governance decisions tied to causality, idempotency, feedback time, and capacity. Lesson 4 applies a disciplined diagnostic sequence to the failure modes that commonly defeat those policies.

Knowledge check

Official references and version notes

• Selenium 4.47 release notes — stable binding/Grid baseline pinned for this chapter.
• Selenium downloads — current stable Selenium client and Server/Grid versions.
• Waiting strategies — race conditions between test and application readiness are a primary source of flaky browser tests.
• Overview of Test Automation — browser tests should keep setup, actions, and evaluation compact and intentional.
• Avoid sharing state — isolate data and create a new WebDriver instance per test where practical.
• Fresh browser per test — begin from a clean, known browser state.
• Test independency — scenarios should not depend on another test's success or state.
• Python unittest — hard assertion semantics, subtests, setup/cleanup, and failure reporting used in the mandatory path.