Assertions, Failure Semantics, Retries, and Flaky-Test Control: Core Concepts and Mental Model

1. The practical problem: “green after retry” is not the same as green

Chapter 14 made the data, environment, and fixture lifecycle reproducible. That makes Chapter 15 possible: when the same controlled case fails, the team can investigate the failure rather than wonder whether the case silently changed.

A weak suite records only pass/fail. A production suite needs richer semantics. Was the assertion wrong? Did the AUT produce the wrong stable outcome? Did an explicit wait expire before the expected state? Did the browser session fail to start? Did Grid lose capacity? Did the first attempt fail and a retry pass? Those outcomes demand different owners and different release decisions.

2. Mental model: outcome → assertion → evidence → classification

flowchart TD
  S[Scenario + controlled inputs] --> A[Actions]
  A --> O[Observable outcome]
  O --> X[Assertion]
  X -->|pass first attempt| G[Clean green signal]
  X -->|fail| E[Preserve first-failure evidence]
  E --> C[Classify failing layer]
  C --> P[Product defect]
  C --> T[Test defect / synchronization]
  C --> I[Environment / infrastructure]
  C --> U[Unknown - investigate]
  C --> R[Optional bounded retry experiment]
  R -->|later pass| F[Pass-after-retry != clean pass]
  F --> M[Flake metric / governance]
  M --> Q[Fix, monitor, or explicit quarantine]

The assertion is the point where the observed outcome is compared with expected meaning. If it fails, preserve evidence before changing the browser, data, timeout, or environment. Classification follows evidence. A later retry can tell you whether the failure reproduces, but it must never delete the first observation.

3. Define the result vocabulary

The following table organizes the key choices and evidence for Define the result vocabulary. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

4. Product, test, and infrastructure are hypotheses—not exception-name aliases

An AssertionError can mean a product defect, a stale expected value, bad test data, or a test that asserted too early. A TimeoutException can mean poor synchronization, a backend failure that prevented readiness, or a genuinely slow environment. A WebDriverException can come from Selenium, a browser driver, Grid, a browser crash, or the runner host.

5. Assertions encode business meaning

The following example makes the Assertions encode business meaning behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

# Strong: stable business outcome with expected/observed context.
result = driver.find_element(By.CSS_SELECTOR, "[data-testid='order-state']").text
self.assertEqual(
    result,
    "approved",
    msg=f"expected approved order state; observed={result!r}",
)

# Weak: proves only that Selenium issued a click command without exception.
driver.find_element(By.CSS_SELECTOR, "[data-testid='approve']").click()
# no business assertion follows

A click succeeding is an interaction fact, not the business outcome. A meaningful assertion describes the user-visible/application-visible state that should follow. Where multiple read-only facts explain one outcome, record them together; do not continue destructive actions after a prerequisite has already failed.

6. Hard assertions and “assert-all” behavior in the chosen framework

Python unittest assertions are hard: the current test method stops at the failed assertion. It has no built-in soft/assert-all feature. If several related read-only observations are useful, collect them first and fail once with the full record. Do not use collection to keep mutating the AUT after a prerequisite is false.

def assert_summary(testcase, facts):
    failures = [f"{name}: expected={expected!r} observed={actual!r}"
                for name, expected, actual in facts if expected != actual]
    testcase.assertFalse(failures, "\n".join(failures))

facts = [
    ("status", "approved", status_text),
    ("banner", "Order approved", banner_text),
]
assert_summary(self, facts)

7. Retry creates another observation, not a replacement result

The following table organizes the key choices and evidence for Retry creates another observation, not a replacement result. Use it together with the surrounding prose so the rows serve as a comparison aid rather than standalone rules.

The exact release policy is organization-specific. The invariant is not: attempt history must remain visible, and later success must not destroy first-failure evidence.

8. Flake metrics are governance data

Useful measurements include first-pass rate, pass-after-retry count, failure-class distribution, quarantine age, and time-to-fix. A simple per-test flake rate can be reported as inconsistent outcomes divided by materially equivalent executions, but the denominator must be defined. Mixing different browsers, environments, code revisions, or data sets can make the metric meaningless.

test_id: checkout.approval
revision: abc123
browser: chrome 142.x
controlled runs: 50
first-pass failures: 4
pass-after-retry: 3
reproduced failures: 1
classified timing flakes: 3
open product defect: 1
quarantine: no

9. Read-only provenance before classifying a failure

The following example makes the Read-only provenance before classifying a failure behavior concrete. Read it with the stated assumptions, then compare its observable output or state changes with the explanation that follows.

import selenium

print({
    "selenium": selenium.__version__,
    "session_id": driver.session_id,
    "browser": driver.capabilities.get("browserName"),
    "browser_version": driver.capabilities.get("browserVersion"),
    "platform": driver.capabilities.get("platformName"),
    "url": driver.current_url,
    "title": driver.title,
})

Pair runtime provenance with Chapter 14 case/environment identity. In remote execution, add Grid session/node/queue evidence. Do not dump credentials, full environment variables, cookies, or personal profiles into artifacts.

10. DevOps connection: pipeline colors need semantics

A trustworthy delivery gate distinguishes clean green, first-attempt failure, retry-pass, quarantined known instability, and infrastructure incident. That lets release engineers decide whether a product build is unsafe, the test is untrustworthy, or the execution platform needs repair. Collapsing all of those into “green eventually” converts automation from evidence into noise.

11. Wrong approaches to reject

• Blanket retry every failed test until one attempt passes.
• Delete attempt-1 screenshot/logs after a successful rerun.
• Increase all waits because one scenario flakes.
• Catch Exception and return “pass” or “skipped”.
• Rerun with different data or environment and call it the same observation.
• Quarantine without an owner, root-cause issue, or review/expiry condition.

12. Summary and next step

Assertions define expected meaning; failure evidence preserves what actually happened; classification identifies the most plausible failing layer; retry adds another observation; and quarantine is a visible governance state, not a green paintbrush. Lesson 2 makes these ideas concrete with a controlled timing flake.

Knowledge check

Official references and version notes

• Selenium 4.47 release notes — stable binding/Grid baseline pinned for this chapter.
• Selenium downloads — current stable Selenium client and Server/Grid versions.
• Waiting strategies — race conditions between test and application readiness are a primary source of flaky browser tests.
• Overview of Test Automation — browser tests should keep setup, actions, and evaluation compact and intentional.
• Avoid sharing state — isolate data and create a new WebDriver instance per test where practical.
• Fresh browser per test — begin from a clean, known browser state.
• Test independency — scenarios should not depend on another test's success or state.
• Python unittest — hard assertion semantics, subtests, setup/cleanup, and failure reporting used in the mandatory path.