Data-Driven Testing, Templates, Embedded Arguments, and Variants: Diagnostics, Failure Modes, and Production Practices

1. Diagnostic sequence: isolate the failing layer

1. Preserve output.xml, log.html, report.html, console output, and the exact variant source.
2. Confirm Python/Robot versions and the executed suite path.
3. Confirm test selection and the test/template name that actually ran.
4. Run --dryrun to separate parse/import/argument-contract errors from runtime behavior.
5. Inspect the failing template call and concrete argument values.
6. Inspect variable scope and any library/external-system state the template touched.
7. Only then inspect timing/parallel/CI issues if relevant.
8. Apply the smallest correction and rerun the smallest named test/controlled grouped slice.

2. Failure map

3. Failure mode: Cartesian expansion without causal coverage

A matrix that combines every browser × environment × locale × role × input boundary can grow faster than the defects it finds. Start by marking which dimensions influence the actual code path. Pairwise or risk-based representative selection can be appropriate, but the rationale must be reviewable and deterministic.

Do not solve a slow matrix by blindly adding Pabot workers. Parallelism cannot repair redundant coverage and may create shared-state contention.

4. Failure mode: one template covers unrelated workflows

If rows require columns such as mode, action, expected_error, cleanup_strategy, and many blanks to steer different paths, the template has become a hidden dispatcher. Split it into purpose-specific keywords/tests. The report should tell the operator which contract failed without reconstructing a mini-program from a row.

5. Failure mode: vague row/test names

Names such as case_17 or row_42 are poor release evidence. Prefer names that encode the risk class, for example Minimum valid length or Dash character rejected. Keep large raw payloads in arguments, not in test names.

6. Failure mode: hidden type conversion changes the meaning of a row

*** Settings ***
Test Template    Compare Numbers

*** Test Cases ***     LEFT    RIGHT
Looks numeric           03      3

*** Keywords ***
Compare Numbers
    [Arguments]    ${left: int}    ${right: int}
    Should Be Equal    ${left}    ${right}

This test passes because both values are converted to integers. That may be correct for a numeric rule but wrong for a lexical identifier rule where 03 and 3 are distinct. Type annotations are a contract, not decoration.

7. Failure mode: “expected failure” passes for the wrong reason

An unsafe anti-pattern is to wrap an entire variant in a broad “any error is expected” check. Then a typo, missing keyword, parser-adjacent runtime error, or wrong dependency can satisfy the row even though the intended validation was never reached.

# INTENTIONALLY UNSAFE — diagnostic anti-pattern, do not copy as a default.
Invalid case
    ${status}=    Run Keyword And Return Status    Validate Complex Workflow    bad-input
    Should Be Equal    ${status}    ${False}

The Boolean only proves that something failed. Repair the design by giving the validator an explicit classification/result contract, or—when failure itself is the contract—assert a narrow, intentional error at the smallest keyword boundary. Chapter 13 covers expected-failure patterns in depth.

8. Failure mode: template failure has no input context

If the assertion message says only ACCEPT != REJECT, operators must search the log to find the row. Include non-sensitive context in the assertion message or embedded call name:

Should Be Equal    ${actual}    ${expected}
...    msg=classification mismatch for candidate=${candidate}

Do not add secrets or entire personal records merely to improve diagnostics.

9. Failure mode: generated data cannot be reproduced

Random values without a recorded seed/source make a CI failure impossible to recreate. If generation is necessary, store the seed, generator version, generation parameters, and failing concrete value with the result artifacts. For core business boundaries, prefer explicit deterministic rows.

10. Failure mode: secrets in data tables

A template row is not a credential channel. Source control and result files may expose it. Use fake values in this course. In production, inject secrets from an approved boundary and keep the test row limited to a symbolic case such as valid credential/expired credential rather than the credential itself.

11. Intentionally broken diagnostic exercise

*** Settings ***
Test Template    Identifier Should Have Classification

*** Test Cases ***          CANDIDATE    EXPECTED
case_1                       abc          ACCEPT
case_2                       ab           ACCEPT
case_3                       abc          ACCEPT
case_4                       abc          ACCEPT

*** Keywords ***
Identifier Should Have Classification
    [Arguments]    ${candidate}    ${expected}
    ${actual}=    Classify Identifier    ${candidate}
    Should Be Equal    ${actual}    ${expected}

Three problems exist: the failing row name is vague, ab has the wrong expectation, and three rows duplicate the same case. Preserve the failure first. Then rename case_2 to Too short rejected, repair the expected value to REJECT, and remove redundant duplicates with a recorded rationale.

12. Troubleshooting shortcuts to reject

• Blanket retries around a failing matrix.
• Giant sleeps/timeouts to make generated cases “stable.”
• Broad EXCEPT or status conversion around all template logic.
• Arbitrary PYTHONPATH changes instead of fixing imports.
• Global variables to share mutable row state.
• Deleting the original failing result directory before comparing the repair.
• Adding workers before measuring whether rows are actually independent.
• Copying production customer data into a “realistic” test table.

13. Knowledge check

14. Summary and next step

Data-driven failures are diagnosable when test identity, variant values, input contracts, and result artifacts remain explicit. Preserve the first failure, reduce redundant dimensions, reject false-green error suppression, and keep sensitive data out of rows. Lesson 5 applies these rules in a compact checkpoint matrix and a deliberate matrix-reduction exercise.

Further reading

• Robot Framework 7.4.2 User Guide — Test templates — basic templates, multiple iterations, embedded arguments, FOR, and IF/ELSE.
• Robot Framework 7.4.2 User Guide — Different test case styles — keyword-driven, data-driven, and BDD-style presentation.
• Robot Framework 7.4.2 User Guide — Embedded arguments — matching rules, ambiguity, and custom regular expressions.
• Robot Framework 7.4.2 User Guide — User-keyword argument conversion — typed user keyword arguments available from 7.3.
• Robot Framework 7.4.2 on PyPI — pinned stable package metadata and Python requirement.
• Robot Framework releases — re-check stable and pre-release status when updating the course.