Matrix Strategies, Dynamic Matrices, Fail-Fast, and Parallel Test Design: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

Preserve run ID, attempt, exact SHA and the first failed/cancelled job list before editing YAML or rerunning. Then confirm the matrix source and expanded graph, evaluated tolerance/fail-fast/max-parallel policy, runner/image/toolchain, failing step, per-cell outputs/artifacts, and only then downstream aggregation.

2. Failure mode: matrix explosion

A new feature axis with eight values can turn an 8-cell matrix into 64 cells. Add another six-value axis and the requested 384 cells exceed GitHub’s current 256-job matrix limit. The fix is not “increase the limit.” Decide which combinations represent independent risk and move exhaustive combinatorics to a lower-cost local test layer if appropriate.

Before change: 2 OS × 2 runtimes × 2 modes = 8 cells
After change: 2 × 2 × 2 × 8 features = 64 cells
Add 6 database versions: 384 cells -> exceeds current 256 matrix-job limit

3. Failure mode: cancelled or skipped is counted as passed

With fail-fast enabled, a mandatory failure can cancel siblings. A simplistic aggregator that counts only explicit failure values and treats everything else as acceptable can publish a false support verdict. Cancelled and skipped cells are missing evidence, not successful tests.

4. Failure mode: a required cell is hidden behind continue-on-error

# BROKEN POLICY: all Windows cells are tolerated even though Windows is supported.
continue-on-error: ${{ matrix.os == 'windows-2025' }}

The workflow may continue despite a supported Windows failure. Repair the policy by carrying an explicit experimental field derived from the support contract and mapping only that field to continue-on-error. Preserve the original run because it proves the policy bug existed.

5. Failure mode: malformed dynamic JSON

# Broken planner output (single quotes are not valid JSON strings)
echo "versions=['3.12','3.13']" >> "$GITHUB_OUTPUT"

# Valid JSON array
echo 'versions=["3.12","3.13"]' >> "$GITHUB_OUTPUT"

If fromJSON cannot parse the planner output, the matrix cannot be evaluated correctly. Capture the exact output string. Validate generator output locally with a real JSON parser before dispatching a large run.

6. Failure mode: mutable toolchains invalidate compatibility evidence

A versioned runner label still receives image updates, and an unpinned setup action tag can move. Record action commit SHAs and installed tool versions. If Python 3.13 passed last week but fails today, compare the exact setup action SHA, runner ImageVersion and resolved Python patch release before concluding the application regressed.

7. Intentionally broken example and repair

jobs:
  plan:
    runs-on: ubuntu-24.04
    outputs:
      versions: ${{ steps.out.outputs.versions }}
    steps:
      - id: out
        run: echo "versions=['3.12','3.13']" >> "$GITHUB_OUTPUT" # invalid JSON
  test:
    needs: plan
    strategy:
      fail-fast: true
      matrix:
        python: ${{ fromJSON(needs.plan.outputs.versions) }}
    runs-on: ubuntu-24.04
    steps:
      - run: python --version

Interpretation: this is a matrix-generation failure, not a Python test failure and not runner capacity. Preserve the planner step log/output, repair only the JSON serialization, and dispatch a new run. Do not change runner labels, permissions and application code simultaneously.

8. Smallest equivalent rerun

If a single mandatory cell failed from a deterministic test bug, use the smallest rerun scope GitHub supports while preserving the original run/attempt and first-failure evidence. If you edit matrix-generation YAML, that is a new workflow revision and therefore a new experiment—not a faithful rerun of the old graph.

Knowledge check

Official references and version notes

• GitHub Docs — Running variations of jobs in a workflow — matrix expansion, contexts, include/exclude, dynamic outputs, failure handling and max-parallel.
• GitHub Docs — workflow syntax: strategy.matrix — current matrix limit and strategy semantics.
• GitHub Docs — expressions: fromJSON — converting job-output JSON into arrays/objects used by a later matrix.
• GitHub Docs — strategy context — fail-fast, job-index, job-total and max-parallel for the current generated job.
• GitHub Docs — matrix strategy with reusable workflows — matrix-driven reusable workflow calls and output caveats.
• actions/upload-artifact v7.0.1 and actions/download-artifact v8.0.1 — action releases pinned by full commit SHA in the checkpoint.