Continuous Integration Patterns for Build, Test, Lint, and Coverage: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

1. Preserve identity: run ID, attempt, event, ref, GITHUB_SHA, workflow revision and first-failure logs.
2. Confirm checkout: compare git rev-parse HEAD with the SHA the workflow claims to test.
3. Confirm evaluated configuration: job graph, if conditions, permissions, matrix values and cache key.
4. Confirm runner/toolchain: runner OS/image metadata, Python/tool versions and dependency manifest.
5. Inspect the first failing command: exit code, diagnostics, test/JUnit and coverage report.
6. Inspect retained services: cache-hit semantics, artifact names/digests and branch required-check state.
7. Apply the least destructive repair and rerun only the smallest equivalent scope that proves it.

2. Failure mode: CI builds a different SHA than the event SHA

This failure often appears when a workflow checks out a mutable branch name after GitHub created the run. The run claims one event SHA while build steps validate whatever commit the branch points to later.

# Broken for revision-bound CI: mutable ref can advance.
- uses: actions/checkout@FULL_SHA
  with:
    ref: main

# Safer for ordinary event-bound CI: use the event-selected ref/SHA.
- uses: actions/checkout@FULL_SHA
- run: test "$(git rev-parse HEAD)" = "$GITHUB_SHA"

Preserve the original run log showing both values. Do not rerun first: if main moved again, the rerun can produce a third observation and obscure the original mismatch.

3. Failure mode: a shell pipeline swallows the test exit code

A classic example pipes test output to tee. Depending on shell options, the pipeline can report tee success while the test command failed. The check turns green even though the log contains failures.

# Broken if pipeline status is not handled:
pytest -q | tee evidence/pytest.txt

# Explicitly preserve pytest's status in Bash:
set +e
pytest -q 2>&1 | tee evidence/pytest.txt
rc=${PIPESTATUS[0]}
set -e
exit "$rc"

The diagnostic clue is contradictory evidence: failed assertions in the report but a successful step/job. Fix exit-code propagation rather than weakening branch policy.

4. Failure mode: continue-on-error becomes permanent failure masking

# Broken policy: job may remain green despite test failure.
- name: Tests
  continue-on-error: true
  run: pytest

If temporary tolerance is needed to upload reports, capture the raw steps.ID.outcome, upload evidence with if: always(), then explicitly fail a final gate when the raw outcome is not success. The purpose is preservation, not normalization of failure.

5. Failure mode: coverage is generated after the upload step

Artifacts snapshot files that exist when the upload step runs. If coverage.xml is generated afterward, the artifact can be green and valid while simply not containing the report reviewers expect. Inspect artifact contents and step timestamps rather than assuming “upload succeeded” means “correct files uploaded.”

# Correct order
- run: coverage xml -o evidence/coverage.xml
- uses: actions/upload-artifact@FULL_SHA
  with:
    path: evidence/coverage.xml

6. Failure mode: cached build outputs become undeclared source

Caching package-manager downloads is usually safer than caching generated application outputs. If a build cache key omits compiler flags, source generators, environment variables or tool versions, stale output can be restored and accepted as if newly built. A clean run that fails while a warm run passes is strong evidence that correctness depends on retained state.

Repair by narrowing what is cached, strengthening keys with every correctness-relevant input, or using a build system with validated content-addressed caching. Preserve the mismatching output hashes before deleting caches.

7. Failure mode: CI depends on developer-machine state

8. Intentionally broken example: stale checkout plus hidden failure

jobs:
  tests:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@FULL_SHA
        with:
          ref: main            # wrong: mutable ref
      - run: pytest -q | tee pytest.txt  # wrong: exit status may be hidden
      - uses: actions/upload-artifact@FULL_SHA
        with:
          path: coverage.xml   # wrong: never generated

Do not fix all three lines at once. First preserve GITHUB_SHA, checked-out HEAD, the apparently green test step and the artifact warning/missing file. Then repair revision identity; then exit-code propagation; then report generation order. The staged repair proves each causal layer independently.

9. Failure mode: CI passes but merge remains blocked

• Confirm the required check name exactly matches the current Actions job/check name.
• Confirm the check belongs to the latest commit SHA, not an earlier successful revision.
• If a specific GitHub App is required as the check source, confirm Actions produced the expected source.
• If merge queue is enabled, confirm merge_group creates the required check for the queue SHA.
• If trigger-level path filtering suppresses the workflow, branch policy may wait for a check that was never created.

10. Production practices that make failure cheap to understand

• Use stable descriptive job/check names and deterministic step names.
• Record run ID/attempt/SHA, runner image and tool versions in a small evidence manifest.
• Upload JUnit/coverage/lint reports on failure with bounded retention and no secrets.
• Keep cache-hit/miss visible but make clean installation/build valid on a miss.
• Separate untrusted PR validation from publishing/deployment permissions.
• Measure flaky-test rate, queue time and critical path before adding retries or parallelism.

11. Lesson summary

CI diagnosis begins with identity, not with rerun. Preserve the first run and exact SHA, then follow the job graph to runner/toolchain, command exit status, cache/artifact state and merge-policy check. Correct the smallest broken layer and prove the repair with a new, clearly identified run.

Knowledge check

Official references and version notes

• Workflow syntax — Current workflow/job/step, permissions, needs and condition syntax.
• Dependency caching — Current cache matching, cache-hit semantics, security restrictions and eviction behavior.
• Workflow commands — Native annotations, outputs, summaries and runner communication.
• Status checks — How GitHub Actions jobs become checks and how conclusions affect merge policy.
• Troubleshoot required checks — Latest-SHA requirements, skipped checks and merge-queue considerations.
• actions/setup-python v7.0.0 — Immutable action revision used in the labs.
• actions/cache v6.1.0 — Immutable cache action revision used in the labs.
• actions/upload-artifact v7.0.1 — Immutable artifact action revision used in the labs.