Pipeline Troubleshooting, YAML Debugging, Runner Failures, Network Issues, Flaky Jobs, and Recovery Playbooks: Configuration, Design Choices, and Tradeoffs

1. Retry a job, retry failed jobs, or create a new pipeline?

Current GitLab behavior makes this distinction operationally important. A job retry stays in the same pipeline. GitLab’s YAML reference states that included configuration is not fetched again when a job is rerun; jobs in that pipeline use the configuration resolved when the pipeline was created. A new/rerun pipeline resolves includes again, so changed remote/project includes can change the experiment.

2. Retry APIs do not mean “safe to retry”

GitLab exposes a job retry endpoint and a pipeline retry endpoint. The pipeline endpoint retries failed or canceled jobs; if none exist, it has no effect. Those APIs provide mechanics, not idempotency analysis. A failed deployment job may already have partially changed an external system.

# Examples only — use an authorized disposable project.
# Retry one job:
# POST /projects/:id/jobs/:job_id/retry
# Retry failed/canceled jobs in one pipeline:
# POST /projects/:id/pipelines/:pipeline_id/retry

# Before either action, record:
# project/ref/SHA, pipeline ID, job ID/status/failure_reason,
# environment/deployment record, artifact digest, and external target state.

3. Automatic retry should target known transient classes

The retry keyword defaults to zero when omitted and supports a maximum of two retries. Use failure-specific retry only when the failure is understood and side effects are safe. Current documentation is especially version-sensitive: GitLab 19.1 refines several runner failure reasons and deprecates some older aggregate names. If your estate spans versions, build the retry policy against the oldest supported instance/Runner semantics and test it.

test:
  script: ./run-tests.sh
  retry:
    max: 1
    when:
      - runner_external_dependency_failure
      - runner_interrupted

memory_sensitive_test:
  script: ./run-memory-test.sh
  retry:
    max: 1
    exit_codes: 137

Do not use retry: 2 on every script failure to hide flaky tests. A retry policy should state the expected transient cause and preserve first-attempt evidence.

4. Debug tracing versus secret exposure

CI_DEBUG_TRACE=true makes Runner output much more verbose, but GitLab explicitly warns that it exposes all variables and secrets available to the job. CI_DEBUG_SERVICES is also dangerous: service logs can interleave with the job trace in ways that defeat masking. Treat both as security-sensitive incident actions.

5. Broad cleanup versus targeted repair

Deleting caches, artifacts, logs, workspaces, or whole pipelines can make symptoms disappear while destroying the evidence needed to explain them. GitLab documents artifact/job-log deletion as destructive and irreversible. Preserve first, then clean the smallest state that is demonstrably corrupt.

6. Temporary mitigation is not permanent remediation

A mitigation restores flow while containing risk: route a job to a known-good runner, disable one failing optional integration, or pin a known-good immutable image. Remediation removes the root cause: repair DNS trust, fix runner fleet configuration, eliminate flake, or correct the deployment contract. Every mitigation needs an owner, expiry/review trigger, and evidence that it does not silently become permanent.

7. Worked decision table

Use this table before applying a recovery action during the checkpoint.

8. Keep the state boundaries explicit

• Repository YAML and merged pipeline configuration are not runner-host config.toml.
• GitLab pipeline/job/report/artifact/cache state is not the same as an external registry, Kubernetes cluster, cloud account, or IaC state backend.
• CI_JOB_TOKEN, personal/deploy/trigger tokens, and OIDC identities have different trust and lifetime models.
• A build artifact digest is evidence of what was produced; deployment authorization is evidence of who/what could deploy it; target verification is evidence of what is actually running.
• Copied YAML is a local fork; a versioned include/component is an external dependency whose exact revision must be part of evidence.

Knowledge check

Official references and version notes

Further reading — current official GitLab sources

Behavior and version-sensitive assumptions in this chapter were checked against current GitLab documentation on 2026-09-13. Re-check your exact GitLab and GitLab Runner versions before production recovery automation.

• GitLab Docs — Validate GitLab CI/CD configuration
• GitLab Docs — CI Lint API
• GitLab Docs — Pipeline editor / full configuration
• GitLab Docs — CI/CD YAML syntax and retry
• GitLab Docs — Jobs API
• GitLab Docs — Pipelines API
• GitLab Docs — CI/CD jobs and statuses
• GitLab Docs — Troubleshooting CI/CD variables / debug trace
• GitLab Docs — Services and CI_DEBUG_SERVICES
• GitLab Docs — GitLab Runner commands
• GitLab Docs — Troubleshooting GitLab Runner
• GitLab Docs — Runner advanced configuration
• GitLab Docs — Docker executor image pull errors
• GitLab Docs — Docker build troubleshooting (DNS/TLS)
• GitLab Docs — Job artifacts and destructive deletion

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.