Pipeline Troubleshooting, YAML Debugging, Runner Failures, Network Issues, Flaky Jobs, and Recovery Playbooks: Concepts, Architecture, and Mental Model

1. Troubleshooting starts by refusing to guess

A failed pipeline is only a symptom. The causal defect can live before a pipeline exists, during configuration compilation, while rules decide whether jobs exist, in the scheduler/runner boundary, inside the executor, in DNS/TLS or another dependency, in the job script, in artifact/report transfer, or after GitLab reports a successful deployment job while the external target is unhealthy.

Chapter 36 added policy origin and governance evidence. Chapter 37 turns that discipline into an incident method: preserve the first observable failure, identify which state transition did not happen as expected, change the smallest responsible layer, and prove recovery against both GitLab state and any external target.

2. Name the states before touching Retry

3. The evidence-first causal chain

The diagnostic sequence is deliberately ordered from cheapest and most stable evidence toward more invasive inspection. First ask whether GitLab created the intended pipeline for the intended SHA. Then prove what configuration GitLab evaluated. Only after a job exists should you investigate queueing and runner selection. Only after a runner accepts the job should you blame the script, network, image, or tool. Finally, a successful deployment job still requires target-state verification.

Every arrow is a hypothesis boundary. If no pipeline was created, runner debugging is wasted effort. If a job is pending with unmatched tags, application logs are irrelevant. If the script succeeds but the target still serves the old version, the failure is downstream of the job.

            flowchart TD
             A[Symptom or alert] --> B[Pipeline source / ref / SHA]
             B --> C[CI Lint + full/merged config + rules]
             C --> D[Pipeline/job graph + status]
             D --> E[Runner tags / manager / executor / image]
             E --> F[Script / tool / DNS / TLS / network]
             F --> G[Reports / artifacts / cache / registry]
             G --> H[Environment / deployment / external target]
             H --> I[Least-destructive repair]
             I --> J[Smallest safe retry or new pipeline]
             J --> K[Independent recovery verification]
          

4. Separate “invalid configuration” from “no matching pipeline”

These failures look similar to a beginner because both can produce “nothing useful ran,” but they occur at different stages. An invalid YAML/configuration error blocks compilation. A valid configuration can still create no pipeline because workflow:rules deliberately reject the event. A valid pipeline can then omit a specific job because that job’s rules do not match.

5. Read-only inspection before mutation

On a real disposable project, collect identifiers before rerunning anything. Use the UI or API/CLI available to you; the exact commands below are examples, not permission to inspect production systems you do not own.

# Record tool versions first.
git --version
glab --version 2>/dev/null || true
gitlab-runner --version 2>/dev/null || true

# Local source identity.
git rev-parse HEAD
git status --short

# Free CI Lint path when glab is authenticated to an authorized project.
glab ci lint --dry-run --include-jobs --ref main 2>/dev/null || true

# Optional runner-host read-only checks on an authorized runner manager.
gitlab-runner list 2>/dev/null || true
gitlab-runner verify 2>/dev/null || true
gitlab-runner status 2>/dev/null || true

gitlab-runner verify proves that registered runners can connect to GitLab; current Runner documentation explicitly says it does not prove that the Runner service is using them. Treat service state, runner registration, job routing, and actual job execution as separate evidence.

6. CI Lint is a compiler-side diagnostic, not a runner test

Current GitLab CI Lint can do more than parse YAML. Project-scoped lint can resolve local includes and project variables, and dry_run can simulate pipeline creation. The API can return the merged YAML and, with include_jobs, the jobs that would exist. A successful lint does not prove a runner exists, an image can be pulled, DNS works, or a deployment target is healthy.

# Example only: use a disposable authorized project and never put a token in the command history.
# Prefer an authenticated CLI or a header sourced from a protected local environment.
jq --null-input --arg yaml "$(cat .gitlab-ci.yml)" \
  '{content:$yaml,dry_run:true,include_jobs:true,ref:"main"}' \
  > /tmp/ch37-lint-request.json

# POST /projects/:id/ci/lint with the JSON body above.
# Preserve response.valid, response.errors, response.warnings,
# response.merged_yaml, response.includes, and response.jobs.

7. A pending job is evidence, not a generic “runner problem”

GitLab job status narrows the search. pending means the job is queued waiting for a runner. Runner tags are conjunctive: a runner must have every tag listed by the job. preparing means a runner is preparing the execution environment. waiting_for_resource points to resource-group serialization rather than missing capacity. These statuses prevent a blind “restart the runner” reflex.

8. DevOps connection: reproducibility includes failure reproduction

A delivery pipeline is reproducible only if another engineer can reconstruct the same source SHA, compiled configuration, rule outcome, runner/executor context, toolchain, evidence outputs, and external state. Troubleshooting therefore produces an evidence packet, not just a fixed pipeline. The packet is what lets a team distinguish a real remediation from an accidental green rerun.

Knowledge check

Official references and version notes

Further reading — current official GitLab sources

Behavior and version-sensitive assumptions in this chapter were checked against current GitLab documentation on 2026-09-13. Re-check your exact GitLab and GitLab Runner versions before production recovery automation.

• GitLab Docs — Validate GitLab CI/CD configuration
• GitLab Docs — CI Lint API
• GitLab Docs — Pipeline editor / full configuration
• GitLab Docs — CI/CD YAML syntax and retry
• GitLab Docs — Jobs API
• GitLab Docs — Pipelines API
• GitLab Docs — CI/CD jobs and statuses
• GitLab Docs — Troubleshooting CI/CD variables / debug trace
• GitLab Docs — Services and CI_DEBUG_SERVICES
• GitLab Docs — GitLab Runner commands
• GitLab Docs — Troubleshooting GitLab Runner
• GitLab Docs — Runner advanced configuration
• GitLab Docs — Docker executor image pull errors
• GitLab Docs — Docker build troubleshooting (DNS/TLS)
• GitLab Docs — Job artifacts and destructive deletion

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.