Pipeline Performance, Interruptible Jobs, Retry, Failure Handling, Debugging, and Cost Control: Diagnostics, Failure Modes, Security, and Performance

1. Diagnostic rule: preserve the failure before changing the pipeline

Performance incidents are easy to make less diagnosable. Retrying immediately creates a new job ID. Erasing logs destroys evidence. Changing YAML before recording the SHA makes comparisons ambiguous. Adding runners may hide a scheduling defect rather than fix it.

Use one sequence throughout this lesson: preserve evidence → identify scope → inspect configuration/permissions/logs/API/runner state → choose the least destructive correction → verify with the same metrics.

2. Evidence bundle for a slow or failed job

PROJECT_ID="12345678"
PIPELINE_ID="12345"
JOB_ID="67890"
mkdir -p evidence/ch21-incident

glab api "projects/$PROJECT_ID/pipelines/$PIPELINE_ID" > evidence/ch21-incident/pipeline.json
glab api "projects/$PROJECT_ID/jobs/$JOB_ID" > evidence/ch21-incident/job.json
glab api "projects/$PROJECT_ID/jobs/$JOB_ID/trace" > evidence/ch21-incident/job-trace.txt
git rev-parse HEAD > evidence/ch21-incident/local-head.txt

jq "{id,sha,ref,status,duration,queued_duration}" evidence/ch21-incident/pipeline.json
jq "{id,name,status,failure_reason,duration,queued_duration,runner,runner_manager}" evidence/ch21-incident/job.json

Job traces can contain sensitive data. Store incident evidence with the same access discipline as the original job log, and never paste secret-bearing traces into public issues.

3. Failure mode: retry hides a deterministic defect and multiplies cost

Broken configuration:

compile:
  script:
    - ./compile.sh   # always exits 2 for a syntax error
  retry: 2

Expected evidence: three job attempts/instances are processed before the required failure settles. The correct fix is not a larger retry count. Inspect the first failure, reproduce the deterministic defect, fix code/configuration, and set retry to zero unless a separate transient failure class exists.

Remember that GitLab pipeline duration does not include the initial run of a retried/re-run job in its duration calculation. If you are assessing wasted compute, inspect the individual job attempts and current compute-usage evidence rather than relying on pipeline duration alone.

4. Failure mode: a stateful deployment is marked interruptible

Imagine a deployment job writes version A to one target, then is canceled before the second target. GitLab can correctly report the job canceled while the external environment is partially mutated.

deploy:
  stage: deploy
  interruptible: true   # unsafe unless cancellation semantics are proven
  script:
    - ./update-target-one.sh
    - ./update-target-two.sh

Repair: default the stateful deployment to non-interruptible, make the deployment operation idempotent/transactional where possible, serialize shared resources, and document recovery. Do not confuse “newer commit exists” with authorization to abandon a production mutation.

5. Failure mode: allow_failure masks a required quality gate

Broken configuration:

contract-test:
  script: ./verify-public-api.sh
  allow_failure: true

The pipeline can report success even when the API compatibility contract fails. This is a governance defect, not a performance win. Repair by restoring allow_failure: false and fixing the test/runtime. If the check is genuinely advisory, document why downstream release policy does not depend on it.

6. Failure mode: cache tuning increases time and network transfer

Symptoms: a job’s script body is short, but the trace spends longer restoring or uploading cache than the uncached dependency installation used to take. A broad cache key can also create a large archive and increase invalid content reuse.

Inspect cache restore/upload timings in the job trace, archive size, hit/miss messages, runner/network conditions, and key scope. Repair by reducing cached paths, keying to the lockfile/runtime identity, splitting caches, or deleting the cache entirely if recomputation is cheaper.

7. Failure mode: chronic queue time is misdiagnosed as a slow script

If queued_duration is much larger than duration, inspect runner tags, protected-runner eligibility, scope, paused/offline state, concurrency limits, and fleet saturation. A code optimization cannot remove a 90-second queue. Conversely, doubling runners does not help a 90-second serial script that starts immediately.

Where your role permits it, inspect runner metadata or the Runners API. For hosted runners, also check the current namespace compute quota; over-quota jobs may not be picked up by instance runners.

8. Failure mode: “turn on full debug trace” creates a security incident

GitLab explicitly warns that CI_DEBUG_TRACE=true can expose all variables/secrets available to the job in uploaded logs. Do not enable it on a valuable project just because ordinary output is insufficient.

Escalation order: CI Lint/compiled config → ordinary job trace → structured API fields/failure reason → runner metadata → minimal secret-free reproduction → only then deep debug in a controlled context. If a real credential is exposed, revoke/rotate it first; deleting the log is cleanup, not containment.

9. Intentionally broken example: the “make it green” anti-pattern

This YAML combines three wrong fixes:

quality:
  script:
    - ./deterministic-test.sh
  retry: 2
  interruptible: true
  allow_failure: true

Interpret the evidence: deterministic failure is repeated; an obsolete pipeline may cancel it; and even after the final failure the pipeline can remain green. The least destructive repair is to restore the invariant first:

quality:
  script:
    - ./deterministic-test.sh
  retry: 0
  interruptible: true   # only if the test is stateless
  allow_failure: false

Then fix the deterministic test defect. The only retained “optimization” is safe cancellation of obsolete stateless work.

10. Chronic slowness is a security and governance problem

When required feedback is chronically slow, teams create bypasses: local-only “temporary” skips, manual approvals without evidence, direct pushes, or disabled tests. Treat sustained pipeline latency as an operational risk. Publish target feedback budgets, track queue and critical-path metrics, and prioritize the bottlenecks that most affect merge/release decisions rather than chasing isolated micro-optimizations.

11. Rate, storage, network, and cost considerations

Retries and pipeline storms multiply API calls and runner consumption; GitLab installations can also enforce pipeline creation rate limits. Artifact/cache/image transfers may dominate a job even when CPU use is low. Preserve causal evidence: bytes, timestamps, retry count, pipeline count, and runner class. Avoid hardcoded currency estimates because GitLab quotas, runner cost factors, and infrastructure pricing vary.

Knowledge check

Summary

Good diagnosis protects the original evidence and fixes the causal layer: deterministic code defects fail fast; transient infrastructure faults get bounded retry; stateful work is not casually interruptible; required gates remain required; cache must save more than it costs; and runner queueing is measured separately from script duration.

Official references

Primary sources used for the current GitLab 19.3 behavior taught in this lesson:

• GitLab Docs — CI/CD YAML syntax reference
• GitLab Docs — CI/CD pipelines
• GitLab Docs — Jobs API
• GitLab Docs — Pipelines API
• GitLab Docs — Debugging CI/CD pipelines
• GitLab Docs — Validate CI/CD configuration
• GitLab Docs — Compute minutes
• GitLab Docs — Configure runners
• GitLab CLI — glab ci
• GitLab CLI — glab ci trace