Pipeline Performance, Cost Control, Caching Strategy, Queue Reduction, Selective Execution, and Optimization: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

Preserve the original pipeline/job IDs and first-failure evidence. Confirm pipeline source/ref/SHA and compiled YAML; confirm rule results and selected jobs/tests; inspect graph and queue; confirm runner/executor/image/toolchain; inspect script and transfer phases; inspect artifacts/cache metadata and external target state; then change only the smallest proven layer.

2. Failure mode: average improves while tail latency worsens

Before optimization: median 7m, p95 10m. After adding shards: median 5m, p95 18m. The average team experience may look better, but rare queue saturation has become severe. Inspect queue p95 by runner pool and job family; do not declare success from median alone.

3. Failure mode: required tests were skipped

A path rule cuts compute by 40%, but a shared schema change does not trigger integration tests. The pipeline is faster because assurance disappeared. The failing layer is repository rule/dependency modeling, not the test runner.

# INTENTIONALLY BROKEN: too narrow for a shared schema dependency
integration:
  script: ./test-integration.sh
  rules:
    - changes:
        paths: [service-a/**/*]
    - when: never

Repair by including shared triggers or falling back to the full suite when dependency mapping is uncertain.

4. Failure mode: cache dominates network

A 1.2 GB cache saves 20 seconds of package resolution but takes 65 seconds to download and extract on cold autoscaled workers. Measure cache restore/save sections and bytes. Narrow the path, key, or remove it. Do not raise runner size to fix object-storage transfer unless evidence shows CPU compression is the bottleneck.

5. Failure mode: matrix parallelism increases queue time

Eight shards each run 45 seconds, but only two eligible runners exist. Queue p95 rises beyond the original two-shard pipeline. The local job metric improved; the system metric regressed. Reduce shard count or address bounded capacity/request flow.

6. Failure mode: cost reduced by using an insecure shared runner

A protected release job is moved from an ephemeral protected runner to a cheap shared Shell runner. Compute cost falls, but untrusted jobs can share host state and credentials. This violates the trust model; reject the optimization even if latency/cost metrics look better.

7. Intentionally broken “optimization bundle”

This combines multiple unreviewed changes, so any result is hard to attribute and some are unsafe:

# INTENTIONALLY BROKEN — do not copy.
default:
  cache:
    key: global
    paths: [./]

fast_tests:
  parallel: 12
  script: ./test.sh --changed-only
  rules:
    - changes: [src/**/*]

release:
  tags: [cheap-shared]
  script: ./release.sh

Problems: giant global cache, uncontrolled cache content, high shard count without capacity evidence, ambiguous “changed-only” coverage, and release trust boundary regression. Repair each layer separately and compare against the same baseline.

8. Runner configuration can mimic application slowness

If jobs remain pending, inspect eligible runner count and Runner concurrent/limit/request_concurrency. Current GitLab Runner warns about long-polling scenarios such as global concurrency below runner count or high-volume runners capped at request concurrency 1.

9. Blind retries distort optimization data

Retries can warm caches, land on a different runner, or occur after load subsides. Preserve the first attempt and label retries as separate samples. Do not combine them without explaining sampling bias.

10. Do not optimize CI by weakening deployment verification

Removing rollout checks can make the pipeline “finish” earlier while moving failure detection outside the pipeline. Deployment submission and external health are different states; performance work must keep the required health evidence.

11. Least-destructive corrections

Knowledge check

12. Summary

Optimization failures are causal-layer failures: tail queue, rule coverage, transfer overhead, capacity saturation or trust regression. Preserve evidence, isolate the layer and correct only that layer.

Official references and version notes

Documentation verification date: 2026-09-13. Examples use GitLab/GitLab Runner 19.3.2 semantics as the timestamped baseline where a concrete version matters. Project CI/CD analytics, needs, caches, artifacts and job rules are available across Free/Premium/Ultimate. Current project analytics exposes median and p95 pipeline duration. Job execution on instance runners contributes compute usage; created/pending queue time does not, so latency and compute cost are related but distinct metrics. Runner flow is bounded by global concurrent, per-runner limit, and job-request request_concurrency; long-polling misconfiguration can create queue delays. Caches are an optimization and are not guaranteed to exist. With needs, jobs fetch artifacts only from listed dependencies, and artifacts: false can avoid transfers. rules:changes:compare_to can skip unaffected work, but only after correctness requirements are made explicit. The mandatory labs use synthetic local data and Python standard-library tooling; no paid analytics, cloud account, privileged runner or production workload is required.

• Compute minutes — official reference.
• Instance runner compute usage — official reference.
• CI/CD analytics — official reference.
• Runner advanced configuration — official reference.
• Caching in GitLab CI/CD — official reference.
• Caching examples — official reference.
• Job artifacts — official reference.
• CI/CD YAML reference — official reference.
• needs DAGs — official reference.
• Job rules — official reference.
• Pipeline settings and auto-cancel — official reference.
• Runner monitoring — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.