Pipeline Performance, Cost Control, Caching Strategy, Queue Reduction, Selective Execution, and Optimization: Concepts, Architecture, and Mental Model

1. The practical problem: fast-looking jobs can still produce a slow, expensive pipeline

Chapter 34 established an observability baseline. Optimization begins only after that baseline exists. A pipeline can be slow because jobs wait in a queue, because the DAG serializes independent work, because every job downloads large artifacts, because a cache miss is more expensive than rebuilding, or because too many parallel shards saturate the runner pool. These causes require different corrections.

Cost and latency also diverge. GitLab compute usage for instance runners is based on job execution, not created/pending queue time. A pipeline can have terrible developer latency from queueing while consuming little additional compute. Conversely, aggressive parallelism can reduce wall-clock time while consuming more total compute minutes.

2. Mental model: baseline → critical path → hypothesis → bounded change → comparable rerun → verify

Start with evidence, not a YAML trick. Measure pipeline wall time, per-job queue/run time, cache/artifact transfer, runner usage and correctness. Identify the critical path: the dependency chain whose delays determine completion. Form one hypothesis, change one layer, rerun a comparable workload, then verify latency, compute/cost and correctness together.

            flowchart TD
             A[Baseline IDs + SHA + graph] --> B[Queue / critical path / I-O analysis]
             B --> C[One optimization hypothesis]
             C --> D[Bounded config or runner change]
             D --> E[Comparable rerun]
             E --> F[Latency + cost + correctness evidence]
             F -->|better and safe| G[Adopt + monitor]
             F -->|regression / uncertainty| H[Rollback / reject]
          

3. Freeze the state you are about to optimize

4. Four different performance numbers

5. Critical path beats “optimize the biggest job”

Suppose build runs 180 s, unit tests 140 s, integration 220 s and package 40 s. If unit and integration both depend on build and package depends only on integration, the wall-time critical path is build → integration → package. A 60-second improvement to unit tests may save compute but not change completion time at all.

build=180
unit=140
integration=220
package=40
critical=build+integration+package
print("critical_path_s=", critical)

6. Queue is a separate optimization surface

Runner capacity, tags, executor eligibility, global concurrent, per-runner limit and request_concurrency can dominate wait time. Current Runner guidance warns that long polling with too-low concurrency/request settings can create severe processing delays.

Adding more matrix jobs while capacity is saturated can increase p95 queue time even though each shard is shorter. Optimize the system, not a single job.

7. Cache is a speculative optimization, not a dependency guarantee

GitLab caches are designed for reusable dependencies and may live locally or in distributed object storage. They can miss. They can also cost more to compress, upload, download and extract than the work they avoid. Key caches from dependency-lock content where practical, measure bytes and hit rate, and make jobs correct when the cache is absent.

8. Artifacts are dataflow, and dataflow has transfer cost

Later-stage jobs normally fetch artifacts from earlier stages unless you constrain dependencies. With needs, artifact downloads come only from jobs listed in needs; an edge can set artifacts: false. This can shorten both graph wait and transfer time, but only when the downstream job truly does not need those bytes.

9. Selective execution saves resources only if the selection is correct

rules:changes:compare_to can skip jobs when relevant files did not change. In a monorepo this can be powerful. But a “frontend-only” change may still require shared-contract or security tests. Define the dependency model and a safe fallback before skipping anything.

10. More parallelism can reduce latency and increase cost—or increase latency too

Parallel jobs reduce a critical path when capacity exists and setup/merge overhead is modest. If runner capacity is constrained, extra shards become queued work and may worsen tail latency. Total compute often rises because setup and duplicated dependency work occur per shard.

11. Read-only baseline worksheet

Before changing YAML, capture an evidence row per job:

pipeline_id,job_id,name,queue_s,run_s,cache_bytes,artifact_bytes,status,selected_tests
3500,35001,build,8,180,0,42000000,success,build
3500,35002,unit,12,140,18000000,2000000,success,unit:all
3500,35003,integration,160,220,18000000,3000000,success,integration:all
3500,35004,package,4,40,0,45000000,success,package

This already shows two possible causes: integration has both the longest execution and dominant queue. You need graph and runner evidence to know which change attacks wall time.

12. Common wrong approaches

Knowledge check

13. Summary

Optimization is an experiment: freeze identity and correctness evidence, distinguish queue/run/wall/compute, identify the critical path, change one bounded layer, rerun comparably, and adopt only improvements that preserve assurance and trust boundaries.

Official references and version notes

Documentation verification date: 2026-09-13. Examples use GitLab/GitLab Runner 19.3.2 semantics as the timestamped baseline where a concrete version matters. Project CI/CD analytics, needs, caches, artifacts and job rules are available across Free/Premium/Ultimate. Current project analytics exposes median and p95 pipeline duration. Job execution on instance runners contributes compute usage; created/pending queue time does not, so latency and compute cost are related but distinct metrics. Runner flow is bounded by global concurrent, per-runner limit, and job-request request_concurrency; long-polling misconfiguration can create queue delays. Caches are an optimization and are not guaranteed to exist. With needs, jobs fetch artifacts only from listed dependencies, and artifacts: false can avoid transfers. rules:changes:compare_to can skip unaffected work, but only after correctness requirements are made explicit. The mandatory labs use synthetic local data and Python standard-library tooling; no paid analytics, cloud account, privileged runner or production workload is required.

• Compute minutes — official reference.
• Instance runner compute usage — official reference.
• CI/CD analytics — official reference.
• Runner advanced configuration — official reference.
• Caching in GitLab CI/CD — official reference.
• Caching examples — official reference.
• Job artifacts — official reference.
• CI/CD YAML reference — official reference.
• needs DAGs — official reference.
• Job rules — official reference.
• Pipeline settings and auto-cancel — official reference.
• Runner monitoring — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.