Stages, needs DAGs, Dependency Graphs, Early Execution, and Pipeline Critical-Path Design: Configuration, Design Choices, and Tradeoffs

1. DAG design is a tradeoff, not a maturity badge

A stage-only pipeline is not “primitive.” It is often the most maintainable representation when every test genuinely depends on every build, when the pipeline is short, or when runner capacity is limited. A DAG earns its complexity when it encodes real independence that materially improves feedback or delivery latency.

2. Simple stages versus explicit DAG edges

The hybrid model is often best: preserve stage names for human grouping and status meaning while adding needs only where a measurable barrier is unnecessary.

3. Early feedback versus resource contention

Moving a source-only linter to needs: [] can produce feedback earlier. Moving twenty CPU-heavy jobs to the runnable frontier may instead create a queue, evict useful cache, saturate network storage, or slow every job. Pipeline latency is a system property:

effective_completion ≈ graph_dependency_time
                     + runner_queue_delay
                     + execution_time
                     + artifact_transfer
                     + external_wait

Measure these components before concluding that “more parallel” is faster. If queue time dominates, a wider DAG can make the interface look parallel while the execution substrate remains serialized.

4. Fan-out breadth and fan-in pressure

Fan-out lets independent work proceed. Fan-in waits for multiple predecessors. Current GitLab limits a single normal needs array to 50 jobs on GitLab.com (50 by default on Self-Managed/Dedicated). Do not design toward the maximum casually. A 40-edge fan-in may indicate that a smaller verification aggregate, matrix structure, or component boundary would be easier to reason about.

5. Required versus optional needs

needs:optional exists because job membership can be conditional. If a needed job is absent and the need is required, pipeline creation can fail. If the edge is optional and the job is absent, GitLab can proceed after the remaining needs are satisfied; if the optional need is the only need and it is absent, the consumer can start immediately.

docs_quality:
  stage: test
  rules:
    - changes:
        paths: [docs/**/*]
  script: ./ci/check-docs.sh

summary:
  stage: package
  needs:
    - job: docs_quality
      optional: true
  script: ./ci/write-summary.sh

6. needs is also a dataflow decision

With ordinary stages, artifact download can be broad. With needs, only listed producers are eligible artifact sources. artifacts: true is the default on a normal needs edge; use false when the edge is control-only.

publish_preview:
  stage: package
  needs:
    - job: unit_test
      artifacts: false
    - job: build_preview
      artifacts: true
  script:
    - test -s dist/preview.tar

This is easier to audit than relying on “whatever artifacts existed in earlier stages.” Chapter 10 will deepen retention, reports, dependencies, and cross-job dataflow; here the design rule is simply that graph correctness and data correctness must agree.

7. Do not confuse same-pipeline dependencies with cross-pipeline artifact features

The word needs appears in several features with different semantics:

8. Security and trust remain attached to jobs and artifacts

A DAG edge does not authorize anything by itself. Runner trust, protected variables, CI_JOB_TOKEN permissions, external credentials, and artifact sensitivity remain separate state. Earlier execution can increase exposure if a privileged job is made runnable before the checks that were implicitly protecting it through stage order.

When refactoring, identify whether an earlier stage was acting as a policy gate, not merely a technical dependency. If a deployment must not become runnable until review/security checks finish, encode those checks as required needs or preserve an appropriate barrier.

9. Worked decision table

10. Current-version assumptions to record

• Primary docs verified 2026-09-11.
• Normal needs: maximum 50 listed jobs on GitLab.com; Self-Managed/Dedicated default 50 and administrator-configurable.
• Same-stage needs are supported.
• needs: [] enters the runnable frontier immediately when the pipeline is created.
• Normal needs artifact transfer is restricted to listed producers; artifacts: true is the default.
• Do not combine needs and dependencies in the same job.
• needs:project is Premium/Ultimate and has different wait semantics.

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-11. GitLab CI/CD DAG and artifact-transfer semantics are version-sensitive; verify the deployed GitLab version for Self-Managed/Dedicated installations.

• Make jobs start earlier with needs — stage barriers, DAG execution, immediate jobs, and practical examples.
• CI/CD YAML syntax reference — authoritative stages, needs, needs:artifacts, needs:optional, needs:project, and needs:pipeline:job semantics and limits.
• Pipeline editor — visualization of jobs, stages, and needs relationships plus full configuration inspection.
• CI Lint — syntax/logic validation and pipeline simulation that can expose invalid needs relationships before execution.
• Job artifacts — default previous-stage artifact fetching and how needs:artifacts changes data transfer.
• Troubleshooting job artifacts — missing/expired/inaccessible artifact failures.
• Jobs API — job IDs, stage/status, created_at, started_at, finished_at, duration, queued duration, and runner metadata for timing evidence.