Stages, needs DAGs, Dependency Graphs, Early Execution, and Pipeline Critical-Path Design: Guided Hands-On Workflow and Core Operations

1. Disposable lab: one pipeline, two graph designs

Create a throwaway project or branch named glci/ch09-dag. The lab uses only synthetic text files and short sleeps so timing is visible without consuming meaningful resources. No cloud account, privileged runner, secret, package registry, deployment, or paid feature is required.

2. Add one tiny timing helper

Create ci/stamp.sh. It does not calculate GitLab queue time; it records job-local wall-clock markers that make the execution order visible in traces and artifacts.

#!/bin/sh
set -eu
mkdir -p evidence
printf '%s job=%s stage=%s event=%s sha=%s\n'   "$(date -u +%Y-%m-%dT%H:%M:%SZ)"   "$CI_JOB_NAME" "$CI_JOB_STAGE" "$1" "$CI_COMMIT_SHA"   >> evidence/timeline.txt

Commit it executable. The authoritative scheduler evidence still comes from GitLab job metadata; the helper is only an independent trace-level cross-check.

3. Baseline: correct but deliberately over-serialized stages

The baseline has two independent build paths. The app build is short; documentation is intentionally slower. Both tests wait for both builds because the stage barrier does not know their true dependency shape. The package waits for the entire test stage.

stages: [build, test, package]

default:
  before_script:
    - chmod +x ci/stamp.sh

build_app:
  stage: build
  script:
    - ci/stamp.sh start
    - sleep 5
    - mkdir -p out
    - printf 'app_sha=%s\n' "$CI_COMMIT_SHA" > out/app.txt
    - ci/stamp.sh finish
  artifacts:
    paths: [out/app.txt, evidence/timeline.txt]
    expire_in: 1 day

build_docs:
  stage: build
  script:
    - ci/stamp.sh start
    - sleep 12
    - mkdir -p out
    - printf 'docs_sha=%s\n' "$CI_COMMIT_SHA" > out/docs.txt
    - ci/stamp.sh finish
  artifacts:
    paths: [out/docs.txt, evidence/timeline.txt]
    expire_in: 1 day

unit_app:
  stage: test
  script:
    - ci/stamp.sh start
    - test -s out/app.txt
    - grep -F "app_sha=$CI_COMMIT_SHA" out/app.txt
    - sleep 8
    - ci/stamp.sh finish
  artifacts:
    paths: [evidence/timeline.txt]
    expire_in: 1 day

check_docs:
  stage: test
  script:
    - ci/stamp.sh start
    - test -s out/docs.txt
    - grep -F "docs_sha=$CI_COMMIT_SHA" out/docs.txt
    - sleep 2
    - ci/stamp.sh finish
  artifacts:
    paths: [evidence/timeline.txt]
    expire_in: 1 day

package_demo:
  stage: package
  script:
    - ci/stamp.sh start
    - test -s out/app.txt
    - test -s out/docs.txt
    - sleep 4
    - ci/stamp.sh finish

The idealized stage-only lower bound from the artificial sleeps is about 12 + 8 + 4 = 24 seconds, excluding checkout, artifact transfer, runner queue, and command overhead. Record the observed pipeline duration separately; do not force it to match the ideal.

4. Measure before changing the graph

Run the baseline once. Capture the pipeline graph and a timing worksheet. From the Jobs API or UI, record each job ID, stage, created_at, started_at, finished_at, duration, and queued duration when exposed. Also download or inspect the producer artifacts by job ID.

5. Refactor only the real dependencies

Now make dependencies explicit. The tests each need only their own producer. The package needs both tested app flow and docs flow. Because unit_app consumes build_app artifacts and check_docs consumes build_docs artifacts, state that on the needs edges.

unit_app:
  stage: test
  needs:
    - job: build_app
      artifacts: true
  script:
    - ci/stamp.sh start
    - test -s out/app.txt
    - grep -F "app_sha=$CI_COMMIT_SHA" out/app.txt
    - sleep 8
    - ci/stamp.sh finish

check_docs:
  stage: test
  needs:
    - job: build_docs
      artifacts: true
  script:
    - ci/stamp.sh start
    - test -s out/docs.txt
    - grep -F "docs_sha=$CI_COMMIT_SHA" out/docs.txt
    - sleep 2
    - ci/stamp.sh finish

package_demo:
  stage: package
  needs:
    - job: unit_app
      artifacts: false
    - job: check_docs
      artifacts: false
    - job: build_app
      artifacts: true
    - job: build_docs
      artifacts: true
  script:
    - ci/stamp.sh start
    - grep -F "app_sha=$CI_COMMIT_SHA" out/app.txt
    - grep -F "docs_sha=$CI_COMMIT_SHA" out/docs.txt
    - sleep 4
    - ci/stamp.sh finish

The package has both control needs (tests must pass) and data needs (producer artifacts must arrive). Listing the build jobs directly makes the data source explicit; setting artifacts false on test needs avoids downloading their evidence artifacts into the package job.

6. Predict the DAG before running it

Write down these predictions before committing the DAG version:

1. unit_app becomes runnable after build_app at roughly 5 seconds, without waiting for build_docs.
2. check_docs becomes runnable after build_docs at roughly 12 seconds.
3. package_demo cannot run until both tests succeed and both build artifacts are available.
4. The idealized dependency-only critical path is roughly max(app: 5+8, docs: 12+2) + 4 = 18 seconds, a 6-second theoretical reduction from the staged 24-second lower bound.

Runner scarcity may reduce or eliminate the observed benefit. That is acceptable evidence: it means the graph was improved but capacity became the bottleneck.

7. Validate graph correctness before execution

Use CI Lint and, where permitted, pipeline simulation before committing. The pipeline editor visualization should show needs relationships as connecting lines. Confirm that the source/ref/SHA you later run is the revision you validated.

# Optional GitLab CLI path when glab is already installed and authenticated.
glab ci lint .gitlab-ci.yml
# A dry-run / pipeline simulation can be used when supported by your current glab/GitLab setup.

8. Run the DAG and compare observed evidence

Trigger the DAG pipeline from the same branch and record its new pipeline ID. Verify the source and SHA, then compare job starts. The key evidence is not merely “DAG pipeline is green”; it is that each consumer started after its required producer, before unrelated work when capacity allowed, and received the expected SHA-bound artifact.

9. Challenge: choose the missing dependency layer

Suppose security_source_scan reads only repository files and does not consume build outputs. Meanwhile integration_test requires out/app.txt. Which graph design is appropriate?

• security_source_scan: consider needs: [] if it truly has no job-produced prerequisite.
• integration_test: add a required needs edge to build_app with artifacts enabled.

The challenge is to classify the dependency, not copy a keyword. Runner tags would not fix a missing artifact edge; adding a stage barrier would restore correctness but might reintroduce unnecessary waiting.

10. No-runner faithful timing simulation

If GitLab execution is unavailable, model only the graph timing locally. This does not simulate GitLab artifact transfer or queueing, but it preserves the dependency reasoning:

# Stage lower bound from synthetic job durations:
# max(builds=5,12) + max(tests=8,2) + package=4 = 24

# DAG lower bound:
# app path: 5 + 8 = 13
# docs path: 12 + 2 = 14
# package waits for both paths, then 4 => 18

printf 'stage_model_seconds=%s\n' 24
printf 'dag_model_seconds=%s\n' 18
printf 'ideal_improvement_seconds=%s\n' 6

Label those values as a model. Do not present them as observed GitLab timings.

11. Cleanup and evidence retention

1. Retain the two pipeline IDs, timing worksheet, graph screenshots/notes, and non-secret artifact identity needed for the learning record.
2. Delete the disposable branch glci/ch09-dag only after verifying its exact identity.
3. If you used an otherwise empty throwaway project, delete only that exact project after preserving the evidence packet.
4. Do not erase job logs/artifacts merely to make the project look tidy before diagnosis is complete.

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-11. GitLab CI/CD DAG and artifact-transfer semantics are version-sensitive; verify the deployed GitLab version for Self-Managed/Dedicated installations.

• Make jobs start earlier with needs — stage barriers, DAG execution, immediate jobs, and practical examples.
• CI/CD YAML syntax reference — authoritative stages, needs, needs:artifacts, needs:optional, needs:project, and needs:pipeline:job semantics and limits.
• Pipeline editor — visualization of jobs, stages, and needs relationships plus full configuration inspection.
• CI Lint — syntax/logic validation and pipeline simulation that can expose invalid needs relationships before execution.
• Job artifacts — default previous-stage artifact fetching and how needs:artifacts changes data transfer.
• Troubleshooting job artifacts — missing/expired/inaccessible artifact failures.
• Jobs API — job IDs, stage/status, created_at, started_at, finished_at, duration, queued duration, and runner metadata for timing evidence.