Artifacts, Reports, Cache, Dependencies, Retention, and Data Flow Between Jobs: Guided Hands-On Workflow and Core Operations

1. Disposable scenario and preflight

Use a disposable GitLab Free project or branch such as ch16/data-flow-lab. The lab writes only tiny text/XML files and a synthetic cache marker. It requires no package registry, cloud account, external database, secret variable, or production runner.

2. Create the three-job data-flow pipeline

The first job produces the authoritative build artifact and a harmless cache directory. The second creates a JUnit report while consuming only the build artifact. The third packages evidence and demonstrates that cache state is optional.

stages: [build, test, verify]

default:
  cache:
    key: "ch16-$CI_COMMIT_REF_SLUG"
    paths:
      - .chapter16-cache/

build_output:
  stage: build
  script:
    - mkdir -p out .chapter16-cache
    - printf 'commit=%s\n' "$CI_COMMIT_SHA" > out/build.txt
    - printf 'cache-created-by=%s\n' "$CI_PIPELINE_ID" > .chapter16-cache/marker.txt
    - sha256sum out/build.txt > out/build.txt.sha256
  artifacts:
    name: "ch16-build-$CI_COMMIT_SHORT_SHA"
    paths:
      - out/build.txt
      - out/build.txt.sha256
    expire_in: 2 days
    access: developer

test_report:
  stage: test
  dependencies:
    - build_output
  script:
    - test -f out/build.txt
    - mkdir -p reports
    - |
      cat > reports/junit.xml <<'EOF'
      <?xml version="1.0" encoding="UTF-8"?>
      <testsuite name="chapter16" tests="1" failures="0">
        <testcase classname="dataflow" name="artifact_received"/>
      </testsuite>
      EOF
  artifacts:
    when: always
    reports:
      junit: reports/junit.xml
    paths:
      - reports/junit.xml
    expire_in: 2 days

verify_flow:
  stage: verify
  needs:
    - job: build_output
      artifacts: true
    - job: test_report
      artifacts: false
  script:
    - test -f out/build.txt
    - sha256sum -c out/build.txt.sha256
    - |
      if test -f .chapter16-cache/marker.txt; then
        echo "cache-state=present"
      else
        echo "cache-state=miss-rebuildable"
      fi

3. Predict state before the first run

4. Validate before committing

Use Pipeline Editor/CI Lint to validate the configuration. Then inspect the staged diff so artifact/cache paths are exactly the synthetic directories.

git switch -c ch16/data-flow-lab
git add .gitlab-ci.yml
git diff --cached --check
git diff --cached
git commit -m "ch16: add artifact report cache lab"
git push -u origin ch16/data-flow-lab

5. First run: capture artifact identity

Record pipeline ID and commit SHA. In Build → Jobs → build_output, verify the artifact archive exists and the job log contains the upload step. Do not infer success from the file being visible in a later job; bind it to the producer first.

PROJECT_ID="12345678"
PIPELINE_ID="123456789"

glab api "projects/$PROJECT_ID/pipelines/$PIPELINE_ID/jobs" --paginate \
  --jq '.[] | {id,name,status,stage,ref,artifacts_file,artifacts}'

6. Verify report interpretation independently

Open the pipeline details and its Tests tab/summary. The single synthetic test case should be displayed as passed. Also inspect test_report job metadata. This proves two different facts: the XML was uploaded, and GitLab parsed it as JUnit.

If the Tests view is empty, inspect the report path and XML format before changing runner or permissions. A valid artifact archive is not proof of a valid report schema.

7. Observe cache miss and hit without depending on either

On the first run, runner logs may show a cache miss followed by cache creation. Push a trivial change that does not alter the cache key and run again. On compatible runner/cache topology, the second run may show cache restoration. If it still misses, the pipeline remains correct because no authoritative output depends on the cache.

8. Prove downstream file availability

In test_report, out/build.txt exists because dependencies names the producer. In verify_flow, build files exist because needs:artifacts is enabled for build_output. JUnit XML should not be present through the test_report edge because that edge disables artifact download.

Do not use ls -laR or an environment dump as evidence in real pipelines; such broad output can reveal unrelated files. Check only expected paths.

9. Inspect expiry and access

On the Build → Artifacts page or job details, verify the artifacts are associated with the expected jobs and have the configured short expiry. access: developer restricts UI/API downloads to Developer-or-higher roles, but downstream CI semantics are separate. Never describe artifact access as encryption or a secret manager.

10. Convert the test job to a DAG edge

Replace dependencies on test_report with an explicit needs edge and validate again:

test_report:
  stage: test
  needs:
    - job: build_output
      artifacts: true
  script:
    - test -f out/build.txt
    - ./write-safe-junit-fixture.sh

This change does two things: it permits earlier scheduling when the producer completes, and it makes the artifact dependency explicit. Do not combine needs and dependencies in the same job as a normal design pattern; choose the model that matches scheduling/data intent.

11. Challenge: choose the correct surface

Classify each requirement before writing YAML:

12. Cleanup

The safest cleanup is to revert/remove the synthetic CI file on the disposable branch and then delete only that exact branch after verifying its identity. Individual artifact deletion is irreversible and normally requires Maintainer/Owner; it is optional in this lab because short expiry already bounds storage.

git switch main
git ls-remote --heads origin refs/heads/ch16/data-flow-lab
# Delete only after confirming this exact disposable branch:
git push origin --delete ch16/data-flow-lab
git branch -D ch16/data-flow-lab

Knowledge check

Summary

You created three different data behaviors and verified them independently: an identified artifact, a parsed report, and a disposable cache. You then constrained downloads with explicit dependency edges and inspected retention/access without exposing sensitive values.

Official references

• GitLab Docs — Job artifacts
• GitLab Docs — Job artifacts troubleshooting
• GitLab Docs — Job Artifacts API
• GitLab Docs — CI/CD artifacts reports types
• GitLab Docs — Unit test reports
• GitLab Docs — Unit test report examples
• GitLab Docs — CI/CD caching
• GitLab Docs — CI/CD caching examples
• GitLab Docs — CI/CD YAML syntax reference
• GitLab Docs — Pass dotenv variables to specific jobs
• GitLab Docs — Jobs API
• GitLab Docs — Pipelines API
• GitLab Docs — Job artifacts administration
• GitLab Docs — Object storage