Checkpoint Lab — Artifacts, Reports, Cache, Dependencies, Retention, and Data Flow Between Jobs

1. Mission and production-model connection

Your checkpoint is a small delivery data contract. One job creates an authoritative artifact, one creates a structured report, and one consumes only the data it needs while tolerating cache absence. You must write the expected file matrix before execution, compare it with evidence, and explain why each object has its chosen lifetime.

This adds the data plane to the operating model: Chapter 12 decides which jobs exist, Chapter 13 controls variables/secrets, Chapter 14 supplies execution, Chapter 15 controls scheduling/concurrency, and Chapter 16 defines what trustworthy data moves between those jobs.

2. Preflight and assumptions

3. Write predictions before the run

Also predict: producer SHA/pipeline identity, report test count, artifact expiry/access, and which runner log lines indicate cache restoration/upload.

4. Checkpoint configuration

stages: [build, test, verify]

default:
  cache:
    key: "ch16-checkpoint-$CI_COMMIT_REF_SLUG"
    paths: [.chapter16-cache/]

produce:
  stage: build
  script:
    - mkdir -p out .chapter16-cache
    - printf 'sha=%s\npipeline=%s\n' "$CI_COMMIT_SHA" "$CI_PIPELINE_ID" > out/product.txt
    - sha256sum out/product.txt > out/product.txt.sha256
    - printf 'rebuildable-cache=%s\n' "$CI_COMMIT_REF_SLUG" > .chapter16-cache/marker.txt
  artifacts:
    name: "product-$CI_COMMIT_SHORT_SHA"
    paths: [out/product.txt, out/product.txt.sha256]
    expire_in: 2 days
    access: developer

test:
  stage: test
  needs:
    - job: produce
      artifacts: true
  script:
    - test -f out/product.txt
    - mkdir -p reports
    - |
      cat > reports/junit.xml <<'EOF'
      <?xml version="1.0" encoding="UTF-8"?>
      <testsuite name="chapter16-checkpoint" tests="1" failures="0">
        <testcase classname="dataflow" name="product_identity_verified"/>
      </testsuite>
      EOF
  artifacts:
    when: always
    reports:
      junit: reports/junit.xml
    expire_in: 2 days

verify:
  stage: verify
  needs:
    - job: produce
      artifacts: true
    - job: test
      artifacts: false
  script:
    - test -f out/product.txt
    - sha256sum -c out/product.txt.sha256
    - test ! -f reports/junit.xml
    - |
      if test -f .chapter16-cache/marker.txt; then
        echo "cache=present"
      else
        echo "cache=miss-acceptable"
      fi

5. Validate configuration and exact paths

Use CI Lint/Pipeline Editor. Inspect merged YAML and confirm no wildcard accidentally captures the repository, home directory, dotenv file, private key, or runner configuration. Then commit only on the disposable branch.

git switch -c ch16/checkpoint
git add .gitlab-ci.yml
git diff --cached --check
git diff --cached
git commit -m "ch16: data-flow checkpoint"
CHECKPOINT_SHA="$(git rev-parse HEAD)"
printf 'checkpoint_sha=%s\n' "$CHECKPOINT_SHA"
git push -u origin ch16/checkpoint

6. Capture pipeline/job evidence

Record pipeline ID/source/SHA and each job ID/status. Inspect artifact metadata without downloading broad content:

PROJECT_ID="12345678"
PIPELINE_ID="123456789"

glab api "projects/$PROJECT_ID/pipelines/$PIPELINE_ID" \
  --jq '{id,sha,ref,source,status,web_url}'
glab api "projects/$PROJECT_ID/pipelines/$PIPELINE_ID/jobs" --paginate \
  --jq '.[] | {id,name,status,stage,artifacts_file,artifacts,runner:(.runner.description // null)}'

7. Verify authoritative artifact identity

In verify, the checksum passes. Tie the artifact to the producer job ID, pipeline ID, and commit SHA. If you manually download it for the lab, inspect only the synthetic two-line file and checksum. In production, prefer digest/metadata verification over printing arbitrary artifact contents.

8. Verify report ingestion

Open pipeline Tests and confirm one test named product_identity_verified. This proves GitLab parsed the JUnit report. The test job also succeeded because its script exited zero. State these as separate observations.

9. Verify cache as disposable state

Inspect runner logs for cache restore/upload. If there is a miss, record it as a performance state and confirm all jobs still succeed. If there is a hit, record the key and runner/backend context but do not treat the marker as release evidence.

10. Cause one safe missing-artifact-style failure

On a second disposable commit, intentionally change verify to disable the artifact on the producer edge:

verify:
  needs:
    - job: produce
      artifacts: false
    - job: test
      artifacts: false
  script:
    - test -f out/product.txt

Predict that the pipeline graph remains valid but verify fails because the file is not downloaded. Run or use the supplied fixture, preserve the non-zero test failure, then repair artifacts: true. Do not use allow_failure, retry, or cache to hide the missing data edge.

11. Alternative failure when compute should be minimized: deliberate cache miss

Instead of the artifact failure, change the cache key to ch16-checkpoint-miss-$CI_PIPELINE_ID. Every pipeline uses a new key, producing a safe miss. Verify correctness remains unchanged. Restore the stable disposable key afterward. This is the preferred failure if you want to avoid a failed job.

12. Retention and access review

Confirm expire_in: 2 days and access: developer for the product artifact. Explain that latest-success retention may extend practical lifetime depending on project settings. The JUnit report uses the same short-lab retention. The cache has no evidence-retention promise and may be evicted independently.

13. Evidence packet

• Checkpoint commit SHA and pipeline ID/source.
• Producer/consumer job IDs and statuses.
• Artifact archive metadata and checksum result.
• JUnit Tests view showing one parsed case.
• Cache hit/miss log lines with no secret/environment dump.
• Prediction matrix versus observed files.
• Preserved intentional failure/miss and repaired configuration.
• Retention/access assumptions and cleanup result.

14. Cleanup / rollback

Restore/remove the checkpoint CI config, verify the exact disposable ref, then delete it. Short artifact expiry is sufficient for mandatory cleanup. If the entire project was created solely for this chapter, project deletion is acceptable only after confirming it contains no valuable data.

git switch main
git ls-remote --heads origin refs/heads/ch16/checkpoint
# Delete only the confirmed disposable ref:
git push origin --delete ch16/checkpoint
git branch -D ch16/checkpoint

15. Final verification checklist

• Artifact is bound to the expected producer job/pipeline/SHA and checksum.
• JUnit report is parsed separately from job status.
• Cache presence/absence changes only performance, not correctness.
• verify receives only the product artifact and no report artifact.
• No secrets, dotenv credentials, private context, runner config, or broad environment listing is stored.
• The intentional failure/miss has a preserved cause and deterministic repair.
• The disposable branch/config is removed and artifact cleanup follows retention policy.

Knowledge check

Checkpoint summary

You can now model GitLab delivery data explicitly: artifacts are identified outputs with access and retention, reports are structured platform inputs, caches are disposable acceleration state, and dependencies/needs define who receives what. That data-plane discipline is the prerequisite for Chapter 17, where environments and deployments attach pipeline outputs to real deployment state.

Official references

• GitLab Docs — Job artifacts
• GitLab Docs — Job artifacts troubleshooting
• GitLab Docs — Job Artifacts API
• GitLab Docs — CI/CD artifacts reports types
• GitLab Docs — Unit test reports
• GitLab Docs — Unit test report examples
• GitLab Docs — CI/CD caching
• GitLab Docs — CI/CD caching examples
• GitLab Docs — CI/CD YAML syntax reference
• GitLab Docs — Pass dotenv variables to specific jobs
• GitLab Docs — Jobs API
• GitLab Docs — Pipelines API
• GitLab Docs — Job artifacts administration
• GitLab Docs — Object storage