Container Registry, Package Registry, Generic Packages, Dependency Proxy, and Artifact Promotion: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Preserve pipeline/job IDs and the first failing trace.
2. Confirm CI_PIPELINE_SOURCE, ref, and CI_COMMIT_SHA.
3. Inspect merged configuration and rule result.
4. Record package/image coordinate and expected checksum/digest.
5. Record token class, source/target project, allowlist assumption, and actor permission—never token value.
6. Inspect queue, runner/executor, DNS/TLS/network only after configuration/authorization evidence.
7. Compare registry/package metadata and consumer digest.
8. Inspect cleanup/protection policies.
9. Apply the least destructive correction and retry only the smallest scope.

2. Failure: latest silently moved

Pipeline A tested digest D1, but a later Pipeline B pushed latest → D2. Production redeploying latest now consumes D2 even if its deployment record still refers to Pipeline A.

Repair: restore the deployment reference to repository@D1 if D1 is still retained and authorized. Do not rebuild D1 from source and call it the same artifact. Then add policy requiring digest evidence for deployments.

3. Failure: same Generic Package version accepts another file

Generic Packages allow duplicates by default at the format/settings level. If two producer pipelines publish the same package name/version/file semantics, a consumer may not have a unique human-readable version contract even though each file has a checksum.

Expected:
  glci-ch23-demo / 1.2.0 / payload.txt -> sha256:AAA
Observed after second publisher:
  same package name/version contains another or replaced-looking asset context -> sha256:BBB
Root cause layer:
  package naming / duplicate policy / publication governance, not runner cache

Repair: mint unique immutable versions, disable/configure duplicates where appropriate, protect release package names, and keep checksum verification mandatory.

4. Failure: cross-project job token denied

A valid CI_JOB_TOKEN from project B is not a universal GitLab credential. For private project A, A must allow B (or its group) when cross-project access is required, and the user who triggered B’s job must have the permissions needed in A.

5. Failure: Dependency Proxy treated as provenance

A pipeline uses ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/vendor/image:stable and assumes the cached result is immutable. The proxy is operating correctly; the design is wrong. A moving upstream tag can still resolve to different content over time.

Repair: record a verified upstream digest and consume that digest through the supported registry path. Keep the proxy for caching/rate-limit behavior, not for trust claims.

6. Failure: credential appears in trace

Common causes include shell tracing, embedding token in a URL, using docker login -p, or archiving a generated Docker config. The safer pattern uses predefined credentials, --password-stdin, masked variables where appropriate, isolated runners, and artifacts that explicitly exclude auth/config directories.

7. Failure: cleanup policy removes needed rollback tag

Preserve the cleanup rule configuration, affected repository, tag list before/after, release/deployment references, and registry digest inventory. Current container cleanup excludes protected and immutable tags, which is useful only when critical rollback identities are actually protected. Cleanup deletes tags according to policy; storage reclamation of unreferenced layers is a separate concern.

8. Performance diagnosis: registry speed is not always registry storage

9. Intentionally broken pipeline and causal repair

stages: [publish, consume]

publish_bad:
  stage: publish
  script:
    - echo "publishing only latest"
    - docker build -t "$CI_REGISTRY_IMAGE:latest" .
    - docker push "$CI_REGISTRY_IMAGE:latest"

consume_bad:
  stage: consume
  script:
    - docker pull "$CI_REGISTRY_IMAGE:latest"
    - echo "assume this is the artifact tested earlier"

The YAML can compile and both jobs can succeed, yet the pipeline has no immutable distribution evidence. The repair is architectural: publish a unique tag, capture the pushed manifest digest, emit it as evidence, and make the consumer pull @sha256:…. Job success was never the missing fact.

10. Recovery drill

Given a failed consumer, write down the exact expected coordinate/digest before touching the registry. Then answer: did the producer publish it, is the consumer authorized, can the runner reach it, did retention delete it, or did a mutable tag resolve elsewhere? Correct only the proven failing layer.

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-12. Generic Packages, Package Registry, Container Registry, Dependency Proxy, protected packages, and protected container tags are documented for Free/Premium/Ultimate unless noted otherwise. Current immutable container-tag rules are Ultimate-only. Dependency Proxy is group-level and supports Docker Hub images. Current docs state job tokens are masked and valid only while the job runs, but runner isolation remains critical. The lesson therefore treats runner security as part of token security rather than assuming masking prevents theft.

• Package registry — official reference.
• Generic packages — official reference.
• Supported package functionality — official reference.
• Protected packages — official reference.
• Reduce package registry storage — official reference.
• Container registry — official reference.
• Container registry authentication — official reference.
• Protected container tags — official reference.
• Immutable container tags — official reference.
• Reduce container registry storage — official reference.
• Dependency Proxy — official reference.
• CI job token — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.