Container Registry, OCI Images, Cleanup, Authentication, Vulnerability Scanning, and Retention: Diagnostics, Failure Modes, Security, and Performance

1. Diagnostic sequence: preserve evidence before changing registry state

Registry incidents become harder once tags are moved or deleted. Use the chapter-wide sequence: preserve evidence → identify host/project/ref/pipeline/job/runner/repository/tag/digest/deployment/policy scope → inspect permission and metadata → choose the least destructive correction → independently verify.

git rev-parse HEAD
glab ci get --output json | jq '{id,status,ref,sha} // .'
glab container-registry repository list --output json
# Then inspect the exact repository/tag with --details before mutation.

2. Failure: deployment uses a mutable tag that now points to another digest

Symptom: production says api:stable, but behavior changed without a deployment record. Preserve the runtime image ID/digest, then compare with current registry tag metadata and the release evidence.

glab container-registry tag view <repository-id> stable --output json > evidence/stable-now.json
jq '{name,digest,created_at} // .' evidence/stable-now.json

Repair: redeploy the known good digest, then change the release process to capture and deploy by digest. Do not “fix” history by force-moving Git tags or deleting registry evidence first.

3. Failure: registry login/push returns 401/403

Separate authentication from authorization. Check: registry host, token lifetime, username/token pairing, read_registry/write_registry scopes, project role, protected repository/tag rules, and whether the token was already revoked/expired.

4. Failure: registry credential has unnecessary write scope

A deployment host that only pulls should not possess write_registry. If exposure is suspected, revoke/rotate the persistent credential first, then replace it with a read-only identity. Cleanup of logs or variables comes after revocation because deleting traces does not invalidate a credential.

5. Failure: cleanup removes the rollback tag

Preserve deployment digest and registry metadata. A missing tag does not necessarily mean the manifest is already physically unavailable, but you must not rely on that accident. If the known digest remains pullable, restore a safe release reference only after authorization review. Then fix the retention rule so deployed/rollback-relevant releases are excluded from cleanup.

If storage is already reclaimed, recover from a trusted immutable artifact/registry replica or reproducibly rebuild only when policy allows—and record that rebuilt content may produce a different digest unless the build is truly reproducible.

6. Failure: vulnerability scan is associated with the wrong image

Symptom: a scan report looks clean, but it was produced for api:main while production runs another digest. Compare CS_IMAGE, pipeline SHA, report metadata/checksum, registry tag digest at scan time, and deployment digest.

Repair: scan the exact release image, persist digest-to-report mapping, and make release checks validate that identity. Never silence findings merely to make the wrong scan look acceptable.

7. Intentionally broken example: “cleanup succeeded, so storage is reclaimed”

Assume a project cleanup policy log shows tags removed, but Self-Managed disk usage is unchanged. The wrong conclusion is “cleanup is broken.” Tag deletion and blob garbage collection are separate.

# Project evidence: tags may be gone
glab container-registry tag list <repository-id> --details --output json

# Self-Managed administrator evidence (read-only concepts):
# - which registry metadata backend is active?
# - online GC queue/metrics when metadata DB is enabled
# - object-store/filesystem usage and registry logs

On current Self-Managed with the metadata database, online garbage collection processes unreferenced data asynchronously. Legacy storage has different offline procedures. Do not run third-party/offline GC against a metadata-database registry; current docs explicitly warn this can delete data associated with tagged images.

8. Self-Managed storage/garbage collection is an administrator boundary

GitLab 19.x prefers the registry metadata database for new Self-Managed installations; it enables online garbage collection and richer registry metadata. The database does not replace object storage/filesystem image data. Backups, migrations, GC health, storage consistency, and registry nodes are platform-administration concerns, not project pipeline steps.

In a production incident, project maintainers should preserve repository/tag/digest evidence and escalate storage/GC diagnostics to administrators rather than editing gitlab.rb from a CI job.

9. Performance symptoms: distinguish registry, runner, network, and image design

A slow pull can come from image size/layer churn, runner network, registry/object storage latency, authentication round trips, or cache misses. Measure before rewriting Dockerfiles or scaling runners. Large mutable images also increase scanning and transfer cost. Use layer-aware image design and digest evidence; do not trade away security controls merely to shave a few seconds.

10. High-frequency publishing creates tag/storage pressure

One tag per pipeline can be useful for traceability but creates lifecycle work. Standardize a tag namespace such as commit/review/release patterns, then write cleanup rules around that taxonomy. Keep release and deployed digests longer than ephemeral branch images. If GitLab.com cleanup runs only partially due to execution limits, verify successive runs instead of assuming one pass is exhaustive.

11. Minimum incident evidence pack

• GitLab host, project path/ID, registry repository ID/path;
• tag and current digest, plus historical/deployment digest if different;
• source commit, pipeline/job IDs, runner/executor context;
• credential type/scope/expiry identifier, never secret value;
• scan report checksum and image reference/digest;
• cleanup/protection policy state;
• for Self-Managed, registry backend/GC/storage observations supplied by administrators.

Knowledge check

Summary

Registry diagnostics are identity-first. Preserve the tag and digest before mutation, separate auth from authorization, keep credential incident response revoke-first, bind scans to image identity, and respect the boundary between project cleanup and Self-Managed storage garbage collection.

Official references

Primary sources used for the current GitLab 19.3 behavior taught in this lesson:

• GitLab Docs — Container Registry
• GitLab Docs — Authenticate with the Container Registry
• GitLab Docs — Reduce Container Registry storage
• GitLab Docs — Protected container repositories
• GitLab Docs — Protected container tags
• GitLab Docs — Immutable container tags
• GitLab Docs — Container Registry API
• GitLab Docs — Container scanning
• GitLab Docs — Container Registry metadata database
• GitLab Docs — Predefined CI/CD variables
• GitLab CLI — container-registry repository list
• GitLab CLI — container-registry tag list
• GitLab CLI — container-registry tag view
• GitLab CLI — container-registry tag delete