GitHub Packages, Container Registry, Package Permissions, Provenance, and Distribution: Diagnostics, Failure Modes, Security, and Performance
Package incidents are often misdiagnosed because registry authentication, package authorization, namespace resolution, mutable tags, retention, and provenance fail independently. This lesson uses preserved evidence and intentionally broken cases to identify which boundary actually failed before changing credentials or deleting versions.
Learning objectives
- Diagnose publish-versus-consume authorization failures without broadening tokens blindly.
- Detect tag drift by comparing requested label with immutable registry digest.
- Separate wrong hostname/namespace errors from token permission errors.
- Recover safely from package deletion/retention mistakes while preserving evidence.
- Recognize the absence of provenance rather than inferring it from package hosting.
Safety rule. All destructive examples are documentation/fixtures unless explicitly performed on the Chapter 21 disposable package. Never delete a supported production package, broaden a registry token, make an internal package public, or move a consumed release label as a troubleshooting shortcut.
1. Diagnostic sequence: preserve identity before changing anything
- Preserve evidence: registry hostname, namespace, tag/digest, workflow run ID/SHA, exact stderr/HTTP status, package page, version ID, and current access list.
- Identify scope: personal/organization package, linked repository, publisher repository, consumer repository, workflow/job permissions, package visibility.
-
Inspect authorization:
GITHUB_TOKENpackagespermission plus package-side Actions access/inheritance. - Inspect identity: resolve tag to digest; compare with the digest recorded at release/deployment.
- Inspect lifecycle/provenance: version deletion state, supported-release policy, attestation presence.
- Choose the least destructive correction and independently verify with a clean pull/API read.
2. Failure: publisher succeeds but downstream repository cannot read
Symptom: producer workflow pushes successfully,
while a consumer with a valid GitHub identity receives
403, denied, or package not found for a
private package.
Likely cause: the publisher repository has package
access, but the consumer repository was never added under Manage
Actions access, or its job lacks packages: read.
# Consumer-side policy is necessary but not sufficient:
permissions:
contents: read
packages: read
# Package settings must ALSO grant this repository read access.
Repair: grant only that repository read access (or
deliberately enable inheritance if repository linkage matches
policy). Do not replace the consumer’s
GITHUB_TOKEN with a broad shared PAT merely because the
first pull failed.
3. Failure: mutable tag delivers different bytes
Version 1.0.0 was approved at digest D1, but a consumer
later pulls :latest and receives D2. The
registry is working as designed: the tag moved.
IMAGE="ghcr.io/OWNER/github-packages-lab"
docker pull "$IMAGE:latest"
docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE:latest"
# Compare the digest after @ with the release/deployment record.
Repair: pull/deploy
$IMAGE@$KNOWN_GOOD_DIGEST. If a semantic tag was
promised immutable and moved, treat that as a governance incident;
do not conceal it by rewriting audit records.
4. Failure: registry hostname or namespace is wrong
Three strings must agree: registry host (ghcr.io on
GitHub.com), owner namespace, and package name. A typo can look like
an authentication failure because registries intentionally avoid
revealing private resource existence.
| Evidence | Likely interpretation | Next inspection |
|---|---|---|
unauthorized before any manifest lookup |
No/invalid registry credential or unsupported token type | Login method, token lifetime/scope; do not print token |
denied on push after successful pull |
Authenticated but missing write authorization |
Job packages permission + package Actions
access
|
manifest unknown / not found |
Wrong tag/digest/package path or deleted version | Package page + exact namespace/version |
| GitHub.com workflow uses enterprise/custom registry hostname | Deployment mismatch |
Use ghcr.io for GitHub.com; enterprise host
differs
|
5. Intentionally broken example: read-only token tries to publish
The Lesson 2 package-denial.yml workflow is
deliberately broken at the authorization boundary. It logs in
successfully, pulls latest, then pushes
denied-probe while packages: read is the
only package permission.
permissions:
contents: read
packages: read
# Later:
docker push "$IMAGE:denied-probe" # expected non-zero exit
Interpret the original error before editing YAML. If authentication
succeeded and pull succeeded, the cause is not “Docker login is
broken.” The denied write is evidence that least privilege is
working. The repair belongs only in the publisher job: grant
packages: write there. Keep consumers read-only.
6. Failure: cleanup deletes a supported package/version
Deletion can break build/deploy reproducibility even when source remains in Git. Preserve the package version ID, tag/digest mapping, release/deployment references, and deletion timestamp. GitHub can normally restore a deleted package/version within 30 days if the namespace/version has not been reused.
Least destructive correction: restore the deleted version rather than rebuild old source into a new digest and pretend it is the original package. If restore is impossible, publish a new version and explicitly document the changed identity.
7. Failure: team assumes provenance because the package page links to source
A source link is useful metadata, not a signed build claim. A digest proves content identity, not build origin. Ask: Is there an attestation for this subject name and digest, and has the consumer verified it against the expected repository/workflow policy?
If the answer is no, the correction is not “trust the GitHub logo.” Record provenance as unavailable and add an attestation/verification requirement in the supply-chain chapter.
8. Credential leakage around package tooling
Never troubleshoot by running docker login with a token
literal, uploading ~/.docker/config.json, caching
home-directory credentials, or printing package-manager config
containing auth tokens. If a registry credential is exposed,
revoke/rotate first. Remove it from logs/artifacts/config second.
On GitHub-hosted runners, still call
docker logout ghcr.io so the lab documents proper
lifecycle; on persistent self-hosted runners, credential residue is
an even stronger reason to isolate and clean the host.
9. Reliability, performance, and billing only where they cause the failure
Large layers can make pushes slow or time out; current GHCR documentation lists a 10 GB limit per layer and a 10-minute upload timeout. Private Packages can also hit plan quota/budget controls. Diagnose those conditions from the actual error and billing/usage page instead of repeatedly retrying non-idempotent publish operations.
For public GHCR in this chapter, the synthetic image is intentionally tiny, so bandwidth/layer limits should not be a cause. If they appear, you are probably operating on the wrong package or workflow.
10. Compact operator runbook
1. Record: host / namespace / package / tag / digest / run ID / source SHA.
2. Record exact Docker/API stderr + exit/HTTP code.
3. Confirm package visibility and linked repository.
4. Confirm job packages permission (read/write) and package Actions access.
5. Resolve requested tag to actual digest; compare release/deployment record.
6. Check version exists and was not deleted/retained out.
7. Check provenance independently; do not infer it from hosting.
8. Correct the smallest boundary; verify with a clean pull by digest.Knowledge check
A consumer logs in successfully but receives denied on pull of a private package. What should you inspect before creating a PAT?
Inspect packages:read on the job and whether the consumer repository has package-side Actions read access/inheritance. Authentication success does not prove package authorization.
Why is “latest changed” not a registry corruption diagnosis?
Tags are mutable selectors. Compare the resolved digest with the previously recorded digest; the content may be valid but a different version.
What is the safest first response after deleting a supported package version accidentally?
Preserve identity/evidence and attempt restore within GitHub’s restore window if the namespace is still available, rather than rebuilding and pretending the new digest is the old artifact.
What evidence distinguishes digest integrity from provenance?
The digest identifies content; an attestation/verified build claim links that digest to source/workflow identity.
Why can a wrong package path return an authorization-looking error?
Registries may avoid disclosing private resource existence, so hostname/namespace/tag resolution must be checked independently from credentials.
Summary
Package failures become tractable when you keep registry identity,
token permission, package ACL, mutable tags, lifecycle state, and
provenance separate. Preserve the exact digest/error first; then
repair only the boundary that failed. The checkpoint now applies
this model across two real versions, a moving
latest label, a denied write, and destructive cleanup
policy.
Official references
Keep the academy open
Support free, practical DevOps education.
Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0
Send only Ethereum/ERC-20 compatible assets to this
address.