Release Automation, Tags, GitHub Releases, Packages, and Container Registries: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

1. Preserve run ID/attempt, first failure logs, event/ref/SHA and workflow revision.
2. Confirm version/tag target and whether that name already exists.
3. Confirm exact built subject digest and whether promotion changed bytes.
4. Confirm evaluated job permissions and which code runs with release/package authority.
5. Inspect artifact transfer, attestation result and release/package API response.
6. Inspect external registry/package state by immutable ID/digest rather than alias alone.
7. Apply the smallest correction, then rerun only the equivalent scope while preserving the original record.

2. Failure: building from a mutable or wrong source ref

# Broken release idea — DO NOT USE
on:
  workflow_dispatch:
jobs:
  release:
    steps:
      - run: git clone https://github.com/example/app && cd app && git checkout main
      - run: ./build-and-publish.sh

This workflow does not prove which commit was built. The branch can move after dispatch, and the clone is detached from the event-selected repository revision. Repair by selecting and recording the exact source SHA, checking out that SHA, then carrying the digest of the resulting artifact across publication.

3. Failure: same version, different bytes

Suppose version 1.4.2 already exists with digest A, but a retry rebuilds and produces digest B. Overwriting the registry or release asset destroys the meaning of the version. The correct investigation asks why bytes changed—toolchain, dependency, timestamp, source, build flags—and either reuse digest A or publish a new version after validation.

sha256sum dist/package.tgz
gh release view v1.4.2 --json assets,tagName,targetCommitish
# For OCI, inspect the immutable digest instead of only the tag:
docker buildx imagetools inspect ghcr.io/example/app:v1.4.2

4. Failure: token is broader than the failing operation

A 403 while creating a Release is not evidence that the workflow needs write-all. Confirm the API endpoint and required permission. GitHub Release/tag mutation belongs to contents: write; GHCR/package publication belongs to packages: write. A third-party registry may require a different credential, but that credential should be scoped to one disposable namespace and never printed.

5. Failure: attesting before final bytes exist

If a workflow creates an attestation for package.zip and then modifies, recompresses or signs the file, the published digest no longer matches the attested subject. Preserve the original digest/attestation as first-failure evidence, finalize the bytes, compute the final digest, then create a new attestation for the exact published subject.

6. Failure: publishing latest from pull requests

PR code is untrusted validation input, not release authorization. A workflow that gives PR jobs packages: write and pushes latest lets ordinary test activity mutate consumer-facing state. Keep PR validation read-only; publish only from a separately guarded tag/manual/release path after exact source verification.

7. Failure: deleting evidence before diagnosis

A failed release may have created a tag but no release, a draft release but no assets, one asset but no package, or a package digest but no deployment. Deleting everything immediately removes clues. Record each existing object ID and digest first, then clean only the partial disposable resources that are safe to remove.

8. Causal layer map

9. Intentionally broken example: hidden failure behind mutable alias

# Broken operational check
image='ghcr.io/example/app:latest'
docker pull "$image"
echo 'release verified'   # proves only that something called latest was pullable

Interpretation: the check never compares the pulled manifest digest with the digest the release workflow recorded. Repair by storing the expected digest as release evidence, resolving the registry reference, and comparing exact digests before deployment or promotion.

10. Least-destructive recovery patterns

• Wrong tag target: stop before publication; create a new correct unique tag rather than force-moving a released immutable tag.
• Wrong asset bytes: preserve digest; fix build; publish a new version or replace only a still-draft disposable release before public consumption.
• Partial prerelease: inspect release/tag/assets first, then delete the exact lab objects if policy allows.
• Bad production version: redeploy a known-good existing digest and publish a superseding fix; do not rewrite history.
• Credential leak: revoke/rotate the credential at its provider, then investigate published state; deleting logs alone is not remediation.

11. Lesson summary

Release debugging is identity debugging plus side-effect accounting. Preserve the exact source, artifact digest and external object IDs, then repair the layer that diverged.

Knowledge check

Official references and version notes

• About releases — GitHub Releases are based on Git tags and add release metadata/assets around a versioned source point.
• Managing releases — Current release creation, editing and deletion behavior.
• Immutable releases — Current tag/asset immutability and automatic release-attestation behavior.
• GHCR — Current GitHub Container Registry authentication, package linking and digest guidance.
• GitHub Packages permissions — Repository/granular package permissions and GitHub Actions access.
• GITHUB_TOKEN authentication — Least-privilege token use in workflows.
• gh release create — Current CLI release creation, --verify-tag and immutable-release handling.
• gh release verify — Verification of cryptographically signed immutable-release attestations.
• gh release verify-asset — Verify a local asset against the release attestation and digest.
• actions/attest v4.2.2 — Immutable attestation action revision referenced by optional provenance examples.