Registries, Docker Hub, Private Registries, Authentication, Credential Stores, Push/Pull, and Distribution: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Preserve the exact failing command, timestamp, registry reference, status/error, push/pull digest output, and relevant logs.
2. Confirm Docker host/platform, docker version, docker info, and docker context show.
3. Confirm daemon/API availability before blaming the registry.
4. Confirm the exact source/build/image digest you intended to publish or retrieve.
5. Inspect local image/container state if the symptom appears after pull.
6. Inspect registry DNS/TLS/network reachability.
7. Inspect authentication and repository authorization separately.
8. Inspect registry content/policy/storage only after identity and transport are proven.
9. Apply the smallest reversible correction and rerun only the failed scope.

2. Failure map: classify the layer before changing anything

3. Intentionally broken example: password on the command line

The following is intentionally wrong and should not be executed with a real credential:

# WRONG: secret can land in shell history / process inspection
docker login registry.example.com -u ci-publisher -p REAL_SECRET

The correction changes only credential delivery:

printf '%s\n' "$REGISTRY_PASSWORD" \
  | docker login registry.example.com \
      --username ci-publisher --password-stdin

Then verify that logs do not echo the variable and that the configured helper/store owns persistence. This fixes secret exposure; it does not grant missing repository permissions.

4. Intentionally broken example: certificate failure is not an invitation to disable trust

If push returns an x509 trust error, preserve the exact error, registry hostname, certificate SANs, issuing CA, and daemon platform. The repair is normally to present a correct server certificate and place the issuing CA where the Docker daemon expects it for that platform. Docker Desktop, native Linux Engine, rootless Engine, and Windows Engine have different trust locations.

5. Authentication success but push denied: inspect token scope

A token can authenticate correctly and still lack push for team/app. In a Distribution token flow, preserve the WWW-Authenticate challenge without copying secret tokens. Check the requested repository and action scope. If pull works but push fails, that asymmetry is useful evidence.

Repair authorization at the registry/provider policy layer. Do not grant a broader organization-wide credential merely because one repository scope is wrong.

6. Overwriting a tag without recording the digest destroys release clarity

If stable was pushed twice and only the tag is in the incident ticket, you may not know which bytes a consumer obtained earlier. Preserve push logs, registry digest, and consumer RepoDigest before moving the tag again. The correction is a release process that records the digest at push/promotion time.

7. Debug logging can become a credential leak

Do not paste raw HTTP traces containing Authorization: Bearer ..., Basic auth values, cookies, or signed URLs into tickets/chat. Prefer registry server request IDs, status codes, repository names, non-secret scopes, and redacted headers. If a token is accidentally exposed, treat it as compromised and revoke/rotate according to provider policy.

8. Shared blobs: never delete backend files to fix a single image

Content-addressed blobs can be referenced by many manifests. Direct filesystem/object-store deletion bypasses registry reference knowledge and can corrupt several images. Use supported deletion/retention mechanisms and administrator-controlled garbage collection. For CNCF Distribution, current GC guidance recommends read-only/stopped state during collection and offers a dry-run mode.

9. Performance diagnosis: locate transfer, registry, or storage bottlenecks

Push progress shows uncompressed layer size while network transfer is compressed, so the progress bars are not a direct wire-byte meter. Reused blobs can make subsequent pushes dramatically smaller. Diagnose performance using layer reuse, daemon concurrency settings, registry logs, backend latency/throughput, network path, and client/server resource saturation before altering global upload/download concurrency.

10. Safe failure exercise: wrong repository path, then smallest correction

Against the disposable Chapter 14 registry, intentionally request a nonexistent repository by digest-like workflow:

REGISTRY=127.0.0.1:5000
docker pull "$REGISTRY/devops-academy/does-not-exist:1.0.0" \
  2>&1 | tee "$EVIDENCE/expected-manifest-unknown.log" || true

docker logs da-ch14-registry 2>&1 | tail -n 60 \
  | tee "$EVIDENCE/registry-after-bad-pull.log"

Interpret the error first. The registry is reachable; the path is simply absent. The repair is to use the intended repository/reference — not restart Docker, not disable TLS, and not prune images.

11. Recovery checklist

Knowledge check

Official references and version notes

• docker login — authentication, --password-stdin, Docker Desktop native keychains, credsStore, and per-registry credHelpers.
• docker image push — repository naming, layer reuse, push progress, and digest output.
• docker image pull — pull-by-tag versus pull-by-digest and registry-qualified references.
• Registry certificates — platform-specific CA/client-certificate configuration and TLS verification.
• Docker daemon: insecure registries — why plaintext/untrusted registries are testing-only and why trusted CA configuration is preferred.
• Docker Hub pull usage and limits — current rate-limit behavior and authentication attribution.
• CNCF Distribution Registry — current Registry v3 documentation and local-registry workflow.
• Registry v2 token authentication — 401 challenge, token service, repository scope, Bearer token, and retry flow.
• Distribution HTTP API V2 — manifests, blobs, uploads, errors, and digest-addressed operations.
• Deploying Distribution Registry — local development registry, production TLS/auth expectations, and storage.
• Registry garbage collection — shared blobs, reference-aware deletion, mark/sweep behavior, and read-only guidance.
• OCI Distribution Specification v1.1.1 — latest OCI Distribution Specification release used as the standards baseline.
• Distribution Registry v3.1.1 — current stable registry release used as the local-lab baseline.
• Docker Engine 29 release notes — Engine 29.8.1 baseline used when authoring this chapter.
• Buildx releases, BuildKit releases, and Compose releases — current build/orchestration tool release history.