Chapter 30Lesson 04~165 minutes

SBOMs, SPDX/CycloneDX, in-toto Attestations, SLSA Provenance, Signatures, and Verification Workflows: Diagnostics, Failure Modes, Security, and Performance

Diagnose broken supply-chain evidence without hiding the first failure: wrong digest, stale tag, missing referrer, unsupported image store, unverified attestation, or obsolete Content Trust workflow.

DiagnosticsTrust boundariesDCT removalEvidence preservationSecurity

Learning objectives

  • Preserve the first failed verification and subject digest before changing tags, signatures, policies, or registry state.
  • Diagnose subject mismatch, missing attestations, unsupported image stores, signer mismatch, and registry/referrer problems by layer.
  • Recognize removed Docker Content Trust CLI workflows and replace them with explicit modern verification tooling rather than disabling checks.
  • Explain why signatures and generated attestations are not self-authenticating policy decisions.
  • Apply the least destructive correction and rerun only the smallest verification scope.

1. Failure model: preserve the evidence before “fixing” it

Supply-chain failures are easy to destroy accidentally. Retagging, rebuilding, re-signing, or copying artifacts can erase the state that explained why verification failed. Start by freezing the subject digest, raw manifest/index, tool versions, verifier command, stderr, and policy expectation.

date -u +%FT%TZ
docker version
docker buildx version
cosign version || true
printf '%s
' "$SUBJECT"
docker buildx imagetools inspect "$SUBJECT" --raw > first-failure-subject.json
# Run the failing verification exactly once and preserve stdout/stderr + exit code.

2. Evidence-first diagnostic sequence

Layer Question Read-only evidence
Client/context Am I asking the intended daemon/registry/tool? docker context show, versions, environment, exact command.
Build/image Which immutable digest/platform was produced? Build metadata, raw manifest/index, digest.
Attestation Does required SBOM/provenance exist and name this subject? imagetools inspect --format, referrer descriptors.
Signature What exact object was signed, by which identity? Cosign verify output/bundle/public key identity.
Registry Can it store/discover/copy the evidence? Registry response/referrer list, artifact digests.
Policy What did the verifier expect? Policy version, expected signer/builder/source/predicate.
Runtime Is the deployed digest the verified digest? Deployment/container image digest evidence.

3. Broken example A: sign a tag, discuss another digest

Symptom: a report says “app:release is signed,” but the tag was repointed after signing. A later verifier resolves it to a new digest and fails, or worse, a human assumes the old signature applies.

Diagnosis: resolve and compare digests. Preserve registry history/audit evidence if available. The fix is not --check-claims=false; it is to sign and verify immutable references and make the deployment policy digest-aware.

docker buildx imagetools inspect localhost:5005/dca30/app:release
# Compare the resolved digest with the subject recorded in the signature/evidence packet.

4. Broken example B: SBOM exists but belongs to another subject

Symptom: an exported sbom.json is attached to a ticket for digest B, but its in-toto subject or archival record names digest A.

Cause: evidence was copied by filename rather than resolved from the release subject.

Least-destructive correction: keep the mismatched SBOM as first-failure evidence, retrieve/generate the SBOM for digest B, then update the ticket with both the mismatch explanation and the corrected artifact. Never edit the subject field by hand to “make it match.”

5. Broken example C: attestations disappear after local load

Symptom: a registry image has provenance/SBOM, but after a build or transfer into a classic Docker image store, the local result does not expose them.

Reason: attestations attach to an image index. Docker documents that the classic image store cannot hold these indices/attestations. The docker build driver requires the containerd image store for attestation support; other BuildKit drivers can preserve them when pushing directly to a registry.

Correction: verify in the registry, use a supported image store/driver, or export evidence explicitly. Do not disable attestations merely to make an old local store accept the image unless policy explicitly permits the reduced evidence path.

6. Broken example D: valid signature, wrong signer

Symptom: cryptographic verification says a signature is structurally valid, but policy rejects it because the public key/certificate identity is not on the allowlist.

Interpretation: this is a policy success, not a crypto failure. Authentication asks “who signed?” Authorization asks “is that signer allowed for this release?”

Never widen the identity pattern just to make CI green. Preserve the unexpected identity, investigate why that workflow/key signed the digest, then either correct the signer or change policy through a reviewed process.

7. Broken example E: old Docker Content Trust tutorial on Engine 29

Engine 29 removed Docker Content Trust commands from the Docker CLI. Tutorials that tell you to run docker trust sign or depend on CLI Notary v1 workflows are now legacy guidance. Docker notes the trust command can be built as a separate plugin, but this chapter does not revive it as the default supply-chain architecture.

Use current explicit tooling—such as BuildKit attestations plus Sigstore/Cosign or another reviewed signing/verification system—and document the identity/policy model. The migration is conceptual as well as syntactic: tags signed under old DCT workflows are not equivalent to digest-bound OCI attestation/signature policies.

8. Broken example F: generated provenance treated as self-authenticating

Symptom: policy checks only whether .Provenance exists. An attacker who can publish to the repository can also publish an image with self-generated provenance claiming arbitrary build inputs.

Fix: authenticate the builder/evidence path. Require a trusted signer identity, signed provenance from an approved build platform, or a controlled registry/CI boundary whose guarantees are explicit. Existence checks are schema checks, not trust checks.

9. Security and performance boundaries

Shortcut Why it is dangerous Evidence-based alternative
Disable signature verification Turns integrity failure into silent acceptance Keep failure; correct signer/subject/policy.
Accept any OIDC identity Any valid certificate becomes authorized Constrain certificate identity and issuer.
Use mutable-only tags Evidence can drift between checks Resolve/store digest and verify deployment digest.
Publish mode=max everywhere May leak source/build details Choose minimum claims that satisfy policy; review disclosure.
Rebuild/re-sign before preserving failure Destroys causal evidence Archive digest, manifest, referrers, verifier output first.
Assume zero CVEs from SBOM SBOM is inventory, not vulnerability verdict Run separate digest-bound scanning from Chapter 29.

10. Smallest safe rerun

After correcting one layer, rerun only the corresponding verification first. If the signer was wrong, re-run signature verification before rebuilding the image. If provenance was missing because the output path dropped attestations, republish the same source with the corrected BuildKit exporter and compare digests/evidence. This reduces confounding changes.

Next

Assemble the checkpoint evidence packet

Lesson 5 turns these diagnostics into a reproducible release gate with one expected pass and two intentional failures.

Knowledge check

A Cosign signature is cryptographically valid, but the certificate identity is not approved. Should policy pass?

Why should you not edit an SBOM subject field to match a different digest?

Why can attestations disappear when using an incompatible local image store?

What happened to Docker Content Trust in Engine 29?

Policy only checks that provenance exists. What attack remains?

Official references and version notes

Diagnostic principle: never “repair” a supply-chain verification failure by disabling verification, relaxing signer identity to a wildcard, replacing digests with mutable tags, or deleting the first-failure evidence.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0Send only Ethereum/ERC-20 compatible assets to this address.