SBOMs, SPDX/CycloneDX, in-toto Attestations, SLSA Provenance, Signatures, and Verification Workflows: Diagnostics, Failure Modes, Security, and Performance
Diagnose broken supply-chain evidence without hiding the first failure: wrong digest, stale tag, missing referrer, unsupported image store, unverified attestation, or obsolete Content Trust workflow.
Learning objectives
- Preserve the first failed verification and subject digest before changing tags, signatures, policies, or registry state.
- Diagnose subject mismatch, missing attestations, unsupported image stores, signer mismatch, and registry/referrer problems by layer.
- Recognize removed Docker Content Trust CLI workflows and replace them with explicit modern verification tooling rather than disabling checks.
- Explain why signatures and generated attestations are not self-authenticating policy decisions.
- Apply the least destructive correction and rerun only the smallest verification scope.
1. Failure model: preserve the evidence before “fixing” it
Supply-chain failures are easy to destroy accidentally. Retagging, rebuilding, re-signing, or copying artifacts can erase the state that explained why verification failed. Start by freezing the subject digest, raw manifest/index, tool versions, verifier command, stderr, and policy expectation.
date -u +%FT%TZ
docker version
docker buildx version
cosign version || true
printf '%s
' "$SUBJECT"
docker buildx imagetools inspect "$SUBJECT" --raw > first-failure-subject.json
# Run the failing verification exactly once and preserve stdout/stderr + exit code.
2. Evidence-first diagnostic sequence
| Layer | Question | Read-only evidence |
|---|---|---|
| Client/context | Am I asking the intended daemon/registry/tool? |
docker context show, versions, environment,
exact command.
|
| Build/image | Which immutable digest/platform was produced? | Build metadata, raw manifest/index, digest. |
| Attestation | Does required SBOM/provenance exist and name this subject? |
imagetools inspect --format, referrer
descriptors.
|
| Signature | What exact object was signed, by which identity? | Cosign verify output/bundle/public key identity. |
| Registry | Can it store/discover/copy the evidence? | Registry response/referrer list, artifact digests. |
| Policy | What did the verifier expect? | Policy version, expected signer/builder/source/predicate. |
| Runtime | Is the deployed digest the verified digest? | Deployment/container image digest evidence. |
3. Broken example A: sign a tag, discuss another digest
Symptom: a report says “app:release is
signed,” but the tag was repointed after signing. A later verifier
resolves it to a new digest and fails, or worse, a human assumes the
old signature applies.
Diagnosis: resolve and compare digests. Preserve
registry history/audit evidence if available. The fix is not
--check-claims=false; it is to sign and verify
immutable references and make the deployment policy digest-aware.
docker buildx imagetools inspect localhost:5005/dca30/app:release
# Compare the resolved digest with the subject recorded in the signature/evidence packet.
4. Broken example B: SBOM exists but belongs to another subject
Symptom: an exported sbom.json is
attached to a ticket for digest B, but its in-toto subject or
archival record names digest A.
Cause: evidence was copied by filename rather than resolved from the release subject.
Least-destructive correction: keep the mismatched SBOM as first-failure evidence, retrieve/generate the SBOM for digest B, then update the ticket with both the mismatch explanation and the corrected artifact. Never edit the subject field by hand to “make it match.”
5. Broken example C: attestations disappear after local load
Symptom: a registry image has provenance/SBOM, but after a build or transfer into a classic Docker image store, the local result does not expose them.
Reason: attestations attach to an image index.
Docker documents that the classic image store cannot hold these
indices/attestations. The docker build driver requires
the containerd image store for attestation support; other BuildKit
drivers can preserve them when pushing directly to a registry.
Correction: verify in the registry, use a supported image store/driver, or export evidence explicitly. Do not disable attestations merely to make an old local store accept the image unless policy explicitly permits the reduced evidence path.
6. Broken example D: valid signature, wrong signer
Symptom: cryptographic verification says a signature is structurally valid, but policy rejects it because the public key/certificate identity is not on the allowlist.
Interpretation: this is a policy success, not a crypto failure. Authentication asks “who signed?” Authorization asks “is that signer allowed for this release?”
7. Broken example E: old Docker Content Trust tutorial on Engine 29
Engine 29 removed Docker Content Trust commands from the Docker CLI.
Tutorials that tell you to run docker trust sign or
depend on CLI Notary v1 workflows are now legacy guidance. Docker
notes the trust command can be built as a separate plugin, but this
chapter does not revive it as the default supply-chain architecture.
Use current explicit tooling—such as BuildKit attestations plus Sigstore/Cosign or another reviewed signing/verification system—and document the identity/policy model. The migration is conceptual as well as syntactic: tags signed under old DCT workflows are not equivalent to digest-bound OCI attestation/signature policies.
8. Broken example F: generated provenance treated as self-authenticating
Symptom: policy checks only whether
.Provenance exists. An attacker who can publish to the
repository can also publish an image with self-generated provenance
claiming arbitrary build inputs.
Fix: authenticate the builder/evidence path. Require a trusted signer identity, signed provenance from an approved build platform, or a controlled registry/CI boundary whose guarantees are explicit. Existence checks are schema checks, not trust checks.
9. Security and performance boundaries
| Shortcut | Why it is dangerous | Evidence-based alternative |
|---|---|---|
| Disable signature verification | Turns integrity failure into silent acceptance | Keep failure; correct signer/subject/policy. |
| Accept any OIDC identity | Any valid certificate becomes authorized | Constrain certificate identity and issuer. |
| Use mutable-only tags | Evidence can drift between checks | Resolve/store digest and verify deployment digest. |
Publish mode=max everywhere |
May leak source/build details | Choose minimum claims that satisfy policy; review disclosure. |
| Rebuild/re-sign before preserving failure | Destroys causal evidence | Archive digest, manifest, referrers, verifier output first. |
| Assume zero CVEs from SBOM | SBOM is inventory, not vulnerability verdict | Run separate digest-bound scanning from Chapter 29. |
10. Smallest safe rerun
After correcting one layer, rerun only the corresponding verification first. If the signer was wrong, re-run signature verification before rebuilding the image. If provenance was missing because the output path dropped attestations, republish the same source with the corrected BuildKit exporter and compare digests/evidence. This reduces confounding changes.
Knowledge check
A Cosign signature is cryptographically valid, but the certificate identity is not approved. Should policy pass?
No. Cryptographic validity authenticates the signer; authorization still requires the expected identity/issuer or key.
Why should you not edit an SBOM subject field to match a different digest?
That fabricates evidence. Retrieve or generate evidence for the real subject and preserve the mismatch as diagnostic evidence.
Why can attestations disappear when using an incompatible local image store?
BuildKit attestations attach to an image index; Docker’s classic image store cannot retain that structure.
What happened to Docker Content Trust in Engine 29?
Docker Content Trust was removed from the Docker CLI. Current workflows should use explicit modern signing/attestation verification rather than copy legacy trust commands.
Policy only checks that provenance exists. What attack remains?
An unauthorized publisher could generate self-asserted provenance. Policy must authenticate the expected builder/evidence identity and claims.
Official references and version notes
Diagnostic principle: never “repair” a supply-chain verification failure by disabling verification, relaxing signer identity to a wildcard, replacing digests with mutable tags, or deleting the first-failure evidence.
- Docker Docs — Build attestations — current BuildKit SBOM/provenance model and image-store requirements.
-
Docker Docs — SBOM attestations
— SPDX/in-toto output and
imagetools inspectworkflow. -
Docker Docs — Provenance attestations
—
mode=min|maxand optional SLSA provenance v1. - Docker Docs — buildx imagetools inspect — registry manifest, SBOM, and provenance inspection.
- Docker Engine 29 release notes — Engine 29 compatibility baseline and removal of Docker Content Trust from the CLI.
-
OCI — Image and Distribution Specifications 1.1
— artifact
subject,artifactType, and Referrers API. - SPDX Specification 3.0.1 — current SPDX specification reference.
- CycloneDX Specification Overview — current CycloneDX 1.7 object model and media types.
- SLSA Specification 1.2 — current SLSA model and provenance guidance.
- Sigstore Cosign v3.1.3 — pinned local signing/verifying tool used as the 2026-09-22 lesson baseline.
-
Docker Official Image — registry
— disposable local registry; lesson baseline pins
registry:3.1.1.
Keep the academy open
Support free, practical DevOps education.
Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0Send only Ethereum/ERC-20 compatible assets to this
address.