`cosign load` fails to upload v3 bundle layer blobs before manifest PUT (BLOB_UPLOAD_UNKNOWN)

[phene]

Description

cosign load fails when loading sigstore signed bundles to a registry where the bundle layer blob does not already exist. The command uploads the image index and legacy signature successfully, then attempts to PUT the bundle manifest without first uploading the bundle layer blob. Registries that enforce blob-before-manifest ordering (e.g. AWS ECR) return BLOB_UPLOAD_UNKNOWN.

This breaks cross-registry image replication workflows that use cosign save → cosign load (e.g. commercial → isolated/partitioned registry, cross-account ECR, air-gapped mirrors).

Environment

Component	Version
cosign	v3.1.2 (GitVersion, darwin/arm64)
go-containerregistry	v0.21.7
Registry (source)	Any OCI v2 registry with a dual-signed image
Registry (destination)	Any OCI v2 registry where bundle layer blobs are not already present (fresh repo, different account/region, etc.)

Observed on: AWS ECR (commercial us-east-1 → GovCloud us-gov-west-1, cross-account). The destination registry does accept v3 bundle artifacts when layer blobs are uploaded correctly — this is not a registry limitation.

Debug tip: COSIGN_DEBUG=1 produces no additional output in cosign v3. Use -d / --verbose (and optionally --output-file).

Steps to reproduce

Prerequisites: cosign v3.x, an OCI registry you can push to (source), and a second registry/repo where bundle blobs do not yet exist (destination). Any two registries work — ECR, GCR, registry:2, Harbor, etc.

1. Create a dual-signed image on the source registry.

   With cosign v3 default signing (bundle format enabled), sign a container image so it has a v3 bundle referrer:

   cosign sign --key cosign.key "${SOURCE_REGISTRY}/${REPO}:${TAG}"

   Confirm both signature formats exist (exact inspection method depends on registry; on ECR, look for a referrer with artifactMediaType application/vnd.dev.sigstore.bundle.v0.3+json in addition to the legacy sha256-<digest>.sig tag).

2. Save the image and signatures to a local OCI layout.

   WORKDIR=$(mktemp -d)
   cosign save --dir "$WORKDIR" "${SOURCE_REGISTRY}/${REPO}:${TAG}"

   Verify the layout contains bundle artifacts:

   # At least one manifest should reference sigstore.bundle media types
   grep -rl 'sigstore.bundle' "$WORKDIR/blobs/sha256/" | while read -r m; do
     echo "=== $m ==="
     jq '.layers[].digest, .artifactType' "$m"
   done

3. Load to a destination registry where bundle layer blobs are absent.

   Use a fresh destination repository, a different AWS account, or any registry that does not already contain the bundle layer digest:

   cosign load -d --dir "$WORKDIR" "${DEST_REGISTRY}/${REPO}:${TAG}"

4. Observe failure on the bundle manifest PUT with BLOB_UPLOAD_UNKNOWN.

Minimal reproduction with two local registries (optional)

If you do not have two cloud registries, run two registry:2 instances on different ports:

docker run -d -p 5000:5000 --name reg-src registry:2
docker run -d -p 5001:5001 -e REGISTRY_HTTP_ADDR=0.0.0.0:5001 --name reg-dst registry:2

# push + sign on :5000, save, load to :5001

The bug manifests whenever destination :5001 lacks the bundle layer blob.

Actual behavior

cosign load fails with:

Error: failed to upload manifest: PUT https://<dest-registry>/v2/<repo>/manifests/sha256:<bundle-manifest-digest>: BLOB_UPLOAD_UNKNOWN: Layers with digests '[sha256:<bundle-layer-digest>]' required for pushing image into repository with name '<repo>' in the registry with id '<account>' do not exist

Verbose output (cosign load -d) shows:

1. Image index manifest PUT → 201 Created
2. No HEAD, POST /blobs/uploads/, or PUT for the bundle layer blob
3. Bundle manifest PUT attempted directly → 404 with BLOB_UPLOAD_UNKNOWN

Example excerpt from cosign load -d (cosign v3.1.2, go-containerregistry v0.21.7):

2026/07/30 16:03:34 --> PUT .../manifests/cosign-gov-debug-20260730160022
2026/07/30 16:03:34 <-- 201 Created
...
2026/07/30 16:03:35 --> HEAD .../blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
2026/07/30 16:03:35 <-- 200 OK
2026/07/30 16:03:35 --> PUT .../manifests/sha256:dda90b82e1a6ec6e5d1a1c405f0f75548b4116a3c5028f0a0c48935a8712ac1e
2026/07/30 16:03:35 <-- 404 Not Found

{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"Layers with digests '[sha256:fbb75ba3f55494dc670d6bd5fcaabdbd46a0ca26b7c2d1594bb52d5f73ace64c]' ... do not exist"}]}

Note: cosign HEADs the bundle manifest's empty config blob (sha256:44136fa3…, size 2) but never checks or uploads the bundle layer blob (sha256:fbb75ba3…, 1255 bytes, media type application/vnd.dev.sigstore.bundle.v0.3+json).

The bundle manifest being pushed:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "artifactType": "application/vnd.dev.sigstore.bundle.v0.3+json",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "size": 2,
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
  },
  "layers": [{
    "mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
    "size": 1255,
    "digest": "sha256:fbb75ba3f55494dc670d6bd5fcaabdbd46a0ca26b7c2d1594bb52d5f73ace64c"
  }],
  "subject": {
    "digest": "sha256:ef637aabc052029acfd192d8ed6674a5249a8783e551c08db71e321ca11b421d"
  }
}

Why same-registry load can succeed: If the destination already contains the bundle layer blob (e.g. load within the same account where blobs were previously pushed, or a prior partial upload), the manifest PUT succeeds and the bug is masked.

Legacy-only images are unaffected: Images signed with only the legacy .sig format (no bundle referrer) load successfully.

Expected behavior

cosign load should upload all blobs referenced by bundle manifests in the OCI layout — including bundle layer blobs — before PUTting the bundle manifest, consistent with how container image layers and legacy signature blobs are handled.

After a successful load:

• cosign verify against the destination image should succeed
• Bundle referrers should be present on the destination registry with correct subject linkage

Workaround (verified)

Before cosign load, manually upload missing bundle layer blobs from the OCI layout via the registry API:

1. Scan $WORKDIR/blobs/sha256/* for manifests with artifactType or layer mediaType containing sigstore.bundle
2. For each layer digest in those manifests:
   • HEAD /v2/{repo}/blobs/sha256:{layer} — skip if 200
   • POST /v2/{repo}/blobs/uploads/ then PUT with ?digest=sha256:{layer}
3. Run cosign load as normal → succeeds; cosign verify passes on destination

This confirms the registry accepts the bundle format; the defect is in cosign's load upload ordering for bundle artifacts.

Suggested fix

In the cosign load code path (likely via go-containerregistry's layout → remote write), ensure bundle referrer manifests trigger upload of their layer blobs before the manifest PUT — mirroring behavior for standard image manifests and legacy .sig signatures.

Minimal change: when iterating artifacts in the saved layout, if a manifest is a Sigstore bundle (application/vnd.dev.sigstore.bundle.*), call blob upload for each referenced layer digest before pushing the manifest.

---

Version

$ cosign version
  GitVersion:    v3.1.2
  GitCommit:     193d2153431f8bb0d945a4c1ee721872f73add67
  GitTreeState:  "clean"
  BuildDate:     2026-07-17T14:32:20Z
  GoVersion:     go1.26.5
  Compiler:      gc
  Platform:      darwin/arm64

  go-containerregistry: v0.21.7 (from verbose HTTP User-Agent)

[ogulcanaydogan]

Picking this up. Great writeup, the repro and HTTP trace made this easy to confirm. Will dig into the load path and check whether bundle referrer layer blobs are getting skipped in the upload loop.

[phene]

@ogulcanaydogan why not just proceed with the fix I already submitted?

[ogulcanaydogan]

Fair point, apologies, should have checked for an existing PR before jumping in. Closed mine (#5065) in favor of #5031, yours has the same fix plus a cross-registry e2e test I didn't think to write. Hope it gets reviewed soon, it's been sitting for a few weeks.