Skip to content

[BUG] store copy / store serve registry fail with MANIFEST_BLOB_UNKNOWN for sigstore-bundle (cosign v3) attached images #605

@ymjing

Description

@ymjing

Environmental Info:

Linux ubuntu 6.17.0-23-generic #23~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 14 16:11:48 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Hauler Version:

GitVersion: v1.4.3
GitCommit: d5a56fd
GitTreeState: clean
BuildDate: 2026-05-05T05:27:30Z
GoVersion: go1.25.9 X:boringcrypto
Compiler: gc
Platform: linux/amd64

Describe the Bug:

hauler store copy and the auto-populate inside hauler store serve registry
both fail with MANIFEST_BLOB_UNKNOWN when the source store contains
images that have cosign v3 style sigstore-bundle artifacts attached
(media type application/vnd.dev.sigstore.bundle.v0.3+json).

The sigstore-bundle blob exists in the local store with a correct
digest, but the push to the destination registry commits the
sigstore-bundle's manifest before the layer blob is committed
remotely, so the registry rejects the manifest.

--exclude-extras on store sync does NOT prevent this — sigstore-bundle
artifacts get pulled in regardless.

Steps to Reproduce:

TEST_STORE=$(mktemp -d)
TEST_HAULERDIR=$(mktemp -d)
TEST_SERVE=$(mktemp -d)

cat > /tmp/test-istio.yaml <<'EOF'
---
apiVersion: content.hauler.cattle.io/v1
kind: Images
metadata:
  name: test-istio
spec:
  images:
    - name: docker.io/istio/pilot:1.29.2
EOF

hauler -d "$TEST_HAULERDIR" -s "$TEST_STORE" store sync -f /tmp/test-istio.yaml

hauler -d "$TEST_HAULERDIR" -s "$TEST_STORE" store serve registry \
    --port 5099 --directory "$TEST_SERVE"

The serve process logs:
Error: function execution failed: failed to upload manifest:
PUT http://127.0.0.1:36883/v2/istio/pilot/manifests/sha256:72c59d8b…:
MANIFEST_BLOB_UNKNOWN: blob unknown to registry;
sha256:6135d7dca89fd42d582952c8e349a274eab88410fd09cb9ed587e1bce13ec3f5

Expected Behavior:

Hauler should be able to handle istio/pilot.

Actual Behavior:

hauler store serve registry crashed and exited

Additional Context:

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    To Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions