Environmental Info:
Linux ubuntu 6.17.0-23-generic #23~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 14 16:11:48 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Hauler Version:
GitVersion: v1.4.3
GitCommit: d5a56fd
GitTreeState: clean
BuildDate: 2026-05-05T05:27:30Z
GoVersion: go1.25.9 X:boringcrypto
Compiler: gc
Platform: linux/amd64
Describe the Bug:
hauler store copy and the auto-populate inside hauler store serve registry
both fail with MANIFEST_BLOB_UNKNOWN when the source store contains
images that have cosign v3 style sigstore-bundle artifacts attached
(media type application/vnd.dev.sigstore.bundle.v0.3+json).
The sigstore-bundle blob exists in the local store with a correct
digest, but the push to the destination registry commits the
sigstore-bundle's manifest before the layer blob is committed
remotely, so the registry rejects the manifest.
--exclude-extras on store sync does NOT prevent this — sigstore-bundle
artifacts get pulled in regardless.
Steps to Reproduce:
TEST_STORE=$(mktemp -d)
TEST_HAULERDIR=$(mktemp -d)
TEST_SERVE=$(mktemp -d)
cat > /tmp/test-istio.yaml <<'EOF'
---
apiVersion: content.hauler.cattle.io/v1
kind: Images
metadata:
name: test-istio
spec:
images:
- name: docker.io/istio/pilot:1.29.2
EOF
hauler -d "$TEST_HAULERDIR" -s "$TEST_STORE" store sync -f /tmp/test-istio.yaml
hauler -d "$TEST_HAULERDIR" -s "$TEST_STORE" store serve registry \
--port 5099 --directory "$TEST_SERVE"
The serve process logs:
Error: function execution failed: failed to upload manifest:
PUT http://127.0.0.1:36883/v2/istio/pilot/manifests/sha256:72c59d8b…:
MANIFEST_BLOB_UNKNOWN: blob unknown to registry;
sha256:6135d7dca89fd42d582952c8e349a274eab88410fd09cb9ed587e1bce13ec3f5
Expected Behavior:
Hauler should be able to handle istio/pilot.
Actual Behavior:
hauler store serve registry crashed and exited
Additional Context:
Environmental Info:
Linux ubuntu 6.17.0-23-generic #23~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 14 16:11:48 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Hauler Version:
GitVersion: v1.4.3
GitCommit: d5a56fd
GitTreeState: clean
BuildDate: 2026-05-05T05:27:30Z
GoVersion: go1.25.9 X:boringcrypto
Compiler: gc
Platform: linux/amd64
Describe the Bug:
hauler store copyand the auto-populate insidehauler store serve registryboth fail with
MANIFEST_BLOB_UNKNOWNwhen the source store containsimages that have cosign v3 style sigstore-bundle artifacts attached
(media type
application/vnd.dev.sigstore.bundle.v0.3+json).The sigstore-bundle blob exists in the local store with a correct
digest, but the push to the destination registry commits the
sigstore-bundle's manifest before the layer blob is committed
remotely, so the registry rejects the manifest.
--exclude-extrasonstore syncdoes NOT prevent this — sigstore-bundleartifacts get pulled in regardless.
Steps to Reproduce:
The serve process logs:
Error: function execution failed: failed to upload manifest:
PUT http://127.0.0.1:36883/v2/istio/pilot/manifests/sha256:72c59d8b…:
MANIFEST_BLOB_UNKNOWN: blob unknown to registry;
sha256:6135d7dca89fd42d582952c8e349a274eab88410fd09cb9ed587e1bce13ec3f5
Expected Behavior:
Hauler should be able to handle istio/pilot.
Actual Behavior:
hauler store serve registrycrashed and exitedAdditional Context: