Docker image failing to push with many abandoned multipart uploads (r2 storage backend) #3227

Nikola-Milovic · 2025-06-26T06:29:43Z

Nikola-Milovic
Jun 26, 2025

Hey!

There are a few moving parts in the setup so I am not 100% certain which one could be the culprit, any nudge in what direction I could look into would be amazing.

Issue

Docker image pushes are failing with "bad range format: 0--1" errors. The registry uses Cloudflare R2 for storage backend and is accessed through Cloudflare Tunnel. Push operations start but never complete, leaving multiple orphaned upload sessions in R2.

Environment Details

Infrastructure

Zot Registry Server: Running on Hetzner, cpx11 instance, 2vCPU, 2GB RAM (could it be too small?)
Storage Backend: Cloudflare R2 (S3-compatible)
Registry Version: v2.1.4
Access Method: Through Cloudflare Tunnel
Client: Docker 27.5.1 on Linux

Docker Image Details

$ docker inspect registry.frmlabz.com/image-embedding-nvidia:latest
[
    {
        "Id": "sha256:028c4184976196afcae10e9ed0c09d30e913c27cadf00948c690bce555e34f18",
        "RepoTags": [
            "registry.frmlabz.com/image-embedding-nvidia:latest"
        ],
        "RepoDigests": [],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "1980-01-01T00:00:00Z",
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8001/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/.local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NVIDIA_VISIBLE_DEVICES=all",
                "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
                "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
                "PYTORCH_VERSION=2.7.1"
            ],
            "Cmd": [
                "uvicorn",
                "src.imagembedding.image_embedding:app",
                "--host",
                "0.0.0.0",
                "--port",
                "8001"
            ],
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/app",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.nvidia.volumes.needed": "nvidia_driver",
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.version": "22.04"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 6839595069,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/wiu1e0pvrtjygezkyaq05cks5/diff:/var/lib/docker/overlay2/re77qnwnzt497v9hw0du6lp5a/diff:/var/lib/docker/overlay2/wp9iiv6iufc5dghowhdnscdhs/diff:/var/lib/docker/overlay2/ahy7pfotoj5dyequf6i3oh9ex/diff:/var/lib/docker/overlay2/r07qnyjj06dmi3chytaxixyak/diff:/var/lib/docker/overlay2/7e7afc30abad7eb15fba024894b1cf26bedb3be589ccaed4ed3d3bef50e522f7/diff:/var/lib/docker/overlay2/17a7c06af9b94c260a6128ef85003bcdc2077ae018a4dabf3cbe5d23c211ff29/diff:/var/lib/docker/overlay2/949bcf62adf7b1ef1a6cf9da9a6b38725e4f8a6b92c23ad88dba3b00597d0bb9/diff:/var/lib/docker/overlay2/dddb8137d2bce9da625c9b43656babb2ebe2fd4f534e4e94a4c2419e4b778724/diff:/var/lib/docker/overlay2/99f96f12adb695966f1245a83546f53c9c6f74502590c2e1b5dacd6947fae5d1/diff",
                "MergedDir": "/var/lib/docker/overlay2/3x1ehnmv6ezb7pwx768c36033/merged",
                "UpperDir": "/var/lib/docker/overlay2/3x1ehnmv6ezb7pwx768c36033/diff",
                "WorkDir": "/var/lib/docker/overlay2/3x1ehnmv6ezb7pwx768c36033/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f862e1968e4b4c3c3af141e37d2ec22b19ec0fd50d6a8aaf683de6729e296226",
                "sha256:cc3692b9aab9a9e24fa1610d9f4c6f95a47d06516b5879b89eaeb863b23d37f5",
                "sha256:88060d9cbb0a4e15cf464067784a92d1c349134193113543debf3ea770970e9e",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:fd81be54cfae6890bff58d7703bbc557df5f7f551987025b1767230b2bd1c95f",
                "sha256:fc7bb2f1007745629bb5ebea16018f45c3b37b436730afe5e8e91634f396275e",
                "sha256:b425abfb98d9855cda16eccf467f3332e659224fe07eec7dd71c108b342bb312",
                "sha256:abdcb92661cc8cb55ca0a67e273b1508a1f8a1fd1337932c3ba41d2233a91872",
                "sha256:00f57e9afedd3296c9353eb995892f76124a8613b54303158ae8e24bfd2f9083",
                "sha256:612b5ca85f477786f8deddb480e4a8773b77bc169e788a042b9bba7a3a252374",
                "sha256:401a2a5f865bc93d95c49881ed52d170c242754a6b5ad9a0ca007730249b0f35"
            ]
        },
        "Metadata": {
            "LastTagTime": "2025-06-25T14:19:08.587296444+02:00"
        }
    }
]

Zot Configuration (`/etc/zot/config.json`)

{
  "distSpecVersion": "1.1.0",
  "storage": {
    "dedupe": false,
    "gc": true,
    "gcDelay": "1h",
    "gcInterval": "24h",
    "rootDirectory": "/tmp/zot/",
    "storageDriver": {
      "name": "s3",
      "rootdirectory": "/zot",
      "region": "auto",
      "regionendpoint": "https://***.r2.cloudflarestorage.com",
      "bucket": "zot-registry",
      "secure": true,
      "skipverify": false,
      "accesskey": "[REDACTED]",
      "secretkey": "[REDACTED]"
    }
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000",
    "realm": "zot",
    "compat": ["docker2s2"],
    "auth": {
      "htpasswd": {
        "path": "/etc/zot/htpasswd"
      },
      "failDelay": 5
    }
  },
  "log": {
    "level": "debug",
    "output": "/var/log/zot/zot.log",
    "audit": "/var/log/zot/zot-audit.log"
  },
  "extensions": {
    "search": { "enable": true },
    "ui": { "enable": true },
    "mgmt": { "enable": true },
    "metrics": {
      "enable": true,
      "prometheus": { "path": "/metrics" }
    },
    "scrub": {
      "enable": true,
      "interval": "24h"
    }
  }
}

Observed Behavior

1. Push Operation Patterns

Multiple POST requests to /v2/image-embedding-nvidia/blobs/uploads/ creating new upload sessions
PATCH requests with very small or zero content lengths (0 bytes, 32 bytes, 93 bytes, 99 bytes)
HEAD requests returning 404 for blobs that should be uploaded
Upload sessions appear to stall and never complete

2. Incomplete Multipart Uploads in R2

After attempting to push the image-embedding-nvidia image, the following incomplete multipart uploads remain in R2:

[2025-06-26 07:17:04 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/05a76443-eb2f-46d1-a222-790da2b03131
[2025-06-26 07:23:54 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/1354a17e-34a5-4ffa-8bdb-d0b0075c8d15
[2025-06-26 07:17:10 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/1ac62b08-146d-4529-a2b4-00dc7246de3b
[2025-06-26 07:26:49 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/1ddc977e-39d4-4b14-8f5a-c2f5add5a279
[2025-06-26 07:17:10 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/210a8e51-c9dc-4f66-944a-9dca64d8abbb
[2025-06-26 07:23:59 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/251f05b0-11a9-4d86-83cb-4cb99061ba98
[2025-06-26 07:24:09 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/28042581-9fa9-4f25-b9a2-b530bcfe2d5d
[2025-06-26 07:17:21 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/284c27d8-b906-4c4d-806a-cd29a298cf50
[2025-06-26 07:26:48 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/2c1342f0-b8d0-4526-83ab-f219d1b96984
[2025-06-26 07:17:05 CEST] 0B zot/zot/image-embedding-nvidia/.uploads/3078ef5d-1af1-4b50-859e-3a15a8747a9e
... (48 more incomplete uploads)

Logs

{"level":"debug","component":"scheduler","worker":4,"task":"{Name: DedupeTask, digest: sha256:b3062ee8280a61b11ea01b14d22b8568cc234b565f791cee65d77b64752a6225, dedupe: false}","goroutine":458,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:18.958539602Z","message":"finished task"}
{"level":"debug","component":"scheduler","worker":6,"task":"{taskGenerator: \"image scrub\", repo: \"example-app\"}","goroutine":460,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:19.381582781Z","message":"starting task"}
{"level":"info","goroutine":460,"caller":"zotregistry.dev/zot/pkg/extensions/scrub/scrub.go:19","time":"2025-06-26T06:28:19.381697063Z","message":"executing scrub to check manifest/blob integrity for /zot/example-app"}
{"level":"debug","component":"scheduler","worker":7,"task":"{Name: DedupeTask, digest: sha256:ba1bbeb9846957516e95442e8dce3e98dfb0273ada2d02bd6c39d243bd30619a, dedupe: false}","goroutine":461,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:19.489932243Z","message":"starting task"}
{"level":"debug","component":"scheduler","worker":8,"task":"{Name: DedupeTask, digest: sha256:c2744d03542b33847b2c942c73e7b6e16d424ebe159a2ac79748649add5fbf54, dedupe: false}","goroutine":462,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:19.602219356Z","message":"starting task"}
{"level":"info","digest":"sha256:ba1bbeb9846957516e95442e8dce3e98dfb0273ada2d02bd6c39d243bd30619a","component":"dedupe","goroutine":461,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2091","time":"2025-06-26T06:28:19.602236688Z","message":"restoring deduped blobs for digest"}
{"level":"warn","component":"dedupe","goroutine":461,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1997","time":"2025-06-26T06:28:19.602416745Z","message":"failed to find blob in cache, searching it in storage..."}
{"level":"info","originalBlob":"/zot/eb4640c/blobs/sha256/ba1bbeb9846957516e95442e8dce3e98dfb0273ada2d02bd6c39d243bd30619a","component":"dedupe","goroutine":461,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2007","time":"2025-06-26T06:28:19.666458271Z","message":"found original blob"}
{"level":"info","digest":"sha256:ba1bbeb9846957516e95442e8dce3e98dfb0273ada2d02bd6c39d243bd30619a","component":"dedupe","goroutine":461,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2132","time":"2025-06-26T06:28:19.730242878Z","message":"restoring deduped blobs for digest finished successfully"}
{"level":"debug","component":"scheduler","worker":7,"task":"{Name: DedupeTask, digest: sha256:ba1bbeb9846957516e95442e8dce3e98dfb0273ada2d02bd6c39d243bd30619a, dedupe: false}","goroutine":461,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:19.730324379Z","message":"finished task"}
{"level":"info","module":"http","component":"session","clientIP":"[::1]:48696","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip, br"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4d77cd5d0e3-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:19.854747528Z","message":"HTTP API"}
{"level":"debug","component":"scheduler","worker":1,"task":"{Name: DedupeTask, digest: sha256:d69d4d41cfe2ee680d6972795e2a1eb9e4dc4ec3b3c5e0797c9ab43bb3726fa7, dedupe: false}","goroutine":455,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:19.910385378Z","message":"starting task"}
{"level":"info","identity":"admin","goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:19.965165824Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":669,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:19.969840119Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:19.974479531Z","message":"user profile successfully set"}
{"level":"info","digest":"sha256:c2744d03542b33847b2c942c73e7b6e16d424ebe159a2ac79748649add5fbf54","component":"dedupe","goroutine":462,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2091","time":"2025-06-26T06:28:19.997305324Z","message":"restoring deduped blobs for digest"}
{"level":"warn","component":"dedupe","goroutine":462,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1997","time":"2025-06-26T06:28:19.997537466Z","message":"failed to find blob in cache, searching it in storage..."}
{"level":"info","identity":"admin","goroutine":689,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:20.034210109Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":691,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:20.040182649Z","message":"user profile successfully set"}
{"level":"info","originalBlob":"/zot/eb4640c/blobs/sha256/c2744d03542b33847b2c942c73e7b6e16d424ebe159a2ac79748649add5fbf54","component":"dedupe","goroutine":462,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2007","time":"2025-06-26T06:28:20.057843239Z","message":"found original blob"}
{"level":"info","digest":"sha256:c2744d03542b33847b2c942c73e7b6e16d424ebe159a2ac79748649add5fbf54","component":"dedupe","goroutine":462,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2132","time":"2025-06-26T06:28:20.183454898Z","message":"restoring deduped blobs for digest finished successfully"}
{"level":"debug","component":"scheduler","worker":8,"task":"{Name: DedupeTask, digest: sha256:c2744d03542b33847b2c942c73e7b6e16d424ebe159a2ac79748649add5fbf54, dedupe: false}","goroutine":462,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:20.183534226Z","message":"finished task"}
{"level":"debug","component":"scheduler","worker":2,"task":"{Name: DedupeTask, digest: sha256:f0d4e9d9da8e95eb21d272601ab64c4d5828aa2ddac2bb201dcce1eb02630aa0, dedupe: false}","goroutine":456,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:20.312429304Z","message":"starting task"}
{"level":"debug","blob path":"/zot/image-embedding-nvidia/blobs/sha256/ce6d405461f8ebe0ca2bd4ef03c8e084dbe2e482b3e3bbdf9d2b51c67299c774","goroutine":664,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1338","time":"2025-06-26T06:28:20.323710694Z","message":"blob path found"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48696","method":"HEAD","path":"/v2/image-embedding-nvidia/blobs/sha256:ce6d405461f8ebe0ca2bd4ef03c8e084dbe2e482b3e3bbdf9d2b51c67299c774","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4d88861b69b-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:20.32386918Z","message":"HTTP API"}
{"level":"debug","blob path":"/zot/image-embedding-nvidia/blobs/sha256/eed6a6318a6cecb77518a13ac98fa927c9f8a65a0dc3121ce8795d7f3e102757","goroutine":671,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1338","time":"2025-06-26T06:28:20.365900908Z","message":"blob path found"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48726","method":"HEAD","path":"/v2/image-embedding-nvidia/blobs/sha256:eed6a6318a6cecb77518a13ac98fa927c9f8a65a0dc3121ce8795d7f3e102757","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4d8898cd0dc-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:20.366098507Z","message":"HTTP API"}
{"level":"info","digest":"sha256:d69d4d41cfe2ee680d6972795e2a1eb9e4dc4ec3b3c5e0797c9ab43bb3726fa7","component":"dedupe","goroutine":455,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2091","time":"2025-06-26T06:28:20.366321213Z","message":"restoring deduped blobs for digest"}
{"level":"warn","component":"dedupe","goroutine":455,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1997","time":"2025-06-26T06:28:20.36645391Z","message":"failed to find blob in cache, searching it in storage..."}
{"level":"info","originalBlob":"/zot/eb4640c/blobs/sha256/d69d4d41cfe2ee680d6972795e2a1eb9e4dc4ec3b3c5e0797c9ab43bb3726fa7","component":"dedupe","goroutine":455,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2007","time":"2025-06-26T06:28:20.428471488Z","message":"found original blob"}
{"level":"info","identity":"admin","goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:20.472615009Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:20.47792484Z","message":"user profile successfully set"}
{"level":"info","digest":"sha256:d69d4d41cfe2ee680d6972795e2a1eb9e4dc4ec3b3c5e0797c9ab43bb3726fa7","component":"dedupe","goroutine":455,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2132","time":"2025-06-26T06:28:20.605715666Z","message":"restoring deduped blobs for digest finished successfully"}
{"level":"debug","component":"scheduler","worker":1,"task":"{Name: DedupeTask, digest: sha256:d69d4d41cfe2ee680d6972795e2a1eb9e4dc4ec3b3c5e0797c9ab43bb3726fa7, dedupe: false}","goroutine":455,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:20.605836692Z","message":"finished task"}
{"level":"debug","blob path":"/zot/image-embedding-nvidia/blobs/sha256/01c183a1cbcdd4616bbe68cf75ce03cfc6db58f8b1ca7174f6320834b971af68","goroutine":671,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1338","time":"2025-06-26T06:28:20.724913902Z","message":"blob path found"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48726","method":"HEAD","path":"/v2/image-embedding-nvidia/blobs/sha256:01c183a1cbcdd4616bbe68cf75ce03cfc6db58f8b1ca7174f6320834b971af68","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4dbbca1d0e3-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:20.725278883Z","message":"HTTP API"}
{"level":"debug","component":"scheduler","worker":3,"task":"{Name: DedupeTask, digest: sha256:f8abf93e7b9ad7a19f831931c046b2a4a9cb50a9656cb57dd6f47c381f5f47a1, dedupe: false}","goroutine":457,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:20.738057538Z","message":"starting task"}
{"level":"debug","blob path":"/zot/image-embedding-nvidia/blobs/sha256/265d1044cdcd4e43029bfd4709a4de9d8caf591a98f7ab973fe38fb6ce113707","goroutine":664,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1338","time":"2025-06-26T06:28:20.76624414Z","message":"blob path found"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48696","method":"HEAD","path":"/v2/image-embedding-nvidia/blobs/sha256:265d1044cdcd4e43029bfd4709a4de9d8caf591a98f7ab973fe38fb6ce113707","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4db5b44e292-BEG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:20.766403296Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:20.842307909Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:20.878429563Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48710","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"1s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4d88df3d0e0-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":669,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:21.036166364Z","message":"HTTP API"}
{"level":"warn","error":"cache miss","digest":"sha256:ab224c3cf399b42d4c9dfa9bdaa36275478cd38d2738fc03bdd45f8231e4dcf0","goroutine":671,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1354","time":"2025-06-26T06:28:21.051310616Z","message":"not found in cache"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48726","method":"HEAD","path":"/v2/image-embedding-nvidia/blobs/sha256:ab224c3cf399b42d4c9dfa9bdaa36275478cd38d2738fc03bdd45f8231e4dcf0","statusCode":404,"latency":"0s","bodySize":405,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4de3a9d1c2f-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:21.051513554Z","message":"HTTP API"}
{"level":"debug","blob path":"/zot/image-embedding-nvidia/blobs/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1","goroutine":664,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1338","time":"2025-06-26T06:28:21.067201922Z","message":"blob path found"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48696","method":"HEAD","path":"/v2/image-embedding-nvidia/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4de0cea89c3-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:21.06735014Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:21.175371299Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:21.181874599Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":669,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:21.206251538Z","message":"user profile successfully set"}
{"level":"debug","component":"scheduler","worker":4,"task":"{Name: DedupeTask, digest: sha256:fc5203bd3420563f66506c7d595b05655e19d6a88a7e02a850f9c33ca9d225f6, dedupe: false}","goroutine":458,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:21.234075472Z","message":"starting task"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48742","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"1s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4d88f4fd0ee-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":689,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:21.477393971Z","message":"HTTP API"}
{"level":"warn","error":"cache miss","digest":"sha256:8a0c3e162a807f339f885fe8cd62f60ea94d64745c74a483a8c1e419f360cb3e","goroutine":671,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1354","time":"2025-06-26T06:28:21.491266339Z","message":"not found in cache"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48726","method":"HEAD","path":"/v2/image-embedding-nvidia/blobs/sha256:8a0c3e162a807f339f885fe8cd62f60ea94d64745c74a483a8c1e419f360cb3e","statusCode":404,"latency":"0s","bodySize":405,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4e02cf7d0cb-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:21.491377756Z","message":"HTTP API"}
{"level":"debug","component":"scheduler","worker":7,"task":"{Name: DedupeTask, digest: sha256:fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236, dedupe: false}","goroutine":461,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:21.512177757Z","message":"starting task"}
{"level":"info","identity":"admin","goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:21.609458987Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":689,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:21.713930311Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48756","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"1s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4d89bdbd0fb-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":691,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:21.939075734Z","message":"HTTP API"}
{"level":"debug","component":"scheduler","worker":8,"task":"{Name: DedupeTask, digest: sha256:035af8781fb431096e247e6e41dfe4248237cd4d133be66901bb4547b8179023, dedupe: false}","goroutine":462,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:21.993284855Z","message":"starting task"}
{"level":"info","digest":"sha256:f0d4e9d9da8e95eb21d272601ab64c4d5828aa2ddac2bb201dcce1eb02630aa0","component":"dedupe","goroutine":456,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2091","time":"2025-06-26T06:28:21.993327976Z","message":"restoring deduped blobs for digest"}
{"level":"warn","component":"dedupe","goroutine":456,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1997","time":"2025-06-26T06:28:21.993879725Z","message":"failed to find blob in cache, searching it in storage..."}
{"level":"info","originalBlob":"/zot/eb4640c/blobs/sha256/f0d4e9d9da8e95eb21d272601ab64c4d5828aa2ddac2bb201dcce1eb02630aa0","component":"dedupe","goroutine":456,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2007","time":"2025-06-26T06:28:22.054384849Z","message":"found original blob"}
{"level":"info","digest":"sha256:f0d4e9d9da8e95eb21d272601ab64c4d5828aa2ddac2bb201dcce1eb02630aa0","component":"dedupe","goroutine":456,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2132","time":"2025-06-26T06:28:22.112304506Z","message":"restoring deduped blobs for digest finished successfully"}
{"level":"debug","component":"scheduler","worker":2,"task":"{Name: DedupeTask, digest: sha256:f0d4e9d9da8e95eb21d272601ab64c4d5828aa2ddac2bb201dcce1eb02630aa0, dedupe: false}","goroutine":456,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:22.112374917Z","message":"finished task"}
{"level":"info","identity":"admin","goroutine":691,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:22.163431573Z","message":"user profile successfully set"}
{"level":"debug","component":"scheduler","worker":1,"task":"{Name: DedupeTask, digest: sha256:09e1189f65845fb91c3ce1d7ddb8f1f198f057ada0d4ebfd185a189b900504b3, dedupe: false}","goroutine":455,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:22.26707482Z","message":"starting task"}
{"level":"info","digest":"sha256:f8abf93e7b9ad7a19f831931c046b2a4a9cb50a9656cb57dd6f47c381f5f47a1","component":"dedupe","goroutine":457,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2091","time":"2025-06-26T06:28:22.382902603Z","message":"restoring deduped blobs for digest"}
{"level":"warn","component":"dedupe","goroutine":457,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1997","time":"2025-06-26T06:28:22.383065868Z","message":"failed to find blob in cache, searching it in storage..."}
{"level":"info","originalBlob":"/zot/eb4640c/blobs/sha256/f8abf93e7b9ad7a19f831931c046b2a4a9cb50a9656cb57dd6f47c381f5f47a1","component":"dedupe","goroutine":457,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2007","time":"2025-06-26T06:28:22.450685263Z","message":"found original blob"}
{"level":"info","digest":"sha256:f8abf93e7b9ad7a19f831931c046b2a4a9cb50a9656cb57dd6f47c381f5f47a1","component":"dedupe","goroutine":457,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2132","time":"2025-06-26T06:28:22.576675348Z","message":"restoring deduped blobs for digest finished successfully"}
{"level":"debug","component":"scheduler","worker":3,"task":"{Name: DedupeTask, digest: sha256:f8abf93e7b9ad7a19f831931c046b2a4a9cb50a9656cb57dd6f47c381f5f47a1, dedupe: false}","goroutine":457,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:22.576741552Z","message":"finished task"}
{"level":"debug","component":"scheduler","worker":2,"task":"{Name: DedupeTask, digest: sha256:09fffc1a7f354c68aa57ba12a676ef5246941b41f3631efc140932578f97d0cf, dedupe: false}","goroutine":456,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:22.688324871Z","message":"starting task"}
{"level":"info","image":"eb4640c","tag":"","status":"ok","component":"scrub","goroutine":459,"caller":"zotregistry.dev/zot/pkg/extensions/scrub/scrub.go:37","time":"2025-06-26T06:28:23.282325604Z","message":"blobs/manifest ok"}
{"level":"info","image":"eb4640c","tag":"","status":"ok","component":"scrub","goroutine":459,"caller":"zotregistry.dev/zot/pkg/extensions/scrub/scrub.go:37","time":"2025-06-26T06:28:23.282378342Z","message":"blobs/manifest ok"}
{"level":"info","image":"eb4640c","tag":"","status":"ok","component":"scrub","goroutine":459,"caller":"zotregistry.dev/zot/pkg/extensions/scrub/scrub.go:37","time":"2025-06-26T06:28:23.282395053Z","message":"blobs/manifest ok"}
{"level":"info","image":"eb4640c","tag":"","status":"ok","component":"scrub","goroutine":459,"caller":"zotregistry.dev/zot/pkg/extensions/scrub/scrub.go:37","time":"2025-06-26T06:28:23.282540946Z","message":"blobs/manifest ok"}
{"level":"info","image":"eb4640c","tag":"","status":"ok","component":"scrub","goroutine":459,"caller":"zotregistry.dev/zot/pkg/extensions/scrub/scrub.go:37","time":"2025-06-26T06:28:23.28256935Z","message":"blobs/manifest ok"}
{"level":"info","goroutine":459,"caller":"zotregistry.dev/zot/pkg/extensions/scrub/scrub.go:50","time":"2025-06-26T06:28:23.282587574Z","message":"scrub successfully completed for /zot/eb4640c"}
{"level":"debug","component":"scheduler","worker":5,"task":"{taskGenerator: \"image scrub\", repo: \"eb4640c\"}","goroutine":459,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:23.282613932Z","message":"finished task"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48696","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"2s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4e00f020c7f-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":664,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:23.538171218Z","message":"HTTP API"}
{"level":"debug","component":"scheduler","worker":3,"task":"{Name: DedupeTask, digest: sha256:10b4e60c32ad074a006dee01b2ac19a21dd26e41889b0f8ffe0cdeace0803f9e, dedupe: false}","goroutine":457,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:23.540228068Z","message":"starting task"}
{"level":"debug","component":"scheduler","worker":5,"task":"{Name: DedupeTask, digest: sha256:1610cf4810097396dbce76b18fb57f9af51136e84f13d936648bcdb69efab5dd, dedupe: false}","goroutine":459,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:24.422571003Z","message":"starting task"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48710","method":"PATCH","path":"/v2/image-embedding-nvidia/blobs/uploads/f4acade1-841d-4829-b5a4-54a26e15d6cb","statusCode":202,"latency":"3s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4dffa26d0ce-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":669,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:24.425659979Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":669,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:24.460566434Z","message":"user profile successfully set"}
{"level":"info","digest":"sha256:fc5203bd3420563f66506c7d595b05655e19d6a88a7e02a850f9c33ca9d225f6","component":"dedupe","goroutine":458,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2091","time":"2025-06-26T06:28:24.624241772Z","message":"restoring deduped blobs for digest"}
{"level":"warn","component":"dedupe","goroutine":458,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1997","time":"2025-06-26T06:28:24.62439073Z","message":"failed to find blob in cache, searching it in storage..."}
{"level":"info","originalBlob":"/zot/eb4640c/blobs/sha256/fc5203bd3420563f66506c7d595b05655e19d6a88a7e02a850f9c33ca9d225f6","component":"dedupe","goroutine":458,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2007","time":"2025-06-26T06:28:24.687705068Z","message":"found original blob"}
{"level":"info","digest":"sha256:fc5203bd3420563f66506c7d595b05655e19d6a88a7e02a850f9c33ca9d225f6","component":"dedupe","goroutine":458,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2132","time":"2025-06-26T06:28:24.818898898Z","message":"restoring deduped blobs for digest finished successfully"}
{"level":"debug","component":"scheduler","worker":4,"task":"{Name: DedupeTask, digest: sha256:fc5203bd3420563f66506c7d595b05655e19d6a88a7e02a850f9c33ca9d225f6, dedupe: false}","goroutine":458,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:24.818989026Z","message":"finished task"}
{"level":"debug","component":"scheduler","worker":4,"task":"{Name: DedupeTask, digest: sha256:18933b04a6910c939115116c75bbfc4e04669cc970f4c29f3d290b7df5e0f553, dedupe: false}","goroutine":458,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:24.930807705Z","message":"starting task"}
{"level":"info","digest":"sha256:fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236","component":"dedupe","goroutine":461,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2091","time":"2025-06-26T06:28:25.095040254Z","message":"restoring deduped blobs for digest"}
{"level":"warn","component":"dedupe","goroutine":461,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1997","time":"2025-06-26T06:28:25.095347258Z","message":"failed to find blob in cache, searching it in storage..."}
{"level":"info","originalBlob":"/zot/eb4640c/blobs/sha256/fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236","component":"dedupe","goroutine":461,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2007","time":"2025-06-26T06:28:25.16216219Z","message":"found original blob"}
{"level":"info","digest":"sha256:fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236","component":"dedupe","goroutine":461,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2132","time":"2025-06-26T06:28:25.352526262Z","message":"restoring deduped blobs for digest finished successfully"}
{"level":"debug","component":"scheduler","worker":7,"task":"{Name: DedupeTask, digest: sha256:fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236, dedupe: false}","goroutine":461,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:25.352676541Z","message":"finished task"}
{"level":"debug","component":"scheduler","worker":7,"task":"{Name: DedupeTask, digest: sha256:190db352f50c98b2fc29af4c2a7171ee095801d088fb67b053ec7a478bd80aa4, dedupe: false}","goroutine":461,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:25.480055891Z","message":"starting task"}
{"level":"debug","component":"scheduler","generator":"GCTaskGenerator","goroutine":466,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:311","time":"2025-06-26T06:28:26.225666664Z","message":"waiting generator is ready, pushing to ready generators"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48726","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"4s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4e2cc3d1ca0-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:26.272755613Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":671,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-26T06:28:26.441383096Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:48742","method":"PATCH","path":"/v2/image-embedding-nvidia/blobs/uploads/94180346-a8ea-4d22-91a7-0f713b16b2f5","statusCode":202,"latency":"4s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["955aa4e2be52d0ca-SOF"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":689,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-26T06:28:26.65382308Z","message":"HTTP API"}
{"level":"info","digest":"sha256:035af8781fb431096e247e6e41dfe4248237cd4d133be66901bb4547b8179023","component":"dedupe","goroutine":462,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2091","time":"2025-06-26T06:28:27.252403201Z","message":"restoring deduped blobs for digest"}
{"level":"warn","component":"dedupe","goroutine":462,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1997","time":"2025-06-26T06:28:27.252508518Z","message":"failed to find blob in cache, searching it in storage..."}
{"level":"info","originalBlob":"/zot/example-app/blobs/sha256/035af8781fb431096e247e6e41dfe4248237cd4d133be66901bb4547b8179023","component":"dedupe","goroutine":462,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2007","time":"2025-06-26T06:28:27.315984577Z","message":"found original blob"}
{"level":"info","digest":"sha256:035af8781fb431096e247e6e41dfe4248237cd4d133be66901bb4547b8179023","component":"dedupe","goroutine":462,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:2132","time":"2025-06-26T06:28:27.376749677Z","message":"restoring deduped blobs for digest finished successfully"}
{"level":"debug","component":"scheduler","worker":8,"task":"{Name: DedupeTask, digest: sha256:035af8781fb431096e247e6e41dfe4248237cd4d133be66901bb4547b8179023, dedupe: false}","goroutine":462,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:157","time":"2025-06-26T06:28:27.376825438Z","message":"finished task"}
{"level":"debug","component":"scheduler","worker":8,"task":"{Name: DedupeTask, digest: sha256:19f88d70ebe83a1955c94043d3f88fe959a6723a4c758b5f7d61236c50fc48f9, dedupe: false}","goroutine":462,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:133","time":"2025-06-26T06:28:27.376856617Z","message":"starting task"}

Docker daemon logs on the client

Jun 26 07:46:02 workstation dockerd[2526]: time="2025-06-26T07:46:02.742405465+02:00" level=error msg="Upload failed, retrying: bad range format: 0--1"
Jun 26 07:46:08 workstation dockerd[2526]: time="2025-06-26T07:46:08.389004450+02:00" level=error msg="Upload failed, retrying: bad range format: 0--1"
Jun 26 07:46:08 workstation dockerd[2526]: time="2025-06-26T07:46:08.527792343+02:00" level=error msg="Upload failed, retrying: bad range format: 0--1"
Jun 26 07:46:08 workstation dockerd[2526]: time="2025-06-26T07:46:08.972173614+02:00" level=error msg="Upload failed, retrying: bad range format: 0--1"
Jun 26 07:46:11 workstation dockerd[2526]: time="2025-06-26T07:46:11.373523960+02:00" level=error msg="Upload failed, retrying: bad range format: 0--1"

Key Observations from Logs

Cache Miss Warnings: Normal for new blob uploads but indicate blobs aren't being stored properly
Zero or Small Content Lengths: PATCH requests show minimal data transfer
Repeated Upload Sessions: Multiple POST requests create new upload sessions instead of completing existing ones

Reproduction Steps

Configure Zot registry with Cloudflare R2 backend
Access registry through Cloudflare proxy
Attempt to push a Docker image: docker push registry.example.com/image-embedding-nvidia:latest
Observe incomplete uploads accumulating in R2 bucket

Workarounds Attempted

Cleaned up all incomplete multipart uploads from R2
Restarted Zot registry service
Verified S3 credentials and connectivity

Cloudflare limits docs

I am suspecting that it might be

Needs more than 2GB RAM?
Image size is around 6.7GB maybe that is causing issues on some end?
Too small of a chunk size for uploads?
Some tunnel or r2 limit that I am hitting?

But before changing my whole setup, I'd like to get second opinion, since even with debug logs I haven't gotten very far

rchincha · 2025-06-26T16:02:28Z

rchincha
Jun 26, 2025
Maintainer

https://windix.medium.com/self-hosted-docker-registry-with-cloudflare-r2-s3-compatible-as-backend-3028cec33555
^ found this pointer. Helpful?

5 replies

Nikola-Milovic Jun 27, 2025
Author

EDIT: potentionally related to the cloudflare tunnel, since I tried the docker registry and it has the exact same issue.

Thanks, I updated my config to include those parameters

But still getting the same issue

{
  "distSpecVersion": "1.1.0",
  "storage": {
    "dedupe": false,
    "gc": true,
    "gcDelay": "1h",
    "gcInterval": "24h",
    "rootDirectory": "/tmp/zot/",
    "storageDriver": {
      "name": "s3",
      "rootdirectory": "/",
      "region": "auto",
      "regionendpoint": "https://xxxx.r2.cloudflarestorage.com",
      "bucket": "zot-registry",
      "secure": true,
      "skipverify": false,
      "accesskey": "xxxx",
      "secretkey": "xxxx",
      "forcepathstyle": true,
      "chunksize": "104857600",
      "multipartcopychunksize": "33554432",
      "multipartcopymaxconcurrency": "100",
      "multipartcopythresholdsize": "33554432"
    }
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000",
    "realm": "zot",
    "compat": ["docker2s2"],
    "auth": {
      "htpasswd": {
        "path": "/etc/zot/htpasswd"
      },
      "failDelay": 5
    }
  },
  "log": {
    "level": "debug",
    "output": "/var/log/zot/zot.log",
    "audit": "/var/log/zot/zot-audit.log"
  },
  "extensions": {
   "search": {
      "enable": true
   },
    "ui": {
      "enable": true
    },
    "mgmt": {
      "enable": true
    },
    "metrics": {
      "enable": true,
      "prometheus": {
        "path": "/metrics"
      }
    },
    "scrub": {
                        "enable": true,
      "interval": "24h"
    }
  }
}

b425abfb98d9: Retrying in 1 second
fc7bb2f10077: Pushed
fd81be54cfae: Pushed
5f70bf18a086: Pushed
88060d9cbb0a: Retrying in 6 seconds
cc3692b9aab9: Retrying in 8 seconds
f862e1968e4b: Waiting
bad range format: 0--1

,"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:39.121845362Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:39.310371879Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47818","method":"PATCH","path":"/v2/image-embedding-nvidia/blobs/uploads/037940fc-cbbe-4f79-b6cf-b611604ca3dc","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ecf20a42bde7-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:39.54191237Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:41.20061441Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47818","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ecfe5f92f96e-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:41.425745523Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:41.630485855Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47818","method":"PATCH","path":"/v2/image-embedding-nvidia/blobs/uploads/f4d55548-bfe7-4765-b55f-acc89ea9cb46","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ed00784bf988-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:41.868482149Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:48.386234214Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47818","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ed2b2f92b36c-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:48.619505306Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:48.734524677Z","message":"user profile successfully set"}
{"level":"info","identity":"admin","goroutine":76,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:48.835910113Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47818","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ed2d5879710d-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:48.969155804Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:49.092766241Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47830","method":"PATCH","path":"/v2/image-embedding-nvidia/blobs/uploads/920d4e9c-617a-42fd-ac78-f563c2d993de","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ed2d7901c63c-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":76,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:49.159701129Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":76,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:49.171610777Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47818","method":"POST","path":"/v2/image-embedding-nvidia/blobs/uploads/","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ed2f9f41b391-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":60,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:49.369035668Z","message":"HTTP API"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47830","method":"PATCH","path":"/v2/image-embedding-nvidia/blobs/uploads/621ecce0-791b-4a71-94e1-599a537ce785","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ed2f9d8980e8-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":76,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:49.567212239Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":76,"caller":"zotregistry.dev/zot/pkg/api/authn.go:142","time":"2025-06-27T06:35:49.573247128Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"[::1]:47830","method":"PATCH","path":"/v2/image-embedding-nvidia/blobs/uploads/d8ceb129-2f7e-49e5-9ca1-b36e7a68304b","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip, br"],"Authorization":["******"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["93.86.166.189"],"Cf-Ipcountry":["RS"],"Cf-Ray":["9562ed321e5036f5-PRG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["0469b5e2-21e8-43b2-bebb-28e3076484eb"],"Connection":["keep-alive"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["docker/27.5.1 go/go1.24.3 git-commit/v27.5.1 kernel/6.15.3 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.1 \\(linux\\))"],"X-Forwarded-For":["93.86.166.189"],"X-Forwarded-Proto":["https"]},"goroutine":76,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-06-27T06:35:49.892729529Z","message":"HTTP API"}

I'll debug some more, it only fails on these larger images

rchincha Jun 27, 2025
Maintainer

| bad range format: 0--1

^ this seems to indicate 0 to -1?

Nikola-Milovic Jun 27, 2025
Author

I guess so, the error message is weird, I am guessing the tunnel is doing something weird since the requests contain "Content-Length":["0"]

rchincha Jun 27, 2025
Maintainer

Does this work for smaller images? Maybe there is some critical boundary condition we need to chase/resolve.

Nikola-Milovic Jun 28, 2025
Author

Removing the tunnel and its working for larger images as well 👍

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Docker image failing to push with many abandoned multipart uploads (r2 storage backend) #3227

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Docker image failing to push with many abandoned multipart uploads (r2 storage backend) #3227

Uh oh!

Nikola-Milovic Jun 26, 2025

Issue

Environment Details

Infrastructure

Docker Image Details

Zot Configuration (/etc/zot/config.json)

Observed Behavior

1. Push Operation Patterns

2. Incomplete Multipart Uploads in R2

Logs

Key Observations from Logs

Reproduction Steps

Workarounds Attempted

Replies: 1 comment · 5 replies

Uh oh!

rchincha Jun 26, 2025 Maintainer

Uh oh!

Uh oh!

Nikola-Milovic Jun 27, 2025 Author

Uh oh!

rchincha Jun 27, 2025 Maintainer

Uh oh!

Nikola-Milovic Jun 27, 2025 Author

Uh oh!

Uh oh!

rchincha Jun 27, 2025 Maintainer

Uh oh!

Nikola-Milovic Jun 28, 2025 Author

Nikola-Milovic
Jun 26, 2025

Zot Configuration (`/etc/zot/config.json`)

Replies: 1 comment 5 replies

rchincha
Jun 26, 2025
Maintainer

Nikola-Milovic Jun 27, 2025
Author

rchincha Jun 27, 2025
Maintainer

Nikola-Milovic Jun 27, 2025
Author

rchincha Jun 27, 2025
Maintainer

Nikola-Milovic Jun 28, 2025
Author