[Bug]: Zot doesn't remove blobs from Minio repository

[Kras4ooo]

zot version

v2.1.3-rc4

Describe the bug

When I try to remove the tag, it is not removed from my Minio Bucket. I've successfully uploaded an image. I can see that the index.json is configured correctly, but when I remove a tag, I can see again that index.json has the right info, but the blobs are still there :( . What can be the issue?
If I remove it by tab name: curl -X DELETE -u 'admin:<pass>' http://localhost:5000/v2/busybox/manifests/latest it is not removed, but If I use digest: curl -X DELETE -u 'admin:admin' http://localhost:5000/v2/busybox/manifests/sha256:f2e98ad37e4970f48e85946972ac4acb5574c39f27c624efbd9b17a3a402bfe4 it is removed correctly. I tried via UI, but it doesn't work from there.

This is my config:

    {
      "storage": {
        "rootDirectory": "/tmp/zot",
        "dedupe": true,
        "gc": true,
        "commit": true,
        "gcDelay": "1s",
        "gcInterval": "5s",
        "storageDriver": {
            "name": "s3",
            "region": "eu-central-1",
            "regionendpoint": ".......",
            "forcepathstyle": true,
            "bucket": "........",
            "v4auth": true,
            "secure": true,
            "skipverify": false,
            "accesskey": ".......",
            "secretkey": "........."
        },
        "cacheDriver": {
            "name": "redis",
            "url": "redis://user:pass@host:6379/4"
        }
      },
      "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "compat": ["docker2s2"],
        "auth": { "htpasswd": { "path": "/secret/htpasswd" } },
        "accessControl": {
          "repositories": {
            "**": {
              "policies": [{
                "users": ["user"],
                "actions": ["read"]
              }],
              "defaultPolicy": []
            }
          },
          "adminPolicy": {
            "users": ["admin"],
            "actions": ["read", "create", "update", "delete"]
          }
        }
      },
      "log": { "level": "debug" },
      "extensions": {
        "search": {
          "enable": true
        },
        "mgmt": {
          "enable": true
        },
        "ui": {
          "enable": true
        },
        "scrub": {
          "enable": true,
          "interval": "10s"
        }
      },
      "scheduler": {
        "numWorkers": 30
      }
    }

To reproduce

1. Configuration
2. Client tool used
3. Seen error

Expected behavior

No response

Screenshots

No response

Additional context

No response

[rchincha]

"gc": true,

^ could you try with gc: false?

[Kras4ooo]

Unfortunately, without change :(

[rchincha]

@Kras4ooo in general, always do deletes only on images (or manifests) and not on blobs directly.
Two or more images (or manifests) may share the same blob in which case the blob is left alone and then garbage-collected later.
Pls check if this is the case for you?

[rchincha]

@Kras4ooo

Also pls track the requests down in the zot logs.

[Kras4ooo]

@rchincha As you can see:

I don't have any tags here. Meanwhile, in redis, I can see all the blobs/tags there:

and the situation is the same in minio:

The interesting part is that I also cannot see the tag in the index.json file:

{"schemaVersion":2,"manifests":[{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","digest":"sha256:f2e98ad37e4970f48e85946972ac4acb5574c39f27c624efbd9b17a3a402bfe4","size":527}]}

Now, if I try to remove the image/tag via this digest ^^ I will remove it successfully, and the GC will clean it, but I cannot do it via tag name.

About the requests, This is when I create a tag (all is good)

{"level":"info","module":"http","component":"session","clientIP":"127.0.0.1:35856","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/28.0.1 go/go1.23.6 git-commit/bbd0a17 kernel/6.11.0-21-generic os/linux arch/amd64 UpstreamClient(Docker-Client/28.0.1 \\(linux\\))"]},"goroutine":1989,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-04-11T08:23:51.621634611Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":1984,"caller":"zotregistry.dev/zot/pkg/api/authn.go:141","time":"2025-04-11T08:23:51.632385708Z","message":"user profile successfully set"}
{"level":"debug","blob path":"/busybox/blobs/sha256/b744a4e23f0481ec926f448dd828665f2eec2d601e1efd2833e40f40c52a4daa","goroutine":1984,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1319","time":"2025-04-11T08:23:51.923241099Z","message":"blob path found"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"127.0.0.1:35862","method":"HEAD","path":"/v2/busybox/blobs/sha256:b744a4e23f0481ec926f448dd828665f2eec2d601e1efd2833e40f40c52a4daa","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Authorization":["******"],"Connection":["close"],"User-Agent":["docker/28.0.1 go/go1.23.6 git-commit/bbd0a17 kernel/6.11.0-21-generic os/linux arch/amd64 UpstreamClient(Docker-Client/28.0.1 \\(linux\\))"]},"goroutine":1984,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-04-11T08:23:51.92327301Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":2038,"caller":"zotregistry.dev/zot/pkg/api/authn.go:141","time":"2025-04-11T08:23:51.926972187Z","message":"user profile successfully set"}
{"level":"debug","blob path":"/busybox/blobs/sha256/ff7a7936e9306ce4a789cf5523922da5e585dc1216e400efb3b6872a5137ee6b","goroutine":2038,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1319","time":"2025-04-11T08:23:52.039187129Z","message":"blob path found"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"127.0.0.1:35878","method":"HEAD","path":"/v2/busybox/blobs/sha256:ff7a7936e9306ce4a789cf5523922da5e585dc1216e400efb3b6872a5137ee6b","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Authorization":["******"],"Connection":["close"],"User-Agent":["docker/28.0.1 go/go1.23.6 git-commit/bbd0a17 kernel/6.11.0-21-generic os/linux arch/amd64 UpstreamClient(Docker-Client/28.0.1 \\(linux\\))"]},"goroutine":2038,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-04-11T08:23:52.039217236Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":2039,"caller":"zotregistry.dev/zot/pkg/api/authn.go:141","time":"2025-04-11T08:23:52.042896686Z","message":"user profile successfully set"}

When I remove it (via the UI):

{"level":"info","identity":"user","goroutine":2293,"caller":"zotregistry.dev/zot/pkg/api/authn.go:141","time":"2025-04-11T08:27:26.563275958Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"user","component":"session","clientIP":"192.168.88.29:54766","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Authorization":["******"],"Connection":["close"],"User-Agent":["kube-probe/1.32"]},"goroutine":2293,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-04-11T08:27:26.563301607Z","message":"HTTP API"}
{"level":"info","identity":"admin","goroutine":2243,"caller":"zotregistry.dev/zot/pkg/api/authn.go:141","time":"2025-04-11T08:27:28.533043542Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"admin","component":"session","clientIP":"127.0.0.1:55622","method":"DELETE","path":"/v2/busybox/manifests/dddd","statusCode":202,"latency":"1s","bodySize":0,"headers":{"Accept":["application/json"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-US,en;q=0.9,bg;q=0.8"],"Authorization":["******"],"Connection":["keep-alive"],"Cookie":["user=admin; session=MTc0NDM2MDAyNXxEWDhFQVFMX2dBQUJFQUVRQUFCQl80QUFBZ1p6ZEhKcGJtY01CZ0FFZFhObGNnWnpkSEpwYm1jTUJ3QUZZV1J0YVc0R2MzUnlhVzVuREF3QUNtRjFkR2hUZEdGMGRYTUVZbTl2YkFJQ0FBRT180_r7k4qUE8XKLPvcAHLnNFUJTOUxzkK98hHWJgwEqMs="],"Origin":["http://localhost:5000"],"Referer":["http://localhost:5000/image/busybox"],"Sec-Ch-Ua":["\"Chromium\";v=\"134\", \"Not:A-Brand\";v=\"24\", \"Google Chrome\";v=\"134\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"],"X-Zot-Api-Client":["zot-ui"]},"goroutine":2243,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-04-11T08:27:29.55443987Z","message":"HTTP API"}
{"level":"info","identity":"user","goroutine":2315,"caller":"zotregistry.dev/zot/pkg/api/authn.go:141","time":"2025-04-11T08:27:36.561190333Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"user","component":"session","clientIP":"192.168.88.29:47594","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Authorization":["******"],"Connection":["close"],"User-Agent":["kube-probe/1.32"]},"goroutine":2315,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-04-11T08:27:36.561265127Z","message":"HTTP API"}
{"level":"info","identity":"user","goroutine":2344,"caller":"zotregistry.dev/zot/pkg/api/authn.go:141","time":"2025-04-11T08:27:36.562596467Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"user","component":"session","clientIP":"192.168.88.29:47596","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Authorization":["******"],"Connection":["close"],"User-Agent":["kube-probe/1.32"]},"goroutine":2344,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-04-11T08:27:36.562619081Z","message":"HTTP API"}

and after that, nothing was removed from the Minio. :( ,

I've tried with gc true and gc false (nothing) again.

[rchincha]

gc kicks in about an hour, periodically.

Meanwhile, we will try to reproduce this.