Support Non-Recursive Deletion of Empty Object Store "Folders" by Compactor

[bobykus31]

Not all s3 like object storage types support recursive deletion of "folders"

In NetApp Storage Grid object storage "subfolders" can be recursively deleted, like

s3cmd del -r  s3://metrics-k8s/01K2KXE4286YYWK85EW3EB8NC5TEST/
delete: 's3://metrics-k8s/01K2KXE4286YYWK85EW3EB8NC5TEST/chunks/000001'
delete: 's3://metrics-k8s/01K2KXE4286YYWK85EW3EB8NC5TEST/test.json' 
s3cmd ls s3://metrics-k8s/| grep  01K2KXE4286YYWK85EW3EB8NC5TEST

However, in f.x. SeaweedFS object storage "subfolders" can not be recursively deleted, like

s3cmd del -r   s3://metrics/01K1ZGKEEZCSSMPASWSVZRK0YTTEST/
delete: 's3://metrics/01K1ZGKEEZCSSMPASWSVZRK0YTTEST/test.json'
s3cmd ls s3://metrics/| grep  01K1ZGKEEZCSSMPASWSVZRK0YTTEST
                         DIR  s3://metrics/01K1ZGKEEZCSSMPASWSVZRK0YTTEST/chunks

Even files are being removed, empty "folders" remain. This makes Thanos Compactor clean.go component confused

Aug 14 07:49:24 thanos-compact3 thanos[2729]: ts=2025-08-14T07:49:24.636119795Z caller=clean.go:50 level=info msg="found partially uploaded block; marking for deletion" block=01K20T2CG8W6PJVR371E9Q6H2N
Aug 14 07:49:24 thanos-compact3 thanos[2729]: ts=2025-08-14T07:49:24.649023741Z caller=clean.go:60 level=info msg="deleted aborted partial upload" block=01K20T2CG8W6PJVR371E9Q6H2N thresholdAge=48h0m0s

that in fact does not delete any "subfolders" and "folders", and continues every 5 min or when next clean.go run.

To delete "folder" you have to explicitly write it down, like

s3cmd del s3://metrics/01K1ZGKEEZCSSMPASWSVZRK0YTTEST/chunks
s3cmd del s3://metrics/01K1ZGKEEZCSSMPASWSVZRK0YTTEST

It would be a solution to have Thanos Compactor a mode to delete files and then each "folder" like

s3cmd del -r s3://metrics/01K1ZGKEEZCSSMPASWSVZRK0YTTEST
s3cmd del s3://metrics/01K1ZGKEEZCSSMPASWSVZRK0YTTEST/chunks
s3cmd del s3://metrics/01K1ZGKEEZCSSMPASWSVZRK0YTTEST

Such option can be configurable as it is probably resources consuming.

[GiedriusS]

This sounds like a buggy S3 implementation. There is no concept of folders in S3, only objects. Are you sure that NetApp Storage Grid actually follows the S3 API spec?

[bobykus31]

There is no issue with NetApp Storage Grid object storage and Thanos Compactor.

The problem is related to other s3 like object storage implementations, where recursive removal of "subfolders" is not removing this "subfolders". Please see also

#7840

I know about objects, probably. But this does not change the reason the request is created. Thanos Compactor leaving alot of leftovers after removing objects from object storage while compacting. The leftovers looks like

s3://metrics/01K20T2CG8W6PJVR371E9Q6H2N/chunks

The caller=clean.go keep trying to remove it periodically (every 5 min) because of "found partially uploaded block; marking for deletion", with no success, even no error is generated. For the reason described in the request.

Aug 14 07:49:24 thanos-compact3 thanos[2729]: ts=2025-08-14T07:49:24.636119795Z caller=clean.go:50 level=info msg="found partially uploaded block; marking for deletion" block=01K20T2CG8W6PJVR371E9Q6H2N
Aug 14 07:49:24 thanos-compact3 thanos[2729]: ts=2025-08-14T07:49:24.649023741Z caller=clean.go:60 level=info msg="deleted aborted partial upload" block=01K20T2CG8W6PJVR371E9Q6H2N thresholdAge=48h0m0s

I had like 3k such messages in the Thanos Compactor log with his particular "block=01K20T2CG8W6PJVR371E9Q6H2N" until I removed block manually.

[bobykus31]

Here is a cycle for one block from the log

~# journalctl  --no-pager --since "1h ago" | grep 01K2XJA92CRYT735HE95K8DAY1
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.03001815Z caller=block.go:227 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/meta.json bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.03780889Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/index bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.061070783Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000001 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.064707293Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000002 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.067348834Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000003 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.070112608Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000004 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.072310247Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000005 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.074887366Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000006 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.077703479Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000007 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.079943709Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000008 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.082547269Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000009 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.085213437Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000010 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.087549629Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000011 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.09010637Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000012 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.092867629Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000013 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.095068134Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000014 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.09775435Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000015 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.100656216Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000016 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.102907617Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000017 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.105292121Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000018 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.10812197Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000019 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.109985986Z caller=block.go:270 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/chunks/000020 bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.113283302Z caller=block.go:250 level=debug msg="deleted file" file=01K2XJA92CRYT735HE95K8DAY1/deletion-mark.json bucket="tracing: promproxy-metrics"
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.11332998Z caller=blocks_cleaner.go:54 level=info msg="deleted block marked for deletion" block=01K2XJA92CRYT735HE95K8DAY1
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.487360418Z caller=clean.go:50 level=info msg="found partially uploaded block; marking for deletion" block=01K2XJA92CRYT735HE95K8DAY1
Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.49557858Z caller=clean.go:60 level=info msg="deleted aborted partial upload" block=01K2XJA92CRYT735HE95K8DAY1 thresholdAge=48h0m0s
Aug 20 09:25:24 thanos-compact3 thanos[22992]: ts=2025-08-20T09:25:24.438133854Z caller=clean.go:50 level=info msg="found partially uploaded block; marking for deletion" block=01K2XJA92CRYT735HE95K8DAY1
Aug 20 09:25:24 thanos-compact3 thanos[22992]: ts=2025-08-20T09:25:24.446802346Z caller=clean.go:60 level=info msg="deleted aborted partial upload" block=01K2XJA92CRYT735HE95K8DAY1 thresholdAge=48h0m0s
Aug 20 09:26:24 thanos-compact3 thanos[22992]: ts=2025-08-20T09:26:24.485725123Z caller=clean.go:50 level=info msg="found partially uploaded block; marking for deletion" block=01K2XJA92CRYT735HE95K8DAY1
Aug 20 09:26:24 thanos-compact3 thanos[22992]: ts=2025-08-20T09:26:24.495693641Z caller=clean.go:60 level=info msg="deleted aborted partial upload" block=01K2XJA92CRYT735HE95K8DAY1 thresholdAge=48h0m0s
~# s3cmd ls  s3://promproxy-metrics/01K2XJA92CRYT735HE95K8DAY1
                          DIR  s3://promproxy-metrics/01K2XJA92CRYT735HE95K8DAY1/
~# s3cmd ls  s3://promproxy-metrics/01K2XJA92CRYT735HE95K8DAY1/
                          DIR  s3://promproxy-metrics/01K2XJA92CRYT735HE95K8DAY1/chunks/

as you see, the files was deleted from the block and the block was "deleted", probably recursively

Aug 20 09:24:30 thanos-compact3 thanos[22992]: ts=2025-08-20T09:24:30.11332998Z caller=blocks_cleaner.go:54 level=info msg="deleted block marked for deletion" block=01K2XJA92CRYT735HE95K8DAY1

but still remain, empty.

~# s3cmd ls  s3://promproxy-metrics/01K2XJA92CRYT735HE95K8DAY1
                          DIR  s3://promproxy-metrics/01K2XJA92CRYT735HE95K8DAY1/
~# s3cmd ls  s3://promproxy-metrics/01K2XJA92CRYT735HE95K8DAY1/
                          DIR  s3://promproxy-metrics/01K2XJA92CRYT735HE95K8DAY1/chunks/

We do use seeweedfs obj store

[GiedriusS]

What are the attributes of the object s3://metrics/01K20T2CG8W6PJVR371E9Q6H2N/chunks?

[bobykus31]

Dunno as it is already removed by script, but here is a meta of similar empty "folder"

s3cmd ls s3://promproxy-metrics/01K2Z2VKFE5FWJ6VT85PHPQ45N/
                          DIR  s3://promproxy-metrics/01K2Z2VKFE5FWJ6VT85PHPQ45N/chunks

> fs.meta.cat buckets/promproxy-metrics/01K2Z2VKFE5FWJ6VT85PHPQ45N
{
  "name": "01K2Z2VKFE5FWJ6VT85PHPQ45N",
  "isDirectory": true,
  "chunks": [],
  "attributes": {
    "fileSize": "0",
    "mtime": "1755537744",
    "fileMode": 2147484153,
    "uid": 2905,
    "gid": 2905,
    "crtime": "1755537744",
    "mime": "",
    "ttlSec": 0,
    "userName": "",
    "groupName": [],
    "symlinkTarget": "",
    "md5": "",
    "rdev": 0,
    "inode": "0"
  },
  "extended": {},
  "hardLinkId": "",
  "hardLinkCounter": 0,
  "content": "",
  "remoteEntry": null,
  "quota": "0",
  "wormEnforcedAtTsNs": "0"
}chunks 0 meta size: 56 gzip:84

> fs.meta.cat buckets/promproxy-metrics/01K2Z2VKFE5FWJ6VT85PHPQ45N/chunks
{
  "name": "chunks",
  "isDirectory": true,
  "chunks": [],
  "attributes": {
    "fileSize": "0",
    "mtime": "1755537744",
    "fileMode": 2147484153,
    "uid": 2905,
    "gid": 2905,
    "crtime": "1755537744",
    "mime": "",
    "ttlSec": 0,
    "userName": "",
    "groupName": [],
    "symlinkTarget": "",
    "md5": "",
    "rdev": 0,
    "inode": "0"
  },
  "extended": {},
  "hardLinkId": "",
  "hardLinkCounter": 0,
  "content": "",
  "remoteEntry": null,
  "quota": "0",
  "wormEnforcedAtTsNs": "0"
}chunks 0 meta size: 36 gzip:64