Compactor uses unreasonably high amounts of disk space

[MartinEmrich]

We switched away from a pure Prometheus setup (on block storage) to primarily save block storage space.

But the Compactor seems to need more and more disk space. First 70GB were sufficient, and over a few weeks it halted again and again, and I just went from 200 to 250GB and it ate them up quickly and is halted again.

It runs (via the Helm Chart) with

  - args:
    - compact
    - --wait
    - --log.level=info
    - --log.format=logfmt
    - --http-address=0.0.0.0:10902
    - --data-dir=/var/thanos/compact
    - --objstore.config-file=/etc/thanos/objstore.yaml
    - --compact.enable-vertical-compaction
    - --deduplication.replica-label=prometheus_instance

The largest Folder (the 01xxxxxxx hashed top-level prefixes) in the object store is at 112GB; as I understand it that's the data unit the compactor works on at a time. As it's called "compactor", not "expander", the compacted data it produces should be less than the original, so in theory 250GB should be sufficient.

I have read #7198 and #7197, but found nothing that could help me.

I took a look at one "run" after resuming. With approx. Timestamps:

• 14:45 I resized the PV to 250GB, and restarted the compactor to unlock the "halted" state. After start, it correctly cleaned up, and all disk was free. It starts compacting according to the logs, but does not use much space.
• 15:02 It starts another compacting round. From now on, free disk space decreases very fast. It only logs "successfully synchronized block metadata" every minute, while churning along.
• 15:30 It logs "downloaded and verified blocks; compacting blocks". Note it saying blocks, and listing paths containing 6 of these 01XXXXX IDs. At this point, more than 50% are already consumed. Disk usage continues to decrease; I assume now the actual compaction begins.
• 15:53 Finally the disk is full, and compaction halted again.

So the questions are

• Why does it not download one but 6 of these blocks at once, instead of one-by-one to save disk space? Is there a configuration option?
• To my understanding, Prometheus has its own compactor that ran since the dawn of time, and according to Object Store usage, thanos sidecar did upload everything from the prometheus instance. Why does it need to be compacted again?
• I still assume it is working itself through the data imported from the old prometheus (~1.5y/2.4TB of data). The bucket is already grown to 4.2TB.
• Will the compactor need less disk when it has "caught up"? How do I know when it has? A hint could be the disappearance of the "block has no labels left, creating one" messages, but is there a way to estimate progress?
• Is "Found overlapping blocks during compaction" a bad thing? I guess these come only from new blocks (where two prometheus+sidecar pods run redundant).

Thanks!

[GiedriusS]

You are correct, we could indeed not download anything on disk but rather stream from object storage. Because the implementation is easier when you read files off disk, it was chosen a long time ago. There's also this very old issue #3416.

Prometheus runs compaction based on what retention you have set. IIRC if it's longer than 3 or 4 days then only in such case local compaction happens. We uploaded compaction level 1 blocks only to have data available as soon as possible in case of incidents. The newest Thanos version now also supports running with local compaction enabled because previously Prometheus could've started compacting blocks locally before Thanos was able to upload them.

For compactor's progress, you can check the "todo" metrics. There is a loop where it tries to constantly plan the work. Through that you can deduce how much work it needs to do.

[MartinEmrich]

Thanks, I have subscribed to that issue.

Compactor todo blocks is usually 0, with regular peaks to ~5-10. Todo compactions regularly go to 1, maybe 2; only now as it is blocked yet again it obviously accumulates.

So there is clearly nothing stalled or accumulated, yet it randomly sometimes needs even more disk as before. (Now stalled again at whopping 400GB of storage).