Compactor seems okay but no activities show after upgrading

[edoardo-c]

Hi all,
I have been having a problem since upgrading from v0.31.0 to v0.38.0. Compaction, downsampling and deletion all seem to have stopped since the upgrade, although all seems to be fine when following the troubleshooting tips from Troubleshoot Compactor Backlog

The compactor is running, the iteration increases, but all "Compactor progress metrics" show 0.

My current settings are:

- args:
 - compact
 --wait
 --wait-interval=5m
 --log.level=debug
 --log.format=logfmt
 --objstore.config-file=/config/config.yml
 --data-dir=/var/thanos/compact
 --debug.accept-malformed-index
 --retention.resolution-raw=90d
 --retention.resolution-5m=730d
 --retention.resolution-1h=750d
 --delete-delay=48h
 --compact.concurrency=2
 --downsample.concurrency=2
 --block-files-concurrency=2
 --compact.blocks-fetch-concurrency=4
 --consistency-delay=30m
 --deduplication.replica-label=prometheus_replica
 --deduplication.replica-label=rule_replica

My graphs use the todo metrics fund in the above referenced doc.

As you can see in the image, the compactor is healthy and on the right, you can see the S3 bucket seems to be showing activity.

I ran a few queries which all seem to give the same trends but it is not clear to me if I am doing a good or bad job at managing the compactor.

I can only conclude that metric names have changes or that something changed which I didn't understand/see in the change logs.

Can anyone help solve the mystery.

Many thanks,
Ed