Outdated metrics lines of deleted shards persist

[grillbaer]

Steps to reproduce:

1. Fetch metrics lines for storage_shard_disk_size from http://<influxhost>:8086/metrics as a reference.
2. Reduce retention setting of a larger bucket so that old data shards get deleted. Wait until the next retention check and deletion really happened.
3. Fetch the metrics lines again. They will still contain the already deleted shards with their old disk size.
4. Verify that the shards really have been deleted by looking into the file system data. The shard folders will be deleted as expected and their disk space freed.
5. A full restart of InfluxDB2 will update the metrics to the expected real status.

Same behavior shows up for storage_tsm_files_disk_bytes metrics.

See the following example of metrics and file system after a retention based deletion of several shards:
influx-metrics-and-fs-example.txt

Expected behavior:

Metrics lines for already deleted shards must be removed or their disk space gauge set to 0 to correctly indicate the currently used disk space.

Actual behavior:
Metrics lines for already deleted shards persist with the disk space indication from before deletion.

Environment info:

bash-5.1# uname -srm
Linux 5.4.0-91-generic x86_64
bash-5.1# influxd version
InfluxDB v2.6.1 (git: 9dcf880fe0) build_date: 2022-12-29T15:53:07Z

Using docker image influxdb:2.6.1-alpine.

Config:

Config using default values except bolt and engine path.
influx-config.txt

Logs:

influx-retetion-check.log

Performance:

n/a

[ronanbarrett]

Can confirm the same behaviour on /metrics when running the AWS Timestream InfluxDB v2.7 build OSS version "dev". This issue makes it look like the disk is filling up regardless of retention policy, which it isn't. Can also confirm the metrics get cleaned up on a restart.