Ingester local tsdb storage does not stop growing

[alemsh]

We've been running into a problem with our ingesters continually filling up their local pvc storage (i.e. WAL). Our setup uses an s3 gateway for long term bucket storage of metrics.

I had tried messing around with blocks-storage.tsdb.retention-period configuration setting, but that ended up causing a number of other headaches, as changing that requires a myriad of other changes that are not well documented what else needs to be updated. Apparently, these defaults should be sufficient so I am not sure why local disk is filling up. My understanding is that metrics should go along a path of Ingester memory -> local disk (WAL) -> long term s3 buckets. My understanding is that after 13h` blocks/series in the WAL should be compacted and uploaded to our s3, and that those blocks should be marked for deletion, and the compactor should be deleting them. I can confirm however that WAL blocks that are still sitting on local disk after several days. I realize that those blocks might not necessarily be deleted at 13h, but having them be there several days later seems incorrect.

According to monitoring and compactor logs, it seems like things are being deleted, but very sparingly. If I restart the ingesters, it seems like then all the blocks that were marked for deletion actually get deleted on restart, as the amount deletions occurring increases by an order of magnitude, and the disk space usage drops down.
Some stats:
Helm chart version: mimir-distributed 5.8.0
Image version: grafana/mimir:2.17.0

Some cluster metrics

sum(cortex_ingester_memory_series) = 8.2 million
sum(prometheus_tsdb_head_series) = 5.2 million
max(cortex_distributor_replication_factor) = 3

Our helm config:

global:
  podAnnotations:
    bucketSecretVersion: "0"
minio:
  enabled: false
mimir:
  structuredConfig:
    limits:
      align_queries_with_step: true
      max_global_series_per_user: 100000000
      ingestion_rate: 20000000
      ingestion_burst_size: 5000000
      accept_ha_samples: true
      ha_cluster_label: cluster
      ha_replica_label: __replica__
      out_of_order_time_window: 5m
      compactor_block_upload_enabled: true
      ruler_max_rules_per_rule_group: 0
      ruler_max_rule_groups_per_tenant: 0
    common:
      storage:
        backend: s3
        s3:
          endpoint: [redacted]
    frontend:
      split_queries_by_interval: 1h
      cache_results: true
      parallelize_shardable_queries: true
      log_queries_longer_than: 1s
    ingester:
      ring:
        kvstore:
          store: etcd
    distributor:
      ha_tracker:
        enable_ha_tracker: true
        kvstore:
          store: etcd
      ring:
        kvstore:
          store: etcd
    compactor:
      sharding_ring:
        kvstore:
          store: etcd
    store_gateway:
      sharding_ring:
        kvstore:
          store: etcd
    query_scheduler:
      ring:
        kvstore:
          store: etcd
    ruler:
      ring:
        kvstore:
          store: etcd
    alertmanager_storage:
      s3:
        bucket_name: mimir-alertmanager
    blocks_storage:
      backend: s3
      s3:
        bucket_name: mimir-blocks
      tsdb:
        wal_compression_enabled: true
    ruler_storage:
      s3:
        bucket_name: mimir-ruler
alertmanager:
  persistentVolume:
    enabled: true
    storageClass: [redacted]
  replicas: 2
  resources:
    limits:
      memory: 1.4Gi
    requests:
      cpu: 1
      memory: 1Gi
  statefulSet:
    enabled: true
compactor:
  persistentVolume:
    size: 50Gi
    storageClass: [redacted]
  resources:
    limits:
      memory: 4Gi
    requests:
      cpu: 1
      memory: 1.5Gi
distributor:
  replicas: 2
  resources:
    limits:
      memory: 5.7Gi
    requests:
      cpu: 2
      memory: 4Gi
ingester:
  persistentVolume:
    size: 150Gi
    storageClass: [redacted]
  replicas: 5
  resources:
    limits:
      cpu: 12
      memory: 40Gi
    requests:
      cpu: 6
      memory: 20Gi
  zoneAwareReplication:
    enabled: false
    topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target # support for enterprise.legacyLabels
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'

        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'
admin-cache:
  enabled: true
  replicas: 2
chunks-cache:
  enabled: true
  replicas: 2
index-cache:
  enabled: true
  replicas: 3
metadata-cache:
  enabled: true
results-cache:
  enabled: true
  replicas: 2
overrides_exporter:
  replicas: 1
  resources:
    limits:
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 128Mi
querier:
  replicas: 3
  resources:
    limits:
      memory: 5.6Gi
    requests:
      cpu: 2
      memory: 4Gi
query_frontend:
  parallelize_shardable_queries: true
  query_sharding_total_shards: 6
  replicas: 2
  resources:
    limits:
      memory: 2.8Gi
    requests:
      cpu: 3
      memory: 2Gi
ruler:
  replicas: 1
  resources:
    limits:
      memory: 2.8Gi
    requests:
      cpu: 1
      memory: 2Gi
store_gateway:
  persistentVolume:
    size: 10Gi
    storageClass: [redacted]
  replicas: 3
  resources:
    limits:
      memory: 2.1Gi
    requests:
      cpu: 1
      memory: 1.5Gi
  zoneAwareReplication:
    enabled: false