Thanos Store Gateway: ResourceExhausted Errors with Downloaded-Bytes and Request-Series Limits

[ValentinLvr]

Problem Description

We're experiencing ResourceExhausted errors in our Thanos Store Gateway deployment that indicate we're hitting both the downloaded-bytes limit (set to 1GB) and request-series limit (set to 500,000).
Based on what we've seen in Thanos traces, we believe this is leading to long time spent on queues (store_query_gate_ismyturn and query_gate_ismyturn spans can reach 30s) , resulting in query timeouts.

Error Messages:

rpc error: code = ResourceExhausted desc = fetch postings for block 01JXAWJK9NFZNQ9HXRGE6Q3SN1: expanded matching posting: fetch and expand postings: get postings: rpc error: code = ResourceExhausted desc = exceeded bytes limit while fetching postings: limit 1073741824 violated (got 1166720948)

rpc error: code = ResourceExhausted desc = exceeded series limit: limit 500000 violated (got 873182)

Current Configuration

Environment:

• Thanos deployed via Bitnami charts in Kubernetes
• Object storage in S3
• Single StatefulSet replica for thanos-store
• SSD persistence for index-headers cache (100GB)

Store Gateway Config:

storegateway:
  extraFlags:
    - --store.limits.request-samples=18000000
    - --store.limits.request-series=500000
    - --store.grpc.downloaded-bytes-limit=1GB
    - --max-time=-2d
    - --store.grpc.series-max-concurrency=40
    - --cache-index-header
    - --chunk-pool-size=2GB
    - --index-cache-size=2GB
  resources:
    requests:
      memory: 31Gi
      cpu: 4
    limits:
      memory: 31Gi

Questions & Concerns

1. Index File Size: We commonly see blocks with 50GB index files. Is this normal behaviour, or does it indicate a cardinality/configuration issue?
2. Postings vs Downloaded-Bytes Limit: Is it common for thanos store to fetch such big amounts of postings (>= 1GB) ?
3. Scaling Strategy: Given our organic growth, what's the recommended approach ?
   • Simply increase the existing limits ( + CPU/RAM vertical scaling) ?
   • Switch to external caching (Redis/Memcached) for index and/or bucket cache?
   • Implement horizontal scaling with multiple store replicas?
   • Use time-based or label-based sharding?
4. Cache Configuration: Would switching from in-memory to Redis/Memcached caching help with these specific ResourceExhausted errors? Our current setup shows:
   • Index cache: 2GB in-memory
   • Chunk pool: 2GB
   • No bucket caching configured

Potential Solutions We're Considering

Increase Limits:

- --store.limits.request-series=1000000
- --store.grpc.downloaded-bytes-limit=2GB
- --index-cache-size=4GB

• scale vertically our thanos-store pod (CPU & RAM)

Add External Caching:

indexCacheConfig: |
  type: REDIS
  config:
    addr: redis-cluster:6379
    cache_size: 2GB
    ttl: 24h

Any insights from the community on similar setups and their solutions would be greatly appreciated !