Distributor → Ingester context deadline exceeded errors during ingestion despite CPU and memory headroom

[girish-ms]

We are observing frequent distributor errors when pushing samples to ingesters in our Grafana Mimir cluster. The errors surface as context deadline exceeded while sending samples from distributors to ingesters, even though ingester CPU and memory usage remain within expected limits.

This appears to be a latency and backpressure issue during bursty ingestion rather than sustained resource exhaustion.

Symptoms:

Distributor logs frequently show errors such as:

level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been partially ingested)" httpCode=500 err="send data to ingesters: failed pushing to ingester grafana-mimir-ingester-<name>: context deadline exceeded"

These errors occur in short windows and often affect multiple ingesters simultaneously. Requests may be partially ingested, and during some events the ingestion pipeline becomes severely degraded, effectively disrupting overall Mimir ingestion and, in extreme cases, resulting in a near-complete ingestion outage.

Current Ingester Topology

We've around 110-130M samples on an average, we've below configs for ingesters.

• 21 ingester replicas
• 48Gi memory requests, 52Gi memory limits
• CPU sizing 10 cores, and no limits defined

With 21 ingesters with 48Gi requests and 52Gi limits provide ~121M series at requests and ~131M series upper headroom at limits as per doc.

Ingester Resource State

• CPU usage typically ~60–70% of requested cores
• Memory usage typically ~65–75% of requested memory
• No OOM kills observed
• No ingester crashes or restarts correlated with the errors

As per mimir writes dashboard, for the same above timestamp:

We’d appreciate any insights or recommendations from the Mimir team on how to approach this.

[narqo]

Some potential directions to explore

1. Cross-check the latency against Mimir Writes networking dashboard.
2. Check if the underlying nodes where distributors and ingesters seat were under capacity, or experienced temporal issues. That is, even if Mimir pod resource usage shows that they have enough capacity, the nodes where pods run, may struggle. We see this occasionally when Mimir runs on a cloud infrastructure (eg. EKS or GKE). Grafana Kubernetes app can be useful to observe those. Or use however you monitor the infrastructure.