Bug: Memory Spike in Distributor During Ingesters Flush

[KrishnaJyothika]

What is the bug?

We’ve observed a recurring memory spike in the Distributor component, where usage surges to 90–100% whenever the ingesters flush data after the upgrade to 2.14.2 version

Distributors utilizations

Ingesters utilizations

How to reproduce it?

1. Deploy mimir with 2.14.2 version
2. Push 60-65M load
3. Monitor Distributors and Ingesters utilizations - Every 2 hrs when ingesters flushes the data to Object storage distributors memory also spikes

What did you think would happen?

Whenever ingesters flushes the data to Object storage distributors memory also spikes up

What was your environment?

Kubernetes
Helm
Mimir 2.14.2

Any additional context to share?

No response

[deniszh]

Also noticed that. My hypothesis is that ingesters becoming slower during flush, so, distributors naturally accumulate more data hence memory. If you scaling up distributors and/or ingesters effect become less visible.
Should not happen in Kafka based architecture, though, so, I think not solvable in classic architecture.