Bug: tsdb.NewCircularExemplarStorage instantiated with wrong size at startup

[colega]

What is the bug?

We have noticed a strange memory usage pattern in the ingesters after restarting:

After comparing profiles of one of the affected instances, we see that Memory allocated is in tsdb.NewCircularExemplarStorage:

We've narrowed down and saw that ~8GiB were allocated in the slice allocation of tsdb.NewCircularExemplarStorage, concidentally, this is a cluster with one huge tenant that has a global max of 160M exemplars.

How to reproduce it?

Rolling out ingesters

What did you think would happen?

Since an exemplar is ~56 bytes, and 160M * 56 = 8.960.000.000, we can assume that each instance allocated the slice for all exemplars, instead of dividing the limit by the number of instances properly

So it seems that tsdb was instantiated with empty ring somehow, even though we couldn't see how that can happen.

What was your environment?

Grafana Cloud, latest weekly release, this was staging env.

Any additional context to share?

No response

[pracucci]

I investigate it. It happens when the ingesters hash ring is empty at startup, due to issues doing the memberlist fast join at startup.

Series of events in a ingester having this issue:

ts=2025-03-19T08:58:24.324724229Z caller=memberlist_client.go:762 level=error msg="failed to resolve members" addrs=REDACTED:7946 err="lookup IP addresses \"REDACTED\": lookup REDACTED. on 10.51.0.10:53: read udp 10.12.200.128:42658->10.51.0.10:53: i/o timeout"

ts=2025-03-19T08:58:24.324762689Z caller=memberlist_client.go:594 level=info msg="memberlist fast-join starting" nodes_found=0 to_join=0

ts=2025-03-19T08:58:24.324774474Z caller=memberlist_client.go:614 level=warn msg="memberlist fast-join finished" joined_nodes=0 elapsed_time=10.001797157s

...

ts=2025-03-19T08:59:41.679837471Z caller=handler.go:87 level=info user=10428 caller=head.go:756 time=2025-03-19T08:59:41.679832523Z msg="Replaying WAL, this may take a while"

The memberlist client failed to do the fast join due to a networking issue at startup. The memberlist service switches to Running state as soon as the fast join completes, either successful or not. If it fails, the memberlist state is empty, so ingester service will be started with an empty view of any ring.

[pracucci]

What we should do to improve it:

1. Add retries to discoverMembers() (in case the resolution fails). If the resolution succeed, but it resolves to 0 addresses, then it's OK (it can happen when a new Mimir cluster is bootstrapped)
2. Add a config option, similar to -memberlist.abort-if-join-fails, to abort if the fast join fail. We should enable it for ingesters, where it's critical to make sure to have a view of the hash ring at startup