Split memberlist #14179
Unanswered
JBodkin-Amphora
asked this question in
Help and support
Split memberlist
#14179
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I'm running mimir on Azure Kubernetes using spot instances across 3 node pools, with each node pool in a different az availability zone. There are 2 distributors (running in different zones) and 3 ingesters (one in each zone). When Azure pre-empts the node, I'm seeing issues where two different memberlists/rings are getting created under the classic architecture, instead of joining to the existing ring which leads to data not being ingested.
Is there some configuration options I should set? I'm using the mimir helm chart and I haven't overridden any options for the memberlist except for the cluster label
In the above image, you can see that 3 ingesters were active and then splits into 2 and 1. I'm expecting a small difference in the graph for a few minutes rather than for it to run in a split memberlist for hours on end.
Beta Was this translation helpful? Give feedback.
All reactions