Skip to content

Bug: MimirGossipMembersTooLow firing for query-frontend #13795

@CarolineCH

Description

@CarolineCH

What is the bug?

We recently upgraded to Mimir 2.17.1. We using the jsonnet to deploy and configure Mimir. We import https://github.com/grafana/mimir.git version 2.17.1 into our jsonnet repo and have some custom logic on top.

We noticed that the MimirGossipMembersTooLow alert was firing for the query-frontend. I can see from this comment it is intended that the query-frontend joins the ring and that the query-frontend was added to the gossip alerts in #10289.

We are not using the new ingest architecture and we are using -query-scheduler.service-discovery-mode:dns for the query-frontend to discover the query-schedulers so we do not need the query-frontends to join the ring. The query-frontends are initializing their own ring and this is causing the alert to fire as the query-frontend does not see the same number of ring members as the other components.

How to reproduce it?

Deploy Mimir 2.17.1 using the jsonnet to deploy and configure Mimir and use the mimir-mixin to generate alerting rule definitions. Apply these rules to a prometheus instance.
Configure the query-frontend to use -query-scheduler.service-discovery-mode:dns, do not enable the new ingest architecture. Do not configure the query-frontend to join the memberlist ring used by the other components.

What did you think would happen?

Please correct me if I am wrong but it seems like the query-frontend should only be included in the gossip alerts if query-scheduler.service-discovery-mode is set to ring or the new architecture is enabled.

What was your environment?

We are running on Kubernetes version 1.30. Using jsonnet 2.17.1 to deploy Mimir 2.17.1.

Any additional context to share?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions