Mimir 3.0.1 Upgrade issues

[jkrishnarao2003]

I am upgrading Mimir cluster from 2.15.x to 3.0.1. I am deploying in microservices mode through ingester storage architecture.

I am getting this error. I am not setting ingest's ring instance_id. so it's set by Mimir only. Then why am I getting this error.

ts=2025-12-12T10:35:57.092491253Z caller=log.go:157 level=error msg="error running application" err="ingester ID omsdp11e9f.uprr.com doesn't match regular expression \"-([0-9]+)$\"\ncalculating ingester partition ID\ngithub.com/grafana/mimir/pkg/ingester.New\n\t/go/src/github.com/grafana/mimir/pkg/ingester/ingester.go:491\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initIngesterService\n\t/go/src/github.com/grafana/mimir/pkg/mimir/modules.go:746\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:1006\nmain.main\n\t/go/src/github.com/grafana/mimir/cmd/mimir/main.go:235\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nerror initialising module: ingester-service\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:1006\nmain.main\n\t/go/src/github.com/grafana/mimir/cmd/mimir/main.go:235\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"

I tried to set the ingester.ring.instance_id to "omsdp11e9f.uprr.com-1" or "omsdp11e9f-1", I am getting error like network interfaces not found.

ts=2025-12-12T11:37:48.704123097Z caller=netutil.go:90 level=warn msg="error getting addresses for interface" inf=eth0 err="route ip+net: no such network interface"
ts=2025-12-12T11:37:48.704207259Z caller=netutil.go:90 level=warn msg="error getting addresses for interface" inf=en0 err="route ip+net: no such network interface"
ts=2025-12-12T11:37:48.70433732Z caller=log.go:157 level=error msg="error running application" err="no useable address found for interfaces [eth0 en0]\nfailed to get frontend address\ngithub.com/grafana/mimir/pkg/frontend.InitFrontend\n\t/go/src/github.com/grafana/mimir/pkg/frontend/config.go:90\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initQueryFrontend\n\t/go/src/github.com/grafana/mimir/pkg/mimir/modules.go:912\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:1006\nmain.main\n\t/go/src/github.com/grafana/mimir/cmd/mimir/main.go:235\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nerror initialising module: query-frontend\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/go/src/github.com/grafana/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:1006\nmain.main\n\t/go/src/github.com/grafana/mimir/cmd/mimir/main.go:235\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"

[aknuds1]

Converting to discussion, as it looks more like a request for help.

[jkrishnarao2003]

To me it looks like an issue from the top because the cluster was working fine in 2.15 version.

[jkrishnarao2003]

The SAD part with Mimir 3.0.1 upgrade is, when I run the cluster in ingest storage architecture, this is the problem.

1. instance_id of ingester node is forced to have a suffix like -1, -2, etc ex - ingester-1, ingester-2, host1.com-1, host2.com-2 like this. Otherwise I get error like "ingester ID omsdp11e9f.uprr.com doesn't match regular expression "-([0-9]+)$"\ncalculating ingester partition ID. I even saw the code at github.com/grafana/mimir/pkg/ingester.New\n\t/go/src/github.com/grafana/mimir/pkg/ingester/ingester.go:491 line.
2. Then this suffix number is considered as Kafka partition the ingest has to poll with. What happens if I have 4 ingesters running but Kafka topic has more than 4 partitions. I can't have like that.
3. Why the consumer group is unique per ingester? The ideal way of a Kafka consumer should be having a single consumer group to all ingesters and assign partition dynamically at run time right?
4. My question is how at run time my deployment can decide which ingester has to point to which Kafka partition?
5. Instead of that Mimir should have the capability to assign the partition dynamically at run time right?

[deniszh]

ingester node is forced to have a suffix like -1, -2, etc

I think that's not happened in 3.0.1, that was even before, probably introduced later in 2.x. IIRC you can use -ingester.ring.instance-id cli parameter to override.

What happens if I have 4 ingesters running but Kafka topic has more than 4 partitions. I can't have like that.

Nothing critical happen. 4 partitions will be used, rest are not.

My question is how at run time my deployment can decide which ingester has to point to which Kafka partition?

As far as I understand it's very simple - number of ingester is number of partition. I.e. it's statically mapped.

In worst case you can still use old architecture, Kafka is recommended but not mandatory yet.