Skip to content

Bug: tsdb.NewCircularExemplarStorage instantiated with wrong size at startup #10866

Closed
@colega

Description

@colega

What is the bug?

We have noticed a strange memory usage pattern in the ingesters after restarting:

Image

After comparing profiles of one of the affected instances, we see that Memory allocated is in tsdb.NewCircularExemplarStorage:

Image

We've narrowed down and saw that ~8GiB were allocated in the slice allocation of tsdb.NewCircularExemplarStorage, concidentally, this is a cluster with one huge tenant that has a global max of 160M exemplars.

How to reproduce it?

Rolling out ingesters

What did you think would happen?

Since an exemplar is ~56 bytes, and 160M * 56 = 8.960.000.000, we can assume that each instance allocated the slice for all exemplars, instead of dividing the limit by the number of instances properly

So it seems that tsdb was instantiated with empty ring somehow, even though we couldn't see how that can happen.

What was your environment?

Grafana Cloud, latest weekly release, this was staging env.

Any additional context to share?

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions