High memory usage for large configuration using glob as matcher type

[pedro-stanaka]

Summary

Recently, when running the exporter with a large configuration (i.e. around 3k entries), we saw a big heap allocation to store the state inside the FSM for glob matching. This issue is just a sanity check to see if we should have be seeing this amount of memory usage.

For comparison, in the previous version of the configuration (with a couple hundred entries) we were seeing 30MiB~ usage per instance on idle, with the new huge configuration we are seeing around 150MiB usage.

For now, as a workaround we are using regex matching, which (due to LRU matcher in the mapper) has not changed CPU usage a lot.

This is the heap for glob matcher:

And this one is for the regex matcher:

[matthiasr]

Unfortunately this doesn't really seem to show what is using all the space inside fsm.AddState … is there any way we can find out more about that? 50KiB per mapping does seem like quite a lot, however I am spoiled for memory and wouldn't find 150MB too onerous. In what circumstances is this a problematic amount of memory?

From our conversation at PromCon, I am also worried about memory use (allocations) while matching in the FSM. Do you have any insight what a large mapping configuration does to that?

[matthiasr]

Can you tell us more about the shape of this configuration? How did you get to 3000 entries? Are there a lot of different variants at the same match level, or is it a more complicated decision tree? If you could share a (sanitized) version of the configuration, we could throw it in a Benchmark and measure what is happening in more detail.

[matthiasr]

Alternatively, a "recipe" for generating a similar configuration, that also shows the issue, would help me a lot.