flagd idle memory consumption 40mb (higher than expected)

[gnesher]

We've run some initial tests on flagd memory consumption, single file system syncer with < 10 flags with no reads and see ~40mb consumption.

This seems high for a Go application, and I wanted to know if this is in line with what we should expect or if there's anything we can do to reduce the footprint.

The goal is to support relatively small instances where 40mb of memory does make an impact.

[tangenti]

Hi @gnesher ,

Could you provide more details of your test? What version of flagd did you use? Did you run the flagd with or without container? How did you measure the memory usage of flagd?

[gnesher]

@tangenti
We are using flagd: v0.11.4 (from May we can try to upgrade, but I've not seen any memory improvments and didn't want to change version during our tests)

We are installing flagd locally (no container) and using the rss field in 'ps' would be the simplest way to get this value (we use a slightly more complex way of monitoring the entire setup, but I verified that the results are the same)

i.e. ps -o rss= -p <proccess_id>

[m-olko]

Hi @gnesher ,

I did some investigation and wanted to share my thoughts and findings.

When ran newly built flagd (current main so ~v0.12.9) the rss value on my computer was even higher: ~65 MB with only two flags.
I wanted to take a closer look on memory metrics from go, so I utilized pprof built into flagd and available after building flagd with:
go build -a -ldflags "-X main.version=${VERSION} -X main.commit=${COMMIT} -X main.date=${DATE}" -o ./bin/flagd ./flagd/main.go ./flagd/profiler.go
The flagd instance then provides service on port 6060 with the go pprof package

Looking at the heap size (which should be the biggest part of used memory) shows that that the flagd really uses ~14MB of data:
# HeapInuse = 14082048
When forced to run Garbage Collector, the usage drops even more to under 12MB:
# HeapInuse = 11501568
And enforcing soft memory limit with export GOMEMLIMIT=10MiB allows us to reduce footprint to under 10MB:
# HeapInuse = 9969664
All this data is taken from http://localhost:6060/debug/pprof/heap?debug=1 endpoint of pprof.

Quick internet research shows possibility that the memory, even though released by the Go, is not updated in the system - resulting in increased rss value: golang/go#39779 (comment)

So it is possible that the flagd would run without problems with much less resources available. Have you had issues with running your instance on machine with limited resources or limited by the cgroup? Or are your findings purely theoretical for now?

[m-olko]

Also, following up on the resource used, I decided to investigate what is using such amount of data. I knew that sometimes libraries related to cloud resources like to download full configuration files, and pure initialization of them might use significant amount of memory. To test that i commented out all the sync types besides the file one, to remove the footprint of gcp, aws, azure, kubernetes etc related libraries. You can see updated code on my fork https://github.com/m-olko/flagd/tree/memory_investigation

The rss value of such stripped flagd decreased to ~25MB, while the heap size to under 3MB.

Maybe it would be beneficial to create some compile-time flags that will allow for removal of mentioned libraries when used memory needs to be minimal?

@toddbaert maybe you could share your opinion on such feature?

[gnesher]

@m-olko thanks, I've also forwarded your response to our performance team and will see what they say

Our software is installed on customer instances (or local machines). We've seen a rise in the use of low-end setups, where memory/CPU consumption can hurt customers, so we are tracking any increase in our setups.

[m-olko]

Ping @gnesher,

Is there any update on this issue?

[beeme1mr]

I'd imagine that removing the Kubenetes sync would also help reduce flagd's footprint. It may be nice to have a lightweight version of flagd that only supports the gRPC, HTTP, and file syncs. I'm not sure how much effort it would be to maintain this alternative version. I'm assuming this would be non-trivial, though.

It's worth noting that most of the flagd providers now support in-process evaluation, which may be a better approach depending on your use case.

[gnesher]

Ping @gnesher,

Is there any update on this issue?

I had a longer conversation with our performance team. Memory consumption measuring is done based on a set of customer requirements, it's unlikely we could convince them to change those.

In short, this isn't a blocker for us and we are planning to roll this out in a new major release - at the same time if there's a simple way to reduce memory consumption (or at the very least percived memory consumption) I think it would simplify future flagd integrations with a fair number of customers

Hope this makes sense and I'm sorry I can't help with the actual implementation, we are very much new to Go and this goes well beyond what I'd feel comfortable changing

[m-olko]

@beeme1mr maybe you could create an issue with name more related to "lightweight version" without external syncs, where we could discuss potential solutions and close this one? Please let me know if there is any additional info that I can provide

[m-olko]

@beeme1mr is there any update on that? Maybe we could close this issue, as there are no new comments?