Extra memory consumed for unknown reason

[nalepae]

Starting at:

• Use head to validate atts for previous epoch #16109

the beacon node start to consume extra memory.

Metric: go_memstats_alloc_bytes

With the previous commit:

4.82GB < 1H average < 5.17GB when stabilized

With this commit:

6.22GB < 1H average < 6.60GB when stabilized

==> This commit introduces a ~1.3GB extra consumption when stabilized.

Important

• Without this commit, the memory needs ~1H30 to stabilize.
• Wit this commit, the memory needs ~3H to stabilize.

The following ZIP file contains before/after heap pprof memory consumption.

before-after.zip

[nalepae]

States are efficiently stored in caches, especially thanks to multi value slices. If 1 state takes 300 MB and 2 states that are really similar are stored in a cache, then these 2 states could only need let's say 310MB of cache memory (instead of 300 MB x 2 = 600 MB).

Before the commit, new states were regularly stored in the CheckpointStateCache. This cache has 10 slots.
After this cache is full, oldest values (not quite exactly because it's a LRU cache) are pruned.

With this commit, new states are quite rarely inserted into this cache. For example, on the today run, almost 5H (!) were needed before the first value was evicted from this cache. This mean this cache contains multiple states that do not share a lot of values with other states in all other caches/head.
==> A lot of fields stay in memory that are exclusively needed for the (old) states only present in this cache.

On the today run, I first started during multiple hours with the commit's behavior, then switched to the old behavior.
As soon as the old behavior started (22H15 on the following graph), I saw more evictions from this cache, and younger evicted values, and a corresponding memory pressure release.

That also explain why with the commit, the memory needs more time to stabilize.

[nalepae]

On this run, when "Time before switch" hits 0, the checkpoint state cache is wiped.