Description
Description
LMDB uses a lot of shared memory for caching, causing the resident set size to appear large, even when most of the additional memory can be instantly cleared for use if required. See:
- https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilization/
- https://blogs.kolabnow.com/2018/02/13/using-and-abusing-memory-with-lmdb-in-kube
- https://github.com/ledgerwatch/turbo-geth#1-htop-shows-incorrect-memory-usage
Present Behaviour
We report the size of the whole resident set as the amount of memory that Lighthouse is using (the same as htop's RES column).
Expected Behaviour
We should report the size of the resident set less the clean shared memory. As mentioned in the KolabNow link above, the value shown by pmap -x for Dirty memory is the relevant value. We just need a way of accessing this from Rust (TODO).
We shouldn't rush into just reporting RES - SHARED, because that's inaccurate, and could mislead users into thinking that Lighthouse's memory usage is lower than it actually is -- leading to confusion when an OOM or other memory issue occurs unexpectedly.