slow lc_lib:get_sys_memory_usage() with lots of ports

[qzhuyan]

Description

We get report that during 100M connections tests, the function lc_lib:get_sys_memory_usage() is very slow as following.

iex(emqx@emqx-6.int.thales)9> :timer.tc(&:load_ctl.get_sys_memory/0)
{9624, {0.8642148373449763, 134799187968}}
iex(emqx@emqx-6.int.thales)10> :timer.tc(&:load_ctl.get_sys_memory/0)
{717, {0.8661882218141999, 134799187968}}
iex(emqx@emqx-6.int.thales)11> :timer.tc(&:load_ctl.get_sys_memory/0)
{609, {0.8671846375495222, 134799187968}}
iex(emqx@emqx-6.int.thales)12> :timer.tc(&:load_ctl.get_sys_memory/0)
{4690, {0.8890156894153435, 134799187968}}
iex(emqx@emqx-6.int.thales)13> :timer.tc(&:load_ctl.get_sys_memory/0)
{617, {0.890001986632739, 134799187968}}
iex(emqx@emqx-6.int.thales)14> :timer.tc(&:load_ctl.get_sys_memory/0)
{64898181, {0.8134260153112319, 134799187968}}

Impacted nodes

10/20 nodes runs into this issue.

Read from shell

Reading Cgroup/ cgroupv2/ or /proc/meminfo from shell is fast.

runq length:

iex(emqx@emqx-6.int.thales)18> :erlang.statistics(:run_queue_lengths) |> IO.inspect(limit: :infinity)
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

We suspect it relates to the large number of erlang ports. (~5M / node)

cause

Likely it is caused by memsup

iex(emqx@emqx-6.int.thales)18> :timer.tc(&:memsup.get_system_memory_data/0)
{13702147,
 [
   available_memory: 86003736576,
   buffered_memory: 119590912,
   cached_memory: 4353019904,
   free_memory: 81441886208,
   free_swap: 0,
   system_total_memory: 134799183872,
   total_memory: 134799183872,
   total_swap: 0
 ]}