Excessive Memory Usage and Network Errors in below.service

[Rhevin]

Today I observed that the below daemon is consuming an unusually large amount of memory (60 GB). Additionally, the service repeatedly logs errors related to the network subsystem.

Is there any recommended way to debug this issue or identify the cause?

Service status and logs:

● below.service - below system monitor recording daemon
     Loaded: loaded (/etc/systemd/system/below.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-09-12 12:46:35 UTC; 3 weeks 4 days ago
   Main PID: 3669667 (below)
      Tasks: 8 (limit: 618572)
     Memory: 60.4G
        CPU: 3w 1d 4h 45min 54.348s
     CGroup: /system.slice/below.service
Oct 08 06:07:03 sample-host below[3669667]: Oct 08 06:06:52.095 ERRO Os { code: 22, kind: InvalidInput, message: "Invalid argument" }: "/sys/class/net"
Oct 08 06:07:03 sample-host below[3669667]: Oct 08 06:06:52.254 WARN data collection took 6603 ms (>= 1000 ms)
Oct 08 06:07:03 sample-host below[3669667]: Oct 08 06:07:02.227 ERRO Os { code: 22, kind: InvalidInput, message: "Invalid argument" }: "/sys/class/net"
Oct 08 06:07:03 sample-host below[3669667]: Oct 08 06:07:02.385 WARN data collection took 6734 ms (>= 1000 ms)
Oct 08 06:07:12 sample-host below[3669667]: Oct 08 06:07:12.179 ERRO Os { code: 22, kind: InvalidInput, message: "Invalid argument" }: "/sys/class/net"
Oct 08 06:07:12 sample-host below[3669667]: Oct 08 06:07:12.353 WARN data collection took 6702 ms (>= 1000 ms)

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

below          0.10.0-1     amd64        Interactive tool for viewing and recording historical system data

Any advice on how to further troubleshoot or resolve this memory and network error would be appreciated.

[lnyng]

d4f0be8 should fix the network subsystem error.

Regarding memory usage, could you share the composition of it from /sys/fs/cgroup/system.slice/below.service/memory.stat? If you restart below, how long does it take to accumulate the memory up to 60GB?

[Rhevin]

d4f0be8 should fix the network subsystem error.

Nice, Thanks for the update. I will check it.

After the service was restarted around 1 week ago, it's not reaching 60 GB now, only half of it

● below.service - below system monitor recording daemon
     Loaded: loaded (/etc/systemd/system/below.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-10-08 07:37:53 UTC; 2 weeks 1 day ago
   Main PID: 2545114 (below)
      Tasks: 8 (limit: 618572)
     Memory: 32.0G
        CPU: 1w 4d 6h 27min 21.307s
     CGroup: /system.slice/below.service
             └─2545114 /usr/bin/below record --retain-for-s 300 --interval-s 10 --disable-exitstats --disable-disk-stat --skew-detection-threshold-ms 1000

Oct 24 06:54:14 sample.host below[2545114]: Oct 24 06:54:14.572 ERRO Os { code: 22, kind: InvalidInput, message: "Invalid argument" }: "/sys/class/net"
Oct 24 06:54:14 sample.host below[2545114]: Oct 24 06:54:14.992 WARN data collection took 4423 ms (>= 1000 ms)
Oct 24 06:54:25 sample.host below[2545114]: Oct 24 06:54:25.097 ERRO Os { code: 22, kind: InvalidInput, message: "Invalid argument" }: "/sys/class/net"
Oct 24 06:54:25 sample.host below[2545114]: Oct 24 06:54:25.501 WARN data collection took 4932 ms (>= 1000 ms)
Oct 24 06:54:35 sample.host below[2545114]: Oct 24 06:54:35.557 ERRO Os { code: 22, kind: InvalidInput, message: "Invalid argument" }: "/sys/class/net"
Oct 24 06:54:35 sample.host below[2545114]: Oct 24 06:54:35.980 WARN data collection took 5410 ms (>= 1000 ms)
Oct 24 06:54:45 sample.host below[2545114]: Oct 24 06:54:45.771 ERRO Os { code: 22, kind: InvalidInput, message: "Invalid argument" }: "/sys/class/net"
Oct 24 06:54:46 sample.host below[2545114]: Oct 24 06:54:46.209 WARN data collection took 5640 ms (>= 1000 ms)
Oct 24 06:54:55 sample.host below[2545114]: Oct 24 06:54:55.136 ERRO Os { code: 22, kind: InvalidInput, message: "Invalid argument" }: "/sys/class/net"
Oct 24 06:54:55 sample.host below[2545114]: Oct 24 06:54:55.565 WARN data collection took 4995 ms (>= 1000 ms)

cat /sys/fs/cgroup/system.slice/below.service/memory.stat

anon 446357504
file 32762347520
kernel 1212788736
kernel_stack 114688
pagetables 995328
sec_pagetables 0
percpu 0
sock 0
vmalloc 0
shmem 0
zswap 0
zswapped 0
file_mapped 36864
file_dirty 38174720
file_writeback 0
swapcached 0
anon_thp 276824064
file_thp 0
shmem_thp 0
inactive_anon 446345216
active_anon 12288
inactive_file 26353364992
active_file 6408982528
unevictable 0
slab_reclaimable 1211580880
slab_unreclaimable 74408
slab 1211655288
workingset_refault_anon 0
workingset_refault_file 289551
workingset_activate_anon 0
workingset_activate_file 22405
workingset_restore_anon 0
workingset_restore_file 22035
workingset_nodereclaim 0
pgscan 757344912
pgsteal 757334721
pgscan_kswapd 412408534
pgscan_direct 344936378
pgsteal_kswapd 412401651
pgsteal_direct 344933070
pgfault 172283950
pgmajfault 266
pgrefill 170144014
pgactivate 225688875
pgdeactivate 170143246
pglazyfree 0
pglazyfreed 0
zswpin 0
zswpout 0
thp_fault_alloc 22281
thp_collapse_alloc 85