Resource leaks causing radiod to SEGV

[dbrooke]

I'm not sure if this is an inherent radiod problem or due to the way auto_rx uses it so I'm raising this here as I don't see any mention in the issues for ka9q-radio and also this is with the older commit as required for auto_rx.

Here's the content of the email I received from logcheck: radiod.log

This happens every couple of days since doing a clean native install of 1.8.2 on a Pi4 running bookworm (12) and after recovering from the restart the system works correctly until next time so the problem may not be noticed on installs without monitoring.

Munin indicates a leak of memory and also file handles/inodes.

[darksidelemm]

Interesting... I suspect I am probably seeing the same thing on my station, though perhaps not as often?
I see from systemctl that radiod has been up for 3 days, and is using 40% of my memory.

Looking at my logs, i am seeing radiod failing with:

Dec 30 11:52:22 sonderx-airspy radiod@airspymini-autorx[706169]: mirror_alloc 2nd mmap: Cannot allocate memory

Looking at the output from control sonde.local, it does look like auto_rx is correctly closing the channels it stops using, so at least we're not leaving a heap of unused channels active. I only have a few channels active:

darkside@sonderx-airspy:~$ control sonde.local
         SSRC    preset   samprate      freq, Hz   SNR output channel
       401500        fm     24,000   401,500,000  29.4 239.28.97.202:5004
     40050001        iq     48,000   400,500,000 -13.4 239.28.97.202:5004
     40150001        iq     48,000   401,500,000  32.0 239.28.97.202:5004
     40202503                    0   402,024,992   nan (null)
     40250001        iq     48,000   402,500,000  -9.9 239.28.97.202:5004
     40260001        iq     48,000   402,600,000  -inf 239.28.97.202:5004

Do you know if this issue was occurring on 1.8.1? Between 1.8.1 and 1.8.2 there have been changes to open more scan channels, but they all at least get closed again afterwards.
I guess is one of the differences between how auto_rx operates vs other ka9q-radio use-cases - we dynamically start and stop channels as part of the spectrum scan and detection processes, so there are channels being open and closed all the time.

@ka9q - any ideas on what could be causing the memory leak here? Is there any thing changed in the last few months that would have mitigated this? I think most of us are on commit e1224dcd1991637ba8e1caa68cd802e1b22933de which is from November last year, before the spectrum changes started.

Looking at lsof on one of my systems, I see a huuuuuuuge list of entries along the lines of:

radiod    2751252 2751255 fft                  radio  DEL       REG                0,1              3489643 /memfd:mirror_alloc
radiod    2751252 2751255 fft                  radio  DEL       REG                0,1              3491033 /memfd:mirror_alloc
radiod    2751252 2751255 fft                  radio  DEL       REG                0,1              3490087 /memfd:mirror_alloc

[helioshk]

I don't know if it is related to this issue, but since I upgraded the kernel from 6.12 (trixie) to 6.18 (trixie-backports), ka9q-radio seems to be triggering a memory leak in the kernel and after a few days it can't allocate any more memory (despite having 64 GB).

[darksidelemm]

The two machines I'm seeing this on are on kernel version 5.15 and 6.1.58, so I'm not sure its just a kernel version issue.

[helioshk]

Sorry, that was poorly worded. It's not a kernel issue. But the memory is allocated by the kernel when I use >6.12, so the only way to recover is to reboot.

[darksidelemm]

I've just reverted one of my stations to 1.8.1, and restarted radiod. radiod is back to only using 1.7% memory. I'll see how its looking in a few days, to hopefully see if it's a 1.8.1 vs 1.8.2 issue.

[dbrooke]

Do you know if this issue was occurring on 1.8.1?

I didn't notice it but I was late upgrading to 1.8.1 and only ran it for a short time.

[darksidelemm]

Running sudo lsof -c radiod | grep mirror_alloc | wc -l shows the number of handles radiod has open.

I can definitely see this increasing as auto_rx is running (and it was occurring on 1.8.1 as well, so i think it's an issue thats been around for a while).

It does seem to increase when new channels are started. I thought it might have been correlated with when 'powers' was run, but that didn't always seem to be the case.

[darksidelemm]

The number of files open definitely goes up when a new channel is opened.
e.g.

tune --samprate 48000 --mode iq --low -20000 --high 20000 --frequency 405510000 --ssrc 40551004 --radio sonde.local

When the channel is closed the number of files open does not decrease. We close the channel by setting the frequency for that ssrc to 0.

tune --samprate 48000 --mode iq --frequency 0 --ssrc 40551004 --radio sonde.local

I can confirm the channel disappears from the control sonde.local ssrc list when I run that command.

[dbrooke]

It does seem to increase when new channels are started. I thought it might have been correlated with when 'powers' was run, but that didn't always seem to be the case.

Yes, I see it incrementing by the number of peaks shown on the spectrum display. I average about 6 per scan so a system seeing fewer peaks would run for longer.