[defect]: Segmentation fault in multi-threaded mode with rlm_perl module for accounting

[ali2024e]

What type of defect/bug is this?

Crash or memory corruption (segv, abort, etc...)

How can the issue be reproduced?

I am encountering a segmentation fault every time I run the FreeRADIUS server in multi-threaded mode with the rlm_perl module used for accounting, this issue reproduce even with 10 requests per second. When the server is started in single-threaded mode, the issue does not occur. However, running in multi-threaded mode is necessary for performance reasons.

Steps to Reproduce:
Configure FreeRADIUS to use the rlm_perl module for accounting.
Start the FreeRADIUS server in multi-threaded mode.
Perform accounting operations that invoke the rlm_perl module.
Observe the segmentation fault.

Expected Behavior:
The FreeRADIUS server should operate correctly in multi-threaded mode without causing a segmentation fault.

Log output from the FreeRADIUS daemon

Info: Core dumps are enabled
Warning: Ignoring "response_window = 20.000000", forcing to "response_window = 5.000000"
Warning: Ignoring "response_window = 6.000000", forcing to "response_window = 5.000000"
Info: Debugger not attached
Info: Perl version: 5.26.0
..............
CAUGHT SIGNAL: Segmentation fault
Calling: gdb -silent -x /etc/raddb_accounting/panic.gdb /usr/sbin/radiusd 5077 2>&1 | tee /var/log/radius_accounting/gdb-radiusd-5077.log
Panic action exited with 256
CAUGHT SIGNAL: Segmentation fault
Calling: gdb -silent -x /etc/raddb_accounting/panic.gdb /usr/sbin/radiusd 5280 2>&1 | tee /var/log/radius_accounting/gdb-radiusd-5280.log
Panic action exited with 256
CAUGHT SIGNAL: Segmentation fault
Calling: gdb -silent -x /etc/raddb_accounting/panic.gdb /usr/sbin/radiusd 5670 2>&1 | tee /var/log/radius_accounting/gdb-radiusd-5670.log
Panic action exited with 256

Relevant log output from client utilities

No response

Backtrace from LLDB or GDB

Reading symbols from /usr/sbin/radiusd...
Reading symbols from /usr/lib/debug/usr/sbin/radiusd-3.0.25-150400.4.4.1.x86_64.debug...
Attaching to program: /usr/sbin/radiusd, process 5670
[New LWP 5676]
[New LWP 5677]
[New LWP 5678]
[New LWP 5679]
[New LWP 5680]
[New LWP 5681]
[New LWP 5682]
[New LWP 5683]
[New LWP 5684]
[New LWP 5686]
[New LWP 5687]
[New LWP 5688]
[New LWP 5689]
[New LWP 5690]
[New LWP 5691]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fbf2250176f in wait4 () from /lib64/libc.so.6
No symbol table info available.
No symbol table info available.

Thread 16 (Thread 0x7fbf127fc700 (LWP 5691) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc8ab6d0) at src/main/threads.c:755
        self = 0x5602fc8ab6d0
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7fbf12ffd700 (LWP 5690) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc873bc0) at src/main/threads.c:755
        self = 0x5602fc873bc0
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 14 (Thread 0x7fbf137fe700 (LWP 5689) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc881f30) at src/main/threads.c:755
        self = 0x5602fc881f30
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7fbf13fff700 (LWP 5688) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc8a8d70) at src/main/threads.c:755
        self = 0x5602fc8a8d70
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7fbf1886f700 (LWP 5687) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fbfc7e20) at src/main/threads.c:755
        self = 0x5602fbfc7e20
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7fbf1e15d700 (LWP 5686) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fbfa8f60) at src/main/threads.c:755
        self = 0x5602fbfa8f60
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7fbf19070700 (LWP 5684) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc8aba30) at src/main/threads.c:755
        self = 0x5602fc8aba30
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7fbf19871700 (LWP 5683) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc8ab8b0) at src/main/threads.c:755
        self = 0x5602fc8ab8b0
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7fbf1a072700 (LWP 5682) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc8ab730) at src/main/threads.c:755
        self = 0x5602fc8ab730
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7fbf1a8f4700 (LWP 5681) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc88ad60) at src/main/threads.c:755
        self = 0x5602fc88ad60
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7fbf1b0f5700 (LWP 5680) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc21d4d0) at src/main/threads.c:755
        self = 0x5602fc21d4d0
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7fbf1b8f6700 (LWP 5679) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fc7bfa10) at src/main/threads.c:755
        self = 0x5602fc7bfa10
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7fbf1c159700 (LWP 5678) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fbf94ba0) at src/main/threads.c:755
        self = 0x5602fbf94ba0
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7fbf1c95a700 (LWP 5677) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fbf9d000) at src/main/threads.c:755
        self = 0x5602fbf9d000
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7fbf1d15b700 (LWP 5676) "radiusd"):
#0  0x00007fbf22a635f4 in do_futex_wait.constprop () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fbf22a636f8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00005602f9f278c0 in request_handler_thread (arg=0x5602fbfa4930) at src/main/threads.c:755
        self = 0x5602fbfa4930
        re_wait = <optimized out>
#3  0x00007fbf22a596ea in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fbf22537a8f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7fbf23a03040 (LWP 5670) "radiusd"):
#0  0x00007fbf2250176f in wait4 () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007fbf22478bc7 in do_system () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007fbf23a2497f in fr_fault () from /usr/lib64/freeradius/libfreeradius-radius.so
No symbol table info available.
#3  0x00007fbf23a24a8f in ?? () from /usr/lib64/freeradius/libfreeradius-radius.so
No symbol table info available.
#4  0x00007fbf22e7a568 in _talloc_free () from /usr/lib64/libtalloc.so.2
No symbol table info available.
#5  0x00007fbf23a40d44 in fr_event_delete () from /usr/lib64/freeradius/libfreeradius-radius.so
No symbol table info available.
#6  0x00007fbf23a40f9e in fr_event_run () from /usr/lib64/freeradius/libfreeradius-radius.so
No symbol table info available.
#7  0x00007fbf23a4140f in fr_event_loop () from /usr/lib64/freeradius/libfreeradius-radius.so
No symbol table info available.
#8  0x00005602f9f30b21 in radius_event_process () at src/main/process.c:6305
No locals.
#9  0x00005602f9f07f05 in main (argc=<optimized out>, argv=<optimized out>) at src/main/radiusd.c:641
        rcode = 0
        status = <optimized out>
        argval = <optimized out>
        spawn_flag = true
        display_version = false
        flag = 0
        from_child = {-1, -1}
        p = <optimized out>
        state = 0x5602f9f6cc60 <global_state>
        autofree = 0x5602fbcc6b60
A debugging session is active.

        Inferior 1 [process 5670] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
[Inferior 1 (process 5670) detached]

[alandekok]

Which version of FreeRADIUS are you using? Can you try again with the current v3.2.x branch?

What are you doing with the Perl scripts?

Can you try with a simple Perl script that does nothing? i.e. if the problem can be narrowed down to not "just Perl", but some unusual corner case of how Perl is being used, that would help a lot in understanding and fixing the issue.

Many people are using Perl, and no one else has reported this issue. So the problem is likely unusual.