[defect]: Segmentation fault with check_client_connections

[gk-fschubert]

What type of defect/bug is this?

Crash or memory corruption (segv, abort, etc...)

How can the issue be reproduced?

Enable the site tls and set check_client_connections in it to yes.

Then init and close some tls sessions until the server crashes.

Log output from the FreeRADIUS daemon

FreeRADIUS Version 3.2.7
Copyright (C) 1999-2023 The FreeRADIUS server project and contributors
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE
You may redistribute copies of FreeRADIUS under the terms of the
GNU General Public License
For more information about these matters, see the file named COPYRIGHT
Starting - reading configuration files ...
including dictionary file /usr/local/share/freeradius/dictionary
including dictionary file /usr/local/share/freeradius/dictionary.dhcp
including dictionary file /usr/local/share/freeradius/dictionary.vqp
including dictionary file /usr/local/etc/raddb/dictionary
including configuration file /usr/local/etc/raddb/radiusd.conf
including configuration file /usr/local/etc/raddb/proxy.conf
including configuration file /usr/local/etc/raddb/clients.conf
including files in directory /usr/local/etc/raddb/mods-enabled/
including configuration file /usr/local/etc/raddb/mods-enabled/always
including configuration file /usr/local/etc/raddb/mods-enabled/attr_filter
including configuration file /usr/local/etc/raddb/mods-enabled/chap
including configuration file /usr/local/etc/raddb/mods-enabled/date
including configuration file /usr/local/etc/raddb/mods-enabled/detail
including configuration file /usr/local/etc/raddb/mods-enabled/detail.log
including configuration file /usr/local/etc/raddb/mods-enabled/digest
including configuration file /usr/local/etc/raddb/mods-enabled/dynamic_clients
including configuration file /usr/local/etc/raddb/mods-enabled/eap
including configuration file /usr/local/etc/raddb/mods-enabled/echo
including configuration file /usr/local/etc/raddb/mods-enabled/exec
including configuration file /usr/local/etc/raddb/mods-enabled/expiration
including configuration file /usr/local/etc/raddb/mods-enabled/expr
including configuration file /usr/local/etc/raddb/mods-enabled/files
including configuration file /usr/local/etc/raddb/mods-enabled/linelog
including configuration file /usr/local/etc/raddb/mods-enabled/logintime
including configuration file /usr/local/etc/raddb/mods-enabled/mschap
including configuration file /usr/local/etc/raddb/mods-enabled/ntlm_auth
including configuration file /usr/local/etc/raddb/mods-enabled/pap
including configuration file /usr/local/etc/raddb/mods-enabled/passwd
including configuration file /usr/local/etc/raddb/mods-enabled/preprocess
including configuration file /usr/local/etc/raddb/mods-enabled/radutmp
including configuration file /usr/local/etc/raddb/mods-enabled/realm
including configuration file /usr/local/etc/raddb/mods-enabled/replicate
including configuration file /usr/local/etc/raddb/mods-enabled/soh
including configuration file /usr/local/etc/raddb/mods-enabled/sradutmp
including configuration file /usr/local/etc/raddb/mods-enabled/totp
including configuration file /usr/local/etc/raddb/mods-enabled/unix
including configuration file /usr/local/etc/raddb/mods-enabled/unpack
including configuration file /usr/local/etc/raddb/mods-enabled/utf8
including files in directory /usr/local/etc/raddb/policy.d/
including configuration file /usr/local/etc/raddb/policy.d/abfab-tr
including configuration file /usr/local/etc/raddb/policy.d/accounting
including configuration file /usr/local/etc/raddb/policy.d/canonicalization
including configuration file /usr/local/etc/raddb/policy.d/control
including configuration file /usr/local/etc/raddb/policy.d/cui
including configuration file /usr/local/etc/raddb/policy.d/debug
including configuration file /usr/local/etc/raddb/policy.d/dhcp
including configuration file /usr/local/etc/raddb/policy.d/eap
including configuration file /usr/local/etc/raddb/policy.d/filter
including configuration file /usr/local/etc/raddb/policy.d/moonshot-targeted-ids
including configuration file /usr/local/etc/raddb/policy.d/operator-name
including configuration file /usr/local/etc/raddb/policy.d/rfc7542
including files in directory /usr/local/etc/raddb/sites-enabled/
including configuration file /usr/local/etc/raddb/sites-enabled/default
including configuration file /usr/local/etc/raddb/sites-enabled/inner-tunnel
including configuration file /usr/local/etc/raddb/sites-enabled/tls
main {
 security {
 	allow_core_dumps = yes
 }
	name = "radiusd"
	prefix = "/usr/local"
	localstatedir = "/usr/local/var"
	logdir = "/usr/local/var/log/radius"
	run_dir = "/usr/local/var/run/radiusd"
}
Core dumps are enabled
main {
	name = "radiusd"
	prefix = "/usr/local"
	localstatedir = "/usr/local/var"
	sbindir = "/usr/local/sbin"
	logdir = "/usr/local/var/log/radius"
	run_dir = "/usr/local/var/run/radiusd"
	libdir = "/usr/local/lib"
	radacctdir = "/usr/local/var/log/radius/radacct"
	panic_action = "gdb -silent -x /etc/raddb/panic.gdb %e %p 2>&1 | tee /usr/local/var/log/radius/gdb-radiusd-%p.log"
	hostname_lookups = no
	max_request_time = 30
	proxy_dedup_window = 1
	cleanup_delay = 5
	max_requests = 16384
	max_fds = 512
	postauth_client_lost = no
	pidfile = "/usr/local/var/run/radiusd/radiusd.pid"
	checkrad = "/usr/local/sbin/checkrad"
	debug_level = 0
	proxy_requests = yes
 log {
 	stripped_names = no
 	auth = no
 	auth_badpass = no
 	auth_goodpass = no
 	colourise = yes
 	msg_denied = "You are already logged in - access denied"
 }
 resources {
 }
 security {
 	max_attributes = 200
 	reject_delay = 1.000000
 	status_server = yes
 	require_message_authenticator = "auto"
 	limit_proxy_state = "auto"
 	allow_vulnerable_openssl = "no"
 }
 unlang {
 	group_stop_return = no
 	policy_stop_return = no
 }
}
radiusd: #### Loading Realms and Home Servers ####
 proxy server {
 	retry_delay = 5
 	retry_count = 3
 	default_fallback = no
 	dead_time = 120
 	wake_all_if_all_dead = no
 }
 home_server localhost {
 	nonblock = no
 	ipaddr = 127.0.0.1
 	port = 1812
 	type = "auth"
 	secret = <<< secret >>>
 	response_window = 20.000000
 	response_timeouts = 1
 	max_outstanding = 65536
 	zombie_period = 40
 	status_check = "status-server"
 	ping_interval = 30
 	check_interval = 30
 	check_timeout = 4
 	num_answers_to_alive = 3
 	revive_interval = 120
  limit {
  	max_connections = 16
  	max_requests = 0
  	lifetime = 0
  	idle_timeout = 0
  }
  coa {
  	irt = 2
  	mrt = 16
  	mrc = 5
  	mrd = 30
  }
 }
 home_server tls {
 	nonblock = no
 	ipaddr = 127.0.0.1
 	port = 2083
 	type = "auth"
 	proto = "tcp"
 	secret = <<< secret >>>
 	response_window = 30.000000
 	response_timeouts = 1
 	max_outstanding = 65536
 	zombie_period = 40
 	status_check = "none"
 	ping_interval = 30
 	check_timeout = 4
 	num_answers_to_alive = 3
 	revive_interval = 300
  limit {
  	max_connections = 16
  	max_requests = 0
  	lifetime = 0
  	idle_timeout = 0
  }
  coa {
  	irt = 2
  	mrt = 16
  	mrc = 5
  	mrd = 30
  }
 }
  tls {
  	verify_depth = 0
  	ca_path = "/usr/local/etc/raddb/certs"
  	pem_file_type = yes
  	private_key_file = "/usr/local/etc/raddb/certs/client.pem"
  	certificate_file = "/usr/local/etc/raddb/certs/client.pem"
  	ca_file = "/usr/local/etc/raddb/certs/ca.pem"
  	private_key_password = <<< secret >>>
  	fragment_size = 8192
  	include_length = yes
  	check_crl = no
  	cipher_list = "DEFAULT"
  	ca_path_reload_interval = 0
  	ecdh_curve = "prime256v1"
  	tls_min_version = "1.2"
  }
  	connect_timeout = 30
 home_server_pool my_auth_failover {
	type = fail-over
	home_server = localhost
 }
 realm example.com {
	auth_pool = my_auth_failover
 }
 realm LOCAL {
 }
 home_server_pool tls {
	type = fail-over
	home_server = tls
 }
 realm tls {
	auth_pool = tls
 }
radiusd: #### Loading Clients ####
 client localhost {
 	ipaddr = 127.0.0.1
 	secret = <<< secret >>>
 	nas_type = "other"
 	proto = "*"
  limit {
  	max_connections = 16
  	lifetime = 0
  	idle_timeout = 900
  }
 }
Shared secret for client localhost is short, and likely can be broken by an attacker.
 client localhost_ipv6 {
 	ipv6addr = ::1
 	secret = <<< secret >>>
  limit {
  	max_connections = 16
  	lifetime = 0
  	idle_timeout = 30
  }
 }
Shared secret for client localhost_ipv6 is short, and likely can be broken by an attacker.
Debugger not attached
Configuration version: f480-c871-7ada-22bf
 # Creating Auth-Type = mschap
 # Creating Auth-Type = digest
 # Creating Auth-Type = eap
 # Creating Auth-Type = PAP
 # Creating Auth-Type = CHAP
 # Creating Auth-Type = MS-CHAP
 # Creating Autz-Type = New-TLS-Connection
radiusd: #### Instantiating modules ####
 modules {
  # Loaded module rlm_always
  # Loading module "reject" from file /usr/local/etc/raddb/mods-enabled/always
  always reject {
  	rcode = "reject"
  	simulcount = 0
  	mpp = no
  }
  # Loading module "fail" from file /usr/local/etc/raddb/mods-enabled/always
  always fail {
  	rcode = "fail"
  	simulcount = 0
  	mpp = no
  }
  # Loading module "ok" from file /usr/local/etc/raddb/mods-enabled/always
  always ok {
  	rcode = "ok"
  	simulcount = 0
  	mpp = no
  }
  # Loading module "handled" from file /usr/local/etc/raddb/mods-enabled/always
  always handled {
  	rcode = "handled"
  	simulcount = 0
  	mpp = no
  }
  # Loading module "invalid" from file /usr/local/etc/raddb/mods-enabled/always
  always invalid {
  	rcode = "invalid"
  	simulcount = 0
  	mpp = no
  }
  # Loading module "userlock" from file /usr/local/etc/raddb/mods-enabled/always
  always userlock {
  	rcode = "userlock"
  	simulcount = 0
  	mpp = no
  }
  # Loading module "notfound" from file /usr/local/etc/raddb/mods-enabled/always
  always notfound {
  	rcode = "notfound"
  	simulcount = 0
  	mpp = no
  }
  # Loading module "noop" from file /usr/local/etc/raddb/mods-enabled/always
  always noop {
  	rcode = "noop"
  	simulcount = 0
  	mpp = no
  }
  # Loading module "updated" from file /usr/local/etc/raddb/mods-enabled/always
  always updated {
  	rcode = "updated"
  	simulcount = 0
  	mpp = no
  }
  # Loaded module rlm_attr_filter
  # Loading module "attr_filter.post-proxy" from file /usr/local/etc/raddb/mods-enabled/attr_filter
  attr_filter attr_filter.post-proxy {
  	filename = "/usr/local/etc/raddb/mods-config/attr_filter/post-proxy"
  	key = "%{Realm}"
  	relaxed = no
  }
  # Loading module "attr_filter.pre-proxy" from file /usr/local/etc/raddb/mods-enabled/attr_filter
  attr_filter attr_filter.pre-proxy {
  	filename = "/usr/local/etc/raddb/mods-config/attr_filter/pre-proxy"
  	key = "%{Realm}"
  	relaxed = no
  }
  # Loading module "attr_filter.access_reject" from file /usr/local/etc/raddb/mods-enabled/attr_filter
  attr_filter attr_filter.access_reject {
  	filename = "/usr/local/etc/raddb/mods-config/attr_filter/access_reject"
  	key = "%{User-Name}"
  	relaxed = no
  }
  # Loading module "attr_filter.access_challenge" from file /usr/local/etc/raddb/mods-enabled/attr_filter
  attr_filter attr_filter.access_challenge {
  	filename = "/usr/local/etc/raddb/mods-config/attr_filter/access_challenge"
  	key = "%{User-Name}"
  	relaxed = no
  }
  # Loading module "attr_filter.accounting_response" from file /usr/local/etc/raddb/mods-enabled/attr_filter
  attr_filter attr_filter.accounting_response {
  	filename = "/usr/local/etc/raddb/mods-config/attr_filter/accounting_response"
  	key = "%{User-Name}"
  	relaxed = no
  }
  # Loading module "attr_filter.coa" from file /usr/local/etc/raddb/mods-enabled/attr_filter
  attr_filter attr_filter.coa {
  	filename = "/usr/local/etc/raddb/mods-config/attr_filter/coa"
  	key = "%{User-Name}"
  	relaxed = no
  }
  # Loaded module rlm_chap
  # Loading module "chap" from file /usr/local/etc/raddb/mods-enabled/chap
  # Loaded module rlm_date
  # Loading module "date" from file /usr/local/etc/raddb/mods-enabled/date
  date {
  	format = "%b %e %Y %H:%M:%S %Z"
  	utc = no
  }
  # Loading module "wispr2date" from file /usr/local/etc/raddb/mods-enabled/date
  date wispr2date {
  	format = "%Y-%m-%dT%H:%M:%S"
  	utc = no
  }
  # Loaded module rlm_detail
  # Loading module "detail" from file /usr/local/etc/raddb/mods-enabled/detail
  detail {
  	filename = "/usr/local/var/log/radius/radacct/%{%{Packet-Src-IP-Address}:-%{Packet-Src-IPv6-Address}}/detail-%Y%m%d"
  	header = "%t"
  	permissions = 384
  	locking = no
  	dates_as_integer = no
  	escape_filenames = no
  	log_packet_header = no
  }
  # Loading module "auth_log" from file /usr/local/etc/raddb/mods-enabled/detail.log
  detail auth_log {
  	filename = "/usr/local/var/log/radius/radacct/%{%{Packet-Src-IP-Address}:-%{Packet-Src-IPv6-Address}}/auth-detail-%Y%m%d"
  	header = "%t"
  	permissions = 384
  	locking = no
  	dates_as_integer = no
  	escape_filenames = no
  	log_packet_header = no
  }
  # Loading module "reply_log" from file /usr/local/etc/raddb/mods-enabled/detail.log
  detail reply_log {
  	filename = "/usr/local/var/log/radius/radacct/%{%{Packet-Src-IP-Address}:-%{Packet-Src-IPv6-Address}}/reply-detail-%Y%m%d"
  	header = "%t"
  	permissions = 384
  	locking = no
  	dates_as_integer = no
  	escape_filenames = no
  	log_packet_header = no
  }
  # Loading module "pre_proxy_log" from file /usr/local/etc/raddb/mods-enabled/detail.log
  detail pre_proxy_log {
  	filename = "/usr/local/var/log/radius/radacct/%{%{Packet-Src-IP-Address}:-%{Packet-Src-IPv6-Address}}/pre-proxy-detail-%Y%m%d"
  	header = "%t"
  	permissions = 384
  	locking = no
  	dates_as_integer = no
  	escape_filenames = no
  	log_packet_header = no
  }
  # Loading module "post_proxy_log" from file /usr/local/etc/raddb/mods-enabled/detail.log
  detail post_proxy_log {
  	filename = "/usr/local/var/log/radius/radacct/%{%{Packet-Src-IP-Address}:-%{Packet-Src-IPv6-Address}}/post-proxy-detail-%Y%m%d"
  	header = "%t"
  	permissions = 384
  	locking = no
  	dates_as_integer = no
  	escape_filenames = no
  	log_packet_header = no
  }
  # Loaded module rlm_digest
  # Loading module "digest" from file /usr/local/etc/raddb/mods-enabled/digest
  # Loaded module rlm_dynamic_clients
  # Loading module "dynamic_clients" from file /usr/local/etc/raddb/mods-enabled/dynamic_clients
  # Loaded module rlm_eap
  # Loading module "eap" from file /usr/local/etc/raddb/mods-enabled/eap
  eap {
  	default_eap_type = "md5"
  	timer_expire = 60
  	max_eap_type = 52
  	ignore_unknown_eap_types = no
  	cisco_accounting_username_bug = no
  	max_sessions = 16384
  	dedup_key = ""
  }
  # Loaded module rlm_exec
  # Loading module "echo" from file /usr/local/etc/raddb/mods-enabled/echo
  exec echo {
  	wait = yes
  	program = "/bin/echo %{User-Name}"
  	input_pairs = "request"
  	output_pairs = "reply"
  	shell_escape = yes
  }
  # Loading module "exec" from file /usr/local/etc/raddb/mods-enabled/exec
  exec {
  	wait = no
  	input_pairs = "request"
  	shell_escape = yes
  	timeout = 10
  }
  # Loaded module rlm_expiration
  # Loading module "expiration" from file /usr/local/etc/raddb/mods-enabled/expiration
  # Loaded module rlm_expr
  # Loading module "expr" from file /usr/local/etc/raddb/mods-enabled/expr
  expr {
  	safe_characters = "@abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.-_: /äéöüàâæçèéêëîïôœùûüaÿÄÉÖÜßÀÂÆÇÈÉÊËÎÏÔŒÙÛÜŸ"
  }
  # Loaded module rlm_files
  # Loading module "files" from file /usr/local/etc/raddb/mods-enabled/files
  files {
  	filename = "/usr/local/etc/raddb/mods-config/files/authorize"
  	acctusersfile = "/usr/local/etc/raddb/mods-config/files/accounting"
  	preproxy_usersfile = "/usr/local/etc/raddb/mods-config/files/pre-proxy"
  }
  # Loaded module rlm_linelog
  # Loading module "linelog" from file /usr/local/etc/raddb/mods-enabled/linelog
  linelog {
  	filename = "/usr/local/var/log/radius/linelog"
  	escape_filenames = no
  	syslog_severity = "info"
  	permissions = 384
  	format = "This is a log message for %{User-Name}"
  	reference = "messages.%{%{reply:Packet-Type}:-default}"
  }
  # Loading module "log_accounting" from file /usr/local/etc/raddb/mods-enabled/linelog
  linelog log_accounting {
  	filename = "/usr/local/var/log/radius/linelog-accounting"
  	escape_filenames = no
  	syslog_severity = "info"
  	permissions = 384
  	format = ""
  	reference = "Accounting-Request.%{%{Acct-Status-Type}:-unknown}"
  }
  # Loaded module rlm_logintime
  # Loading module "logintime" from file /usr/local/etc/raddb/mods-enabled/logintime
  logintime {
  	minimum_timeout = 60
  }
  # Loaded module rlm_mschap
  # Loading module "mschap" from file /usr/local/etc/raddb/mods-enabled/mschap
  mschap {
  	use_mppe = yes
  	require_encryption = no
  	require_strong = no
  	with_ntdomain_hack = yes
   passchange {
   }
  	allow_retry = yes
  	winbind_retry_with_normalised_username = no
  }
  # Loading module "ntlm_auth" from file /usr/local/etc/raddb/mods-enabled/ntlm_auth
  exec ntlm_auth {
  	wait = yes
  	program = "/path/to/ntlm_auth --request-nt-key --domain=MYDOMAIN --username=%{mschap:User-Name} --password=%{User-Password}"
  	shell_escape = yes
  }
  # Loaded module rlm_pap
  # Loading module "pap" from file /usr/local/etc/raddb/mods-enabled/pap
  pap {
  	normalise = yes
  }
  # Loaded module rlm_passwd
  # Loading module "etc_passwd" from file /usr/local/etc/raddb/mods-enabled/passwd
  passwd etc_passwd {
  	filename = "/etc/passwd"
  	format = "*User-Name:Crypt-Password:"
  	delimiter = ":"
  	ignore_nislike = no
  	ignore_empty = yes
  	allow_multiple_keys = no
  	hash_size = 100
  }
  # Loaded module rlm_preprocess
  # Loading module "preprocess" from file /usr/local/etc/raddb/mods-enabled/preprocess
  preprocess {
  	huntgroups = "/usr/local/etc/raddb/mods-config/preprocess/huntgroups"
  	hints = "/usr/local/etc/raddb/mods-config/preprocess/hints"
  	with_ascend_hack = no
  	ascend_channels_per_line = 23
  	with_ntdomain_hack = no
  	with_specialix_jetstream_hack = no
  	with_cisco_vsa_hack = no
  	with_alvarion_vsa_hack = no
  }
  # Loaded module rlm_radutmp
  # Loading module "radutmp" from file /usr/local/etc/raddb/mods-enabled/radutmp
  radutmp {
  	filename = "/usr/local/var/log/radius/radutmp"
  	username = "%{User-Name}"
  	case_sensitive = yes
  	check_with_nas = yes
  	permissions = 384
  	caller_id = yes
  }
  # Loaded module rlm_realm
  # Loading module "IPASS" from file /usr/local/etc/raddb/mods-enabled/realm
  realm IPASS {
  	format = "prefix"
  	delimiter = "/"
  	ignore_default = no
  	ignore_null = no
  }
  # Loading module "suffix" from file /usr/local/etc/raddb/mods-enabled/realm
  realm suffix {
  	format = "suffix"
  	delimiter = "@"
  	ignore_default = no
  	ignore_null = no
  }
  # Loading module "bangpath" from file /usr/local/etc/raddb/mods-enabled/realm
  realm bangpath {
  	format = "prefix"
  	delimiter = "!"
  	ignore_default = no
  	ignore_null = no
  }
  # Loading module "realmpercent" from file /usr/local/etc/raddb/mods-enabled/realm
  realm realmpercent {
  	format = "suffix"
  	delimiter = "%"
  	ignore_default = no
  	ignore_null = no
  }
  # Loading module "ntdomain" from file /usr/local/etc/raddb/mods-enabled/realm
  realm ntdomain {
  	format = "prefix"
  	delimiter = "\"
  	ignore_default = no
  	ignore_null = no
  }
  # Loaded module rlm_replicate
  # Loading module "replicate" from file /usr/local/etc/raddb/mods-enabled/replicate
  # Loaded module rlm_soh
  # Loading module "soh" from file /usr/local/etc/raddb/mods-enabled/soh
  soh {
  	dhcp = yes
  }
  # Loading module "sradutmp" from file /usr/local/etc/raddb/mods-enabled/sradutmp
  radutmp sradutmp {
  	filename = "/usr/local/var/log/radius/sradutmp"
  	username = "%{User-Name}"
  	case_sensitive = yes
  	check_with_nas = yes
  	permissions = 420
  	caller_id = no
  }
  # Loaded module rlm_totp
  # Loading module "totp" from file /usr/local/etc/raddb/mods-enabled/totp
  totp {
  	time_step = 30
  	otp_length = 6
  	lookback_steps = 1
  	lookback_interval = 30
  	lookforward_steps = 0
  }
  # Loaded module rlm_unix
  # Loading module "unix" from file /usr/local/etc/raddb/mods-enabled/unix
  unix {
  	radwtmp = "/usr/local/var/log/radius/radwtmp"
  }
Creating attribute Unix-Group
  # Loaded module rlm_unpack
  # Loading module "unpack" from file /usr/local/etc/raddb/mods-enabled/unpack
  # Loaded module rlm_utf8
  # Loading module "utf8" from file /usr/local/etc/raddb/mods-enabled/utf8
  instantiate {
  }
  # Instantiating module "reject" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "fail" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "ok" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "handled" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "invalid" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "userlock" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "notfound" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "noop" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "updated" from file /usr/local/etc/raddb/mods-enabled/always
  # Instantiating module "attr_filter.post-proxy" from file /usr/local/etc/raddb/mods-enabled/attr_filter
reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/post-proxy
  # Instantiating module "attr_filter.pre-proxy" from file /usr/local/etc/raddb/mods-enabled/attr_filter
reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/pre-proxy
  # Instantiating module "attr_filter.access_reject" from file /usr/local/etc/raddb/mods-enabled/attr_filter
reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/access_reject
  # Instantiating module "attr_filter.access_challenge" from file /usr/local/etc/raddb/mods-enabled/attr_filter
reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/access_challenge
  # Instantiating module "attr_filter.accounting_response" from file /usr/local/etc/raddb/mods-enabled/attr_filter
reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/accounting_response
  # Instantiating module "attr_filter.coa" from file /usr/local/etc/raddb/mods-enabled/attr_filter
reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/coa
  # Instantiating module "detail" from file /usr/local/etc/raddb/mods-enabled/detail
  # Instantiating module "auth_log" from file /usr/local/etc/raddb/mods-enabled/detail.log
rlm_detail (auth_log): 'User-Password' suppressed, will not appear in detail output
  # Instantiating module "reply_log" from file /usr/local/etc/raddb/mods-enabled/detail.log
  # Instantiating module "pre_proxy_log" from file /usr/local/etc/raddb/mods-enabled/detail.log
  # Instantiating module "post_proxy_log" from file /usr/local/etc/raddb/mods-enabled/detail.log
  # Instantiating module "eap" from file /usr/local/etc/raddb/mods-enabled/eap
   # Linked to sub-module rlm_eap_md5
   # Linked to sub-module rlm_eap_gtc
   gtc {
   	challenge = "Password: "
   	auth_type = "PAP"
   }
   # Linked to sub-module rlm_eap_tls
   tls {
   	tls = "tls-common"
   }
   tls-config tls-common {
   	verify_depth = 0
   	ca_path = "/usr/local/etc/raddb/certs"
   	pem_file_type = yes
   	private_key_file = "/usr/local/etc/raddb/certs/server.pem"
   	certificate_file = "/usr/local/etc/raddb/certs/server.pem"
   	ca_file = "/usr/local/etc/raddb/certs/ca.pem"
   	private_key_password = <<< secret >>>
   	fragment_size = 1024
   	include_length = yes
   	auto_chain = yes
   	check_crl = no
   	check_all_crl = no
   	ca_path_reload_interval = 0
   	cipher_list = "DEFAULT"
   	cipher_server_preference = no
   	reject_unknown_intermediate_ca = no
   	ecdh_curve = ""
   	tls_max_version = "1.2"
   	tls_min_version = "1.2"
    cache {
    	enable = no
    	lifetime = 24
    	max_entries = 255
    }
    verify {
    	skip_if_ocsp_ok = no
    }
    ocsp {
    	enable = no
    	override_cert_url = yes
    	url = "http://127.0.0.1/ocsp/"
    	use_nonce = yes
    	timeout = 0
    	softfail = no
    }
   }
   # Linked to sub-module rlm_eap_ttls
   ttls {
   	tls = "tls-common"
   	default_eap_type = "md5"
   	copy_request_to_tunnel = no
   	use_tunneled_reply = no
   	virtual_server = "inner-tunnel"
   	include_length = yes
   	require_client_cert = no
   }
tls: Using cached TLS configuration from previous invocation
   # Linked to sub-module rlm_eap_peap
   peap {
   	tls = "tls-common"
   	default_eap_type = "mschapv2"
   	copy_request_to_tunnel = no
   	use_tunneled_reply = no
   	proxy_tunneled_request_as_eap = yes
   	virtual_server = "inner-tunnel"
   	soh = no
   	require_client_cert = no
   }
tls: Using cached TLS configuration from previous invocation
   # Linked to sub-module rlm_eap_mschapv2
   mschapv2 {
   	with_ntdomain_hack = no
   	send_error = no
   }
  # Instantiating module "expiration" from file /usr/local/etc/raddb/mods-enabled/expiration
  # Instantiating module "files" from file /usr/local/etc/raddb/mods-enabled/files
reading pairlist file /usr/local/etc/raddb/mods-config/files/authorize
reading pairlist file /usr/local/etc/raddb/mods-config/files/accounting
reading pairlist file /usr/local/etc/raddb/mods-config/files/pre-proxy
  # Instantiating module "linelog" from file /usr/local/etc/raddb/mods-enabled/linelog
  # Instantiating module "log_accounting" from file /usr/local/etc/raddb/mods-enabled/linelog
  # Instantiating module "logintime" from file /usr/local/etc/raddb/mods-enabled/logintime
  # Instantiating module "mschap" from file /usr/local/etc/raddb/mods-enabled/mschap
rlm_mschap (mschap): using internal authentication
  # Instantiating module "pap" from file /usr/local/etc/raddb/mods-enabled/pap
  # Instantiating module "etc_passwd" from file /usr/local/etc/raddb/mods-enabled/passwd
rlm_passwd: nfields: 3 keyfield 0(User-Name) listable: no
  # Instantiating module "preprocess" from file /usr/local/etc/raddb/mods-enabled/preprocess
reading pairlist file /usr/local/etc/raddb/mods-config/preprocess/huntgroups
reading pairlist file /usr/local/etc/raddb/mods-config/preprocess/hints
  # Instantiating module "IPASS" from file /usr/local/etc/raddb/mods-enabled/realm
  # Instantiating module "suffix" from file /usr/local/etc/raddb/mods-enabled/realm
  # Instantiating module "bangpath" from file /usr/local/etc/raddb/mods-enabled/realm
  # Instantiating module "realmpercent" from file /usr/local/etc/raddb/mods-enabled/realm
  # Instantiating module "ntdomain" from file /usr/local/etc/raddb/mods-enabled/realm
  # Instantiating module "totp" from file /usr/local/etc/raddb/mods-enabled/totp
 } # modules
radiusd: #### Loading Virtual Servers ####
server { # from file /usr/local/etc/raddb/radiusd.conf
} # server
server default { # from file /usr/local/etc/raddb/sites-enabled/default
 # Loading authenticate {...}
Compiling Auth-Type PAP for attr Auth-Type
Compiling Auth-Type CHAP for attr Auth-Type
Compiling Auth-Type MS-CHAP for attr Auth-Type
 # Loading authorize {...}
Ignoring "sql" (see raddb/mods-available/README.rst)
Ignoring "ldap" (see raddb/mods-available/README.rst)
Compiling Autz-Type New-TLS-Connection for attr Autz-Type
 # Loading preacct {...}
 # Loading accounting {...}
 # Loading post-proxy {...}
 # Loading post-auth {...}
Compiling Post-Auth-Type REJECT for attr Post-Auth-Type
Compiling Post-Auth-Type Challenge for attr Post-Auth-Type
Compiling Post-Auth-Type Client-Lost for attr Post-Auth-Type
} # server default
server inner-tunnel { # from file /usr/local/etc/raddb/sites-enabled/inner-tunnel
 # Loading authenticate {...}
Compiling Auth-Type PAP for attr Auth-Type
Compiling Auth-Type CHAP for attr Auth-Type
Compiling Auth-Type MS-CHAP for attr Auth-Type
 # Loading authorize {...}
 # Loading session {...}
 # Loading post-proxy {...}
 # Loading post-auth {...}
 # Skipping contents of 'if' as it is always 'false' -- /usr/local/etc/raddb/sites-enabled/inner-tunnel:366
Compiling Post-Auth-Type REJECT for attr Post-Auth-Type
} # server inner-tunnel
 thread pool {
 	start_servers = 5
 	max_servers = 32
 	min_spare_servers = 3
 	max_spare_servers = 10
 	max_requests_per_server = 0
 	cleanup_delay = 5
 	max_queue_size = 65536
 	auto_limit_acct = no
 }
Thread spawned new child 1. Total threads in pool: 1
Thread 1 waiting to be assigned a request
Thread spawned new child 2. Total threads in pool: 2
Thread 2 waiting to be assigned a request
Thread spawned new child 3. Total threads in pool: 3
Thread 3 waiting to be assigned a request
Thread spawned new child 4. Total threads in pool: 4
Thread 4 waiting to be assigned a request
Thread spawned new child 5. Total threads in pool: 5
Thread pool initialized
radiusd: #### Opening IP addresses and Ports ####
Thread 5 waiting to be assigned a request
listen {
 	type = "auth+acct"
 	virtual_server = "default"
 	ipaddr = *
 	port = 2083
 	proto = "tcp"
  tls {
  	verify_depth = 0
  	ca_path = "/usr/local/etc/raddb/certs"
  	pem_file_type = yes
  	private_key_file = "/usr/local/etc/raddb/certs/server.pem"
  	certificate_file = "/usr/local/etc/raddb/certs/server.pem"
  	ca_file = "/usr/local/etc/raddb/certs/ca.pem"
  	private_key_password = <<< secret >>>
  	fragment_size = 8192
  	include_length = yes
  	auto_chain = yes
  	check_crl = no
  	check_all_crl = no
  	ca_path_reload_interval = 3600
  	cipher_list = "DEFAULT"
  	cipher_server_preference = no
  	require_client_cert = yes
  	reject_unknown_intermediate_ca = no
  	ecdh_curve = ""
  	tls_max_version = "1.3"
  	tls_min_version = "1.2"
   cache {
   	enable = no
   	lifetime = 24
   	max_entries = 255
   }
   verify {
   	skip_if_ocsp_ok = no
   }
   ocsp {
   	enable = no
   	override_cert_url = no
   	use_nonce = yes
   	timeout = 0
   	softfail = no
   }
  }
 	check_client_connections = yes
  limit {
  	max_connections = 16
  	lifetime = 0
  	idle_timeout = 30
  }
 	clients = "radsec"
  client 127.0.0.1 {
  	ipaddr = *
  	secret = <<< secret >>>
  	proto = "tls"
   limit {
   	max_connections = 16
   	lifetime = 0
   	idle_timeout = 30
   }
  }
Shared secret for client 127.0.0.1 is short, and likely can be broken by an attacker.
}
listen {
  	type = "auth"
  	ipaddr = *
  	port = 0
   limit {
   	max_connections = 16
   	lifetime = 0
   	idle_timeout = 900
   }
}
listen {
  	type = "acct"
  	ipaddr = *
  	port = 0
   limit {
   	max_connections = 16
   	lifetime = 0
   	idle_timeout = 30
   }
}
listen {
  	type = "auth"
  	ipv6addr = ::
  	port = 0
   limit {
   	max_connections = 16
   	lifetime = 0
   	idle_timeout = 30
   }
}
listen {
  	type = "acct"
  	ipv6addr = ::
  	port = 0
   limit {
   	max_connections = 16
   	lifetime = 0
   	idle_timeout = 30
   }
}
listen {
  	type = "auth"
  	ipaddr = 127.0.0.1
  	port = 18120
}
Listening on auth+acct proto tcp address * port 2083 (TLS) bound to server default
Listening on auth address * port 1812 bound to server default
Listening on acct address * port 1813 bound to server default
Listening on auth address :: port 1812 bound to server default
Listening on acct address :: port 1813 bound to server default
Listening on auth address 127.0.0.1 port 18120 bound to server inner-tunnel
Listening on proxy address * port 39095
Listening on proxy address :: port 55812
Ready to process requests



SOME TLS CONNECTIONS




(0) (TLS) We already have 14 bytes of application data, processing it.
Thread 1 got semaphore
Waking up in 0.2 seconds.
(0) (TLS) Client has closed the TCP connection
Thread 1 handling request 8, (2 handled so far)
(TLS) Closing socket from client port 43372
(0) (TLS) RADIUS/TLS - send TLS 1.3 Alert, warning close_notify
(TLS) Closing connection
(8) (TLS) Checking connection to see if it is authorized.
(0) (TLS) We already have 14 bytes of application data, processing it.
(8) # Executing group from file /usr/local/etc/raddb/sites-enabled/default
(8)   Autz-Type New-TLS-Connection {
(8)     [ok] = ok
(8)   } # Autz-Type New-TLS-Connection = ok
(0) (TLS) We already have 14 bytes of application data, processing it.
(8) (TLS) Connection is authorized
Waking up in 0.2 seconds.
(0) (TLS) Client has closed the TCP connection
(TLS) Closing socket from client port 43412
(8) Sent Access-Accept Id 0 from 0.0.0.0:2083 to 192.168.97.2:43404 length 38
(0) (TLS) RADIUS/TLS - send TLS 1.3 Alert, warning close_notify
(TLS) Closing connection
(0) (TLS) Client has closed the TCP connection
(TLS) Closing socket from client port 43388
(0) (TLS) RADIUS/TLS - send TLS 1.3 Handshake, Finished
(0) (TLS) RADIUS/TLS - send TLS 1.3 Alert, warning close_notify
(TLS) Closing connection
Waking up in 0.2 seconds.
(8) Finished request
Thread 1 waiting to be assigned a request
Segmentation fault (core dumped)

Relevant log output from client utilities

No response

Backtrace from LLDB or GDB

(lldb) bt all
* thread #1, name = 'radiusd', stop reason = signal SIGSEGV: invalid address (fault address: 0xaaaaab519000)
  * frame #0: 0x0000fffff78c7c84 libc.so.6`__memcpy_generic at memcpy.S:186
    frame #1: 0x0000aaaaaab0c5e4 radiusd`tls_socket_write(listener=0x0000aaaaaaffc6a0) at tls_listen.c:147:2
    frame #2: 0x0000aaaaaab0db9c radiusd`tls_socket_recv(listener=0x0000aaaaaaffc6a0) at tls_listen.c:638:11
    frame #3: 0x0000aaaaaab0e24c radiusd`dual_tls_recv(listener=0x0000aaaaaaffc6a0) at tls_listen.c:768:7
    frame #4: 0x0000aaaaaaaf50e8 radiusd`event_socket_handler(xel=0x0000aaaaaaed9720, fd=22, ctx=0x0000aaaaaaffc6a0) at process.c:5390:2
    frame #5: 0x0000fffff7f2e9f4 libfreeradius-radius.so`fr_event_loop(el=0x0000aaaaaaed9720) at event.c:721:4
    frame #6: 0x0000aaaaaaaf7530 radiusd`radius_event_process at process.c:6617:9
    frame #7: 0x0000aaaaaaaddea4 radiusd`main(argc=4, argv=0x0000fffffffff6b8) at radiusd.c:653:19
    frame #8: 0x0000fffff78573fc libc.so.6`__libc_start_call_main(main=(radiusd`main at radiusd.c:97:1), argc=4, argv=0x0000fffffffff6b8) at libc_start_call_main.h:58:16
    frame #9: 0x0000fffff78574cc libc.so.6`__libc_start_main_impl(main=(radiusd`main at radiusd.c:97:1), argc=4, argv=0x0000fffffffff6b8, init=(radiusd`__do_global_dtors_aux_fini_array_entry), fini=<unavailable>, rtld_fini=<unavailable>, stack_end=<unavailable>) at libc-start.c:392:3
    frame #10: 0x0000aaaaaaab5970 radiusd`_start + 48
  thread #2, name = 'radiusd'
    frame #0: 0x0000fffff78a9df8 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12
    frame #1: 0x0000fffff78a9dd4 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:87:9
    frame #2: 0x0000fffff78a9d40 libc.so.6`__GI___futex_abstimed_wait_cancelable64(futex_word=0x0000aaaaaab4c4a0, expected=0, clockid=<unavailable>, abstime=0x0000000000000000, private=<unavailable>) at futex-internal.c:139:10
    frame #3: 0x0000fffff78b5a5c libc.so.6`__new_sem_wait_slow64(sem=0x0000aaaaaab4c4a0, abstime=<unavailable>, clockid=<unavailable>) at sem_waitcommon.c:183:10
    frame #4: 0x0000aaaaaaae7184 radiusd`request_handler_thread(arg=0x0000aaaaaaf22ae0) at threads.c:741:7
    frame #5: 0x0000fffff78ad5c8 libc.so.6`start_thread(arg=0x0000000000000000) at pthread_create.c:442:8
    frame #6: 0x0000fffff7915edc libc.so.6`thread_start at clone.S:79
  thread #3, name = 'radiusd'
    frame #0: 0x0000fffff78a9df8 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12
    frame #1: 0x0000fffff78a9dd4 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:87:9
    frame #2: 0x0000fffff78a9d40 libc.so.6`__GI___futex_abstimed_wait_cancelable64(futex_word=0x0000aaaaaab4c4a0, expected=0, clockid=<unavailable>, abstime=0x0000000000000000, private=<unavailable>) at futex-internal.c:139:10
    frame #3: 0x0000fffff78b5a5c libc.so.6`__new_sem_wait_slow64(sem=0x0000aaaaaab4c4a0, abstime=<unavailable>, clockid=<unavailable>) at sem_waitcommon.c:183:10
    frame #4: 0x0000aaaaaaae7184 radiusd`request_handler_thread(arg=0x0000aaaaaaf220f0) at threads.c:741:7
    frame #5: 0x0000fffff78ad5c8 libc.so.6`start_thread(arg=0x0000000000000000) at pthread_create.c:442:8
    frame #6: 0x0000fffff7915edc libc.so.6`thread_start at clone.S:79
  thread #4, name = 'radiusd'
    frame #0: 0x0000fffff78a9df8 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12
    frame #1: 0x0000fffff78a9dd4 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:87:9
    frame #2: 0x0000fffff78a9d40 libc.so.6`__GI___futex_abstimed_wait_cancelable64(futex_word=0x0000aaaaaab4c4a0, expected=0, clockid=<unavailable>, abstime=0x0000000000000000, private=<unavailable>) at futex-internal.c:139:10
    frame #3: 0x0000fffff78b5a5c libc.so.6`__new_sem_wait_slow64(sem=0x0000aaaaaab4c4a0, abstime=<unavailable>, clockid=<unavailable>) at sem_waitcommon.c:183:10
    frame #4: 0x0000aaaaaaae7184 radiusd`request_handler_thread(arg=0x0000aaaaaaf22bc0) at threads.c:741:7
    frame #5: 0x0000fffff78ad5c8 libc.so.6`start_thread(arg=0x0000000000000000) at pthread_create.c:442:8
    frame #6: 0x0000fffff7915edc libc.so.6`thread_start at clone.S:79
  thread #5, name = 'radiusd'
    frame #0: 0x0000fffff78a9df8 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12
    frame #1: 0x0000fffff78a9dd4 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:87:9
    frame #2: 0x0000fffff78a9d40 libc.so.6`__GI___futex_abstimed_wait_cancelable64(futex_word=0x0000aaaaaab4c4a0, expected=0, clockid=<unavailable>, abstime=0x0000000000000000, private=<unavailable>) at futex-internal.c:139:10
    frame #3: 0x0000fffff78b5a5c libc.so.6`__new_sem_wait_slow64(sem=0x0000aaaaaab4c4a0, abstime=<unavailable>, clockid=<unavailable>) at sem_waitcommon.c:183:10
    frame #4: 0x0000aaaaaaae7184 radiusd`request_handler_thread(arg=0x0000aaaaaaf22970) at threads.c:741:7
    frame #5: 0x0000fffff78ad5c8 libc.so.6`start_thread(arg=0x0000000000000000) at pthread_create.c:442:8
    frame #6: 0x0000fffff7915edc libc.so.6`thread_start at clone.S:79
  thread #6, name = 'radiusd'
    frame #0: 0x0000fffff78a9df8 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12
    frame #1: 0x0000fffff78a9dd4 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:87:9
    frame #2: 0x0000fffff78a9d40 libc.so.6`__GI___futex_abstimed_wait_cancelable64(futex_word=0x0000aaaaaab4c4a0, expected=0, clockid=<unavailable>, abstime=0x0000000000000000, private=<unavailable>) at futex-internal.c:139:10
    frame #3: 0x0000fffff78b5a5c libc.so.6`__new_sem_wait_slow64(sem=0x0000aaaaaab4c4a0, abstime=<unavailable>, clockid=<unavailable>) at sem_waitcommon.c:183:10
    frame #4: 0x0000aaaaaaae7184 radiusd`request_handler_thread(arg=0x0000aaaaaaf22050) at threads.c:741:7
    frame #5: 0x0000fffff78ad5c8 libc.so.6`start_thread(arg=0x0000000000000000) at pthread_create.c:442:8
    frame #6: 0x0000fffff7915edc libc.so.6`thread_start at clone.S:79

[alandekok]

Please try the latest code in the v3.2.x branch. I've pushed a number of changes which should fix it.

There's also tests for it in src/tests/tls/. See the https://github.com/FreeRADIUS/freeradius-server/blob/v3.2.x/src/tests/tls/README.md file there for details.

[gk-fschubert]

Thanks!
It's working much better with the code changes and the default settings. But when i set max_connections to 0 and stress it a bit more, after a while the server crashes with:

(0) (TLS) RADIUS/TLS - recv TLS 1.3 Handshake, Finished                                  
Thread 3 got semaphore                                                                                                                                                             
Thread 3 handling request 2869, (575 handled so far)                                                                                                                                                                                                                                                                                                                  
(2869) (TLS) Checking connection to see if it is authorized.                                                                                                                                                                                                                                                                                                          
(2869) # Executing group from file /usr/local/etc/raddb/sites-enabled/default     
(2869)   Autz-Type New-TLS-Connection {                        
(2869)     [ok] = ok         
(2869)   } # Autz-Type New-TLS-Connection = ok                                                                                                                                                                                                                                                                                                                        
(2869) (TLS) Connection is authorized                     
(0) (TLS) We already have 14 bytes of application data, processing it.
(2869) Sent Access-Accept Id 0 from 0.0.0.0:2083 to 192.168.97.2:60936 length 38                                                                                                                                                                                                                                                                                      
radiusd: pthread_mutex_lock.c:94: ___pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.                                                                            
(0) (TLS) RADIUS/TLS - send TLS 1.3 Handshake, Finished                                                                                                                            
Aborted (core dumped)      

backtrace:

(gdb) info threads                                                                                                                                                                                                                                                                                                                                                    
  Id   Target Id                      Frame  
* 1    Thread 0xffff8ab02440 (LWP 49) __pthread_kill_implementation (threadid=281473008542784, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
  2    Thread 0xffff85935100 (LWP 55) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:57
  3    Thread 0xffff86145100 (LWP 54) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:57
  4    Thread 0xffff87165100 (LWP 52) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:57
  5    Thread 0xffff86955100 (LWP 53) __GI___libc_write (nbytes=60, buf=0xaaab1f3a5e60, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/write.c:26                                                                                                                                                                                                                  
  6    Thread 0xffff87975100 (LWP 51) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:57  

(gdb) thread apply all bt full

Thread 6 (Thread 0xffff87975100 (LWP 51)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:57
        _x3tmp = 0
        _x0tmp = 187650720855200
        _x0 = 187650720855200
        _x3 = 0
        _x4tmp = 0
        _x1tmp = <optimized out>
        _x1 = 393
        _x4 = 0
        _x5tmp = 4294967295
        _x2tmp = 0
        _x2 = 0
        _x5 = 4294967295
        _x8 = 98
        _sys_result = <optimized out>
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x5tmp = <optimized out>
        _x4tmp = <optimized out>
        _x3tmp = <optimized out>
        _x2tmp = <optimized out>
        _x1tmp = <optimized out>
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        _x1 = <optimized out>
        _x2 = <optimized out>
        _x3 = <optimized out>
        _x4 = <optimized out>
        _x5 = <optimized out>
        _x8 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaad68ef4a0 <thread_pool+176>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x0000ffff8a3f59a4 in do_futex_wait (sem=sem@entry=0xaaaad68ef4a0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x0000ffff8a3f5a5c in __new_sem_wait_slow64 (sem=0xaaaad68ef4a0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0xffff8a3f5970 <__sem_wait_cleanup>, __arg = 0xaaaad68ef4a0 <thread_pool+176>, __canceltype = -695700908, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x0000aaaad6887254 in request_handler_thread (arg=0xaaab16a615b0) at src/main/threads.c:741
        self = 0xaaab16a615b0
#6  0x0000ffff8a3ed5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472956584192, 281474722413232, 281474722413230, 8448320, 281474722413231, 0, 281472948133888, 8448320, 281473008542784, 281472948133888, 281472956581920, 4139586479708792988, 0, 4139586479668393120, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#7  0x0000ffff8a455edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 5 (Thread 0xffff86955100 (LWP 53)):
#0  __GI___libc_write (nbytes=60, buf=0xaaab1f3a5e60, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/write.c:26
        _x2tmp = 60
        _x0tmp = <optimized out>
        _x0 = 60
        _x2 = 60
        _x8 = 64
        _x1tmp = 187651940048480
        _x1 = 187651940048480
        _sys_result = <optimized out>
        sc_ret = <optimized out>
        sc_cancel_oldtype = 0
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        sc_ret = <optimized out>
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x1tmp = <optimized out>
        _x1 = <optimized out>
        _x2tmp = <optimized out>
        _x2 = <optimized out>
        _x8 = <optimized out>
#1  __GI___libc_write (fd=<optimized out>, buf=0xaaab1f3a5e60, nbytes=60) at ../sysdeps/unix/sysv/linux/write.c:24
No locals.
#2  0x0000aaaad68af608 in tls_socket_write (listener=0xaaab195e5180) at src/main/tls_listen.c:112
        rcode = 6637240131620415231
        sock = 0xaaab195e5350
#3  0x0000aaaad68b1e80 in dual_tls_send (listener=0xaaab195e5180, request=0xaaab1968ffc0) at src/main/tls_listen.c:1045
        sock = 0xaaab195e5350
#4  0x0000aaaad688c54c in request_finish (request=0xaaab1968ffc0, action=1) at src/main/process.c:1645
        vp = 0x0
#5  0x0000aaaad688c91c in request_running (request=0xaaab1968ffc0, action=1) at src/main/process.c:1774
        rcode = 0
        __FUNCTION__ = "request_running"
#6  0x0000aaaad68873cc in request_handler_thread (arg=0xaaab16a61690) at src/main/threads.c:812
        self = 0xaaab16a61690
#7  0x0000ffff8a3ed5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472939675904, 281474722413232, 281474722413230, 8448320, 281474722413231, 0, 281472931225600, 8448320, 281473008542784, 281472931225600, 281472939673632, 4139586479708792988, 0, 4139586479651746976, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#8  0x0000ffff8a455edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 4 (Thread 0xffff87165100 (LWP 52)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:57
        _x3tmp = 0
        _x0tmp = 187650720855200
        _x0 = 187650720855200
        _x3 = 0
        _x4tmp = 0
        _x1tmp = <optimized out>
        _x1 = 393
        _x4 = 0
        _x5tmp = 4294967295
        _x2tmp = 0
        _x2 = 0
        _x5 = 4294967295
        _x8 = 98
        _sys_result = <optimized out>
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x5tmp = <optimized out>
        _x4tmp = <optimized out>
        _x3tmp = <optimized out>
        _x2tmp = <optimized out>
        _x1tmp = <optimized out>
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        _x1 = <optimized out>
        _x2 = <optimized out>
        _x3 = <optimized out>
        _x4 = <optimized out>
        _x5 = <optimized out>
        _x8 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaad68ef4a0 <thread_pool+176>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x0000ffff8a3f59a4 in do_futex_wait (sem=sem@entry=0xaaaad68ef4a0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x0000ffff8a3f5a5c in __new_sem_wait_slow64 (sem=0xaaaad68ef4a0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0xffff8a3f5970 <__sem_wait_cleanup>, __arg = 0xaaaad68ef4a0 <thread_pool+176>, __canceltype = -695700908, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x0000aaaad6887254 in request_handler_thread (arg=0xaaab16a60bc0) at src/main/threads.c:741
        self = 0xaaab16a60bc0
#6  0x0000ffff8a3ed5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472948130048, 281474722413232, 281474722413230, 8448320, 281474722413231, 0, 281472939679744, 8448320, 281473008542784, 281472939679744, 281472948127776, 4139586479708792988, 0, 4139586479660070048, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#7  0x0000ffff8a455edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 3 (Thread 0xffff86145100 (LWP 54)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:57
        _x3tmp = 0
        _x0tmp = 187650720855200
        _x0 = 187650720855200
        _x3 = 0
        _x4tmp = 0
        _x1tmp = <optimized out>
        _x1 = 393
        _x4 = 0
        _x5tmp = 4294967295
        _x2tmp = 0
        _x2 = 0
        _x5 = 4294967295
        _x8 = 98
        _sys_result = <optimized out>
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x5tmp = <optimized out>
        _x4tmp = <optimized out>
        _x3tmp = <optimized out>
        _x2tmp = <optimized out>
        _x1tmp = <optimized out>
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        _x1 = <optimized out>
        _x2 = <optimized out>
        _x3 = <optimized out>
        _x4 = <optimized out>
        _x5 = <optimized out>
        _x8 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaad68ef4a0 <thread_pool+176>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x0000ffff8a3f59a4 in do_futex_wait (sem=sem@entry=0xaaaad68ef4a0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x0000ffff8a3f5a5c in __new_sem_wait_slow64 (sem=0xaaaad68ef4a0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0xffff8a3f5970 <__sem_wait_cleanup>, __arg = 0xaaaad68ef4a0 <thread_pool+176>, __canceltype = -695700908, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x0000aaaad6887254 in request_handler_thread (arg=0xaaab16a61440) at src/main/threads.c:741
        self = 0xaaab16a61440
#6  0x0000ffff8a3ed5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472931221760, 281474722413232, 281474722413230, 8448320, 281474722413231, 0, 281472922771456, 8448320, 281473008542784, 281472922771456, 281472931219488, 4139586479708792988, 0, 4139586479643423904, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#7  0x0000ffff8a455edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 2 (Thread 0xffff85935100 (LWP 55)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:57
        _x3tmp = 0
        _x0tmp = 187650720855200
        _x0 = 187650720855200
        _x3 = 0
        _x4tmp = 0
        _x1tmp = <optimized out>
        _x1 = 393
        _x4 = 0
        _x5tmp = 4294967295
        _x2tmp = 0
        _x2 = 0
        _x5 = 4294967295
        _x8 = 98
        _sys_result = <optimized out>
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x5tmp = <optimized out>
        _x4tmp = <optimized out>
        _x3tmp = <optimized out>
        _x2tmp = <optimized out>
        _x1tmp = <optimized out>
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        _x1 = <optimized out>
        _x2 = <optimized out>
        _x3 = <optimized out>
        _x4 = <optimized out>
        _x5 = <optimized out>
        _x8 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaad68ef4a0 <thread_pool+176>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaad68ef4a0 <thread_pool+176>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x0000ffff8a3f59a4 in do_futex_wait (sem=sem@entry=0xaaaad68ef4a0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x0000ffff8a3f5a5c in __new_sem_wait_slow64 (sem=0xaaaad68ef4a0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0xffff8a3f5970 <__sem_wait_cleanup>, __arg = 0xaaaad68ef4a0 <thread_pool+176>, __canceltype = -695700908, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x0000aaaad6887254 in request_handler_thread (arg=0xaaab16a60b20) at src/main/threads.c:741
        self = 0xaaab16a60b20
#6  0x0000ffff8a3ed5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472922767616, 281474722413232, 281474722413230, 8448320, 281474722413231, 0, 281472914317312, 8448320, 281473008542784, 281472914317312, 281472922765344, 4139586479708792988, 0, 4139586479701685408, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#7  0x0000ffff8a455edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 1 (Thread 0xffff8ab02440 (LWP 49)):
#0  __pthread_kill_implementation (threadid=281473008542784, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
        tid = 49
        ret = 0
        pd = 0xffff8ab02440
        old_mask = {__val = {281473000843404, 281474722412000, 281473001031028, 281473002292472, 281474722412112, 281473001995304, 0, 281473001974448, 281473002283008, 281474722412368, 281474722412368, 281474722412320, 18446743528248704976, 42975585592, 281473001995304, 0}}
        ret = <optimized out>
#1  0x0000ffff8a3ef254 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
No locals.
#2  0x0000ffff8a3aa67c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#3  0x0000ffff8a397130 in __GI_abort () at ./stdlib/abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0xffff8a4beab0, sa_sigaction = 0xffff8a4beab0}, sa_mask = {__val = {281473002283008, 0, 281474722414248, 4222124650659840, 17298308644996116495, 17298308644996116495, 17298308644996116495, 281474722412320, 281473001033328, 281473002292472, 15575427602093527808, 281474722412384, 281473000816588, 281473008758784, 281473001964880, 281473001974256}}, sa_flags = -1974745776, sa_restorer = 0xffff8a4be9f0}
        sigs = {__val = {32, 18446743528248704976, 281474722412384, 281473000816448, 281474722412368, 281474722412368, 281474722412320, 18446743528248704976, 2987131702627757358, 281474722412368, 281474722412304, 281473001184388, 281473008544768, 64, 0, 94}}
#4  0x0000ffff8a3a3fd0 in __assert_fail_base (fmt=0xffff8a4bc550 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xffff8a4beab0 "mutex->__data.__owner == 0", file=file@entry=0xffff8a4be9f0 "pthread_mutex_lock.c", line=line@entry=94, function=function@entry=0xffff8a4bec38 <__PRETTY_FUNCTION__.0> "___pthread_mutex_lock") at ./assert/assert.c:92
        str = 0xaaab17acf8c0 "\317z\261\252\n"
        total = 4096
#5  0x0000ffff8a3a4040 in __GI___assert_fail (assertion=0xffff8a4beab0 "mutex->__data.__owner == 0", file=0xffff8a4be9f0 "pthread_mutex_lock.c", line=94, function=0xffff8a4bec38 <__PRETTY_FUNCTION__.0> "___pthread_mutex_lock") at ./assert/assert.c:101
No locals.
#6  0x0000ffff8a3f08a8 in ___pthread_mutex_lock (mutex=0xaaab195e5478) at ./nptl/pthread_mutex_lock.c:94
        type = <optimized out>
        __PRETTY_FUNCTION__ = "___pthread_mutex_lock"
        id = <optimized out>
#7  0x0000aaaad68b06a4 in tls_socket_recv (listener=0xaaab195e5180) at src/main/tls_listen.c:495
        doing_init = false
        already_read = false
        rcode = 281474722412848
        data_len = 187650720602212
        packet = 0x3b
        request = 0xaaab195e5ad0
        sock = 0xaaab195e5350
        status = 43691
#8  0x0000aaaad68b1398 in dual_tls_recv (listener=0xaaab195e5180) at src/main/tls_listen.c:769
        packet = 0xaaab185489e0
        fun = 0x0
        sock = 0xaaab195e5350
        client = 0xaaab16a94210
        rbio = 0x3
#9  0x0000aaaad68951b8 in event_socket_handler (xel=0xaaab16a181f0, fd=392, ctx=0xaaab195e5180) at src/main/process.c:5390
        listener = 0xaaab195e5180
#10 0x0000ffff8aa6e9f4 in fr_event_loop (el=0xaaab16a181f0) at src/lib/event.c:721
        ef = 0xaaab16a18468
        i = 10
        rcode = 1
        when = {tv_sec = 1735943242, tv_usec = 742057}
        wake = 0xfffff0d7bb28
        read_fds = {fds_bits = {0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 0, 0, 0}}
        write_fds = {fds_bits = {0 <repeats 16 times>}}
#11 0x0000aaaad6897648 in radius_event_process () at src/main/process.c:6630
No locals.
#12 0x0000aaaad687dea4 in main (argc=3, argv=0xfffff0d7be88) at src/main/radiusd.c:653
        rcode = 0
        status = 0
        argval = -1
        spawn_flag = true
        display_version = false
        flag = 0
        from_child = {-1, -1}
        p = 0x0
        state = 0xaaaad68eed78 <global_state>
        autofree = 0xaaab1668b300

I've wrote a small python script to test this:

import multiprocessing
import time
import socket
import ssl
import logging

class SSLClient:
    def __init__(self, ip, client_cert_file_location, trusted_cas_file_location):
        self.client_cert_file_location = client_cert_file_location
        self.trusted_cas_file_location = trusted_cas_file_location
        self.socket: ssl.SSLSocket = self.connect_to_server(ip, 2083, self.client_cert_file_location, self.trusted_cas_file_location)


    def connect_to_server(self, ip, port, client_cert_file, ca_file_location): 
        try:
            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
            context.verify_mode = ssl.CERT_REQUIRED
            context.load_cert_chain(certfile=client_cert_file)

            context.load_verify_locations(cafile=ca_file_location)
            context.check_hostname = False
            sock = context.wrap_socket(sock, server_hostname=ip)
            
            connection =  sock.connect((ip, int(port)))  
            return sock
            
        except Exception as e:
            logging.error(f'Radius tcp monitor: RADIUS server {ip} is DOWN. err: ' + str(e))
            return None


def send_stuff(x):
    try:
        test = SSLClient('127.0.0.1', './certs/radsec/client.pem', './certs/radsec/ca.crt')

        test.socket.send(b'asdlkfjasldkfj')
        test.socket.shutdown(socket.SHUT_WR)
        test.socket.recv(1024)
        test.socket.close()
    except Exception as e: 
        print('error while sending things')
    finally:
        try:
            test.socket.close()
        except: 
            pass




if __name__ == '__main__':
    with multiprocessing.Pool(16) as p: 
        while True:
            p.map(send_stuff, range(400))
            time.sleep(1)
            

[alandekok]

I've added that python script as src/tests/tls/stress.py. I can't reproduce this on OSX, and haven't had time to check it on Linux.

My guess is that the socket is somehow being deleted while it's still in use. But I'd have to reproduce the issue in order to know for sure.

[gk-fschubert]

I'm also on OSX but running the server is running in a linux docker container.

The entrypoint I'm using will copy the current raddb folder to one which is attached as volume to the host if the environment variable INIT is true and if it's not true it will copy everything from the volume to /etc/raddb. It's done that way because it's a nightmare dealing with file permissions if it's exposed to the host.

The network is made with traefik to be able to test with PROXY_PROTOCOL as well if needed.

The host folder ./config include static files which should be overwritten. In this setup it are just the certificates.

docker-compose.yml:

name: radius-vanilla
services: 
  radius:
    image: radius-server-vanilla
    build: 
      context: .
    container_name: vanillaradius
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp=unconfined
    hostname: radius      
    environment: 
      - INIT=false
    entrypoint: ['/bin/sh', '-c', 'while true; do sleep 5; echo "waiting" ;done']

    volumes: 
      - ./runtime/runningraddb:/etc/runningraddb
      - ./runtime/logs/radius:/usr/local/var/log/radius
      - ./config:/etc/toraddb
      - ./docker-entrypoint.sh:/docker-entrypoint.sh
    labels: 
      - "traefik.enable=true"
      - 'traefik.tcp.routers.radius.entrypoints=radius'
      - 'traefik.tcp.services.radius.loadbalancer.server.port=2083'
      - 'traefik.tcp.services.radius.loadbalancer.server.address=host.docker.internal'
      # - 'traefik.tcp.services.radius.loadbalancer.proxyprotocol.version=1'
      - 'traefik.tcp.routers.radius.rule=HostSNI(`*`)'
      - 'traefik.tcp.routers.radius.service=radius'
      - 'traefik.tcp.routers.radius.tls.passthrough=true'

    extra_hosts:
      - host.docker.internal:host-gateway
    networks:
      - traefik


  traefik:
    image: "traefik:v2.5.5"
    container_name: "traefik"
    hostname: "traefik"
    ports:
      - "80:80"
      - "2083:2083"

    command: 
      - '--providers.docker=true'
      - '--accesslog=true'
      - '--entryPoints.radius.address=:2083'
      - '--providers.docker.network=traefik'
      - '--api.dashboard=true'
      - '--api.insecure=true'
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    extra_hosts:
      - host.docker.internal:host-gateway
    networks:
      - traefik


networks:
  traefik:
    name: traefik

Dockerfile I'm using:

ARG from=ubuntu:22.04
FROM ${from} as build

#
#  Install build tools
#
ARG APT_OPTS='-yq --option=Dpkg::options::=--force-unsafe-io'
ARG DEBIAN_FRONTEND='noninteractive'
ARG REGISTRYUSER=FAIL
ARG REGISTRYPW=FAIL
ARG PYTHONVERSION=3.10

RUN apt clean
RUN apt update $APT_OPTS &&  apt upgrade $APT_OPTS
RUN apt $APT_OPTS install  \
    devscripts \
    equivs \
    git \
    quilt \
    gcc \
    gdb \
    lldb
    
#
#  Create build directory
#
RUN mkdir -p /build/repositories/freeradius-server
WORKDIR /build/repositories

COPY ./freeradius-server-source ./freeradius-server
WORKDIR /build/repositories/freeradius-server


#  Install build dependencies
#
# essential
RUN apt $APT_OPTS update &&  apt $APT_OPTS upgrade &&  apt $APT_OPTS update --fix-missing
RUN apt $APT_OPTS install \
        libc-dev \
        libtalloc-dev \
        libssl-dev 
RUN apt $APT_OPTS install \
        linux-headers-generic \
        libpcre3-dev \
        libidn11-dev \
        krb5-multidev \
        samba-dev \
        libcurl4-openssl-dev \
        libjson-c-dev \
        lsb-release \
        ubuntu-dbgsym-keyring
        

# needed for debug symbols
RUN <<EOR 
    echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
    deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
    deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
    tee -a /etc/apt/sources.list.d/ddebs.list
EOR

RUN apt $APT_OPTS update
RUN apt $APT_OPTS install libssl3-dbgsym

ARG cc=gcc
RUN ./configure  --with-openssl=yes --build=x86_64-unknown-linux-gnu --enable-developer
RUN make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
RUN make install

#
# These are needed for the server to start
#
RUN apt $APT_OPTS update  \
    && apt $APT_OPTS install libtalloc-dev libssl-dev libssl3-dbgsym libpcre3-dev sssd libpam-sss libnss-sss sssd-tools samba \
    && apt $APT_OPTS install libcurl4-openssl-dev libjson-c-dev python3  \
    \
    && ln -s /usr/local/etc/raddb /etc/raddb

RUN ln -s /etc/raddb/sites-available/tls /etc/raddb/sites-enabled/tls

## Set requirements
# allow all ip addresses and set max_connections to 0
RUN sed -i '/client 127\.0\.0\.1 /{n;s/ipaddr = 127\.0\.0\.1/ipaddr = *\nlimit {max_connections = 0  }/}' /etc/raddb/sites-available/tls
# RUN sed -i 's/#.*proxy_protocol.*/proxy_protocol = yes/' /etc/raddb/sites-available/tls
RUN sed -i 's/#.*check_client_connections .*/check_client_connections = yes/' /etc/raddb/sites-available/tls

# set all other default max_connection settings to unlimited
RUN sed -i 's/max_connections = 16.*/max_connections = 0/' /etc/raddb/sites-available/tls

# allow core_dumps
RUN sed -i 's/allow_core_dumps .*/allow_core_dumps = yes/' /etc/raddb/radiusd.conf


RUN  apt $APT_OPTS update  && apt $APT_OPTS upgrade
RUN apt install nano tmux curl dnsutils tcpdump -yq

COPY docker-entrypoint.sh /
RUN mkdir /etc/runningraddb
RUN ln -n /etc/raddb /etc/runningraddb


RUN chmod -R 755 /usr/local/etc/raddb
ENV INIT=true


WORKDIR /etc/raddb
EXPOSE 1812/udp 1813/udp 2083/tcp
ENTRYPOINT ["/bin/bash"]
CMD ["/docker-entrypoint.sh"]

docker-entrypoint.sh

#!/bin/bash
set -e

if [ ! -d "/etc/runningraddb" ]; then
    mkdir /etc/runningraddb 
fi

if [ "${INIT:-}" == 'true' ]; then
    cp -Lr /etc/raddb/* /etc/runningraddb
else
    ulimit -c unlimited
    cp -r /etc/runningraddb/* /etc/raddb
    cp -r /etc/toraddb/raddb/* /etc/raddb
fi

[ndptech]

@gk-fschubert I ran your stress test against the head of v3.2.x today and it did not crash.

[gk-fschubert]

@ndptech
which OS are you using?
And could you may also check against the docker setup?

[alandekok]

@gk-fschubert Please try reproducing it with the latest code. We've fixed the crash. Docker or not shouldn't matter.

[ndptech]

@gk-fschubert I was running natively on Linux (Ubuntu 24.04) - not in Docker

[gk-fschubert]

@alandekok
I have noticed a big difference when the script runs against a radius server that is behind Traefik and when it is addressed directly.
To change this, I have made some changes to the script to not wait until it gets the shutdown and close it directly.
Also, the function to create the SSL client repeats the connection by itself so that the processes don't have to be recreated.
(sorry I could have checked this earlier)

PR: #5503

With the script I'm also able to reproduce it without traefik in front of FreeRADIUS.

Build on commit: e5f156a

root@radius:/usr/local/etc/raddb# gdb $(which radiusd) core 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/radiusd...
[New LWP 142]
[New LWP 145]
[New LWP 147]
[New LWP 144]
[New LWP 146]
[New LWP 148]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `radiusd -fxxxl stdout'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=281472880116800, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0xffff83088440 (LWP 142))]
(gdb) info threads
  Id   Target Id                       Frame 
* 1    Thread 0xffff83088440 (LWP 142) __pthread_kill_implementation (threadid=281472880116800, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
  2    Thread 0xffff7f705100 (LWP 145) __GI___libc_write (nbytes=60, buf=0xaaaad6b32690, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/write.c:26
  3    Thread 0xffff7e6e5100 (LWP 147) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:57
  4    Thread 0xffff7ff15100 (LWP 144) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:57
  5    Thread 0xffff7eef5100 (LWP 146) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:57
  6    Thread 0xffff7ded5100 (LWP 148) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:57
(gdb) thread apply all bt full

Thread 6 (Thread 0xffff7ded5100 (LWP 148)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:57
        _x3tmp = 0
        _x0tmp = 187650080961712
        _x0 = 187650080961712
        _x3 = 0
        _x4tmp = 0
        _x1tmp = <optimized out>
        _x1 = 393
        _x4 = 0
        _x5tmp = 4294967295
        _x2tmp = 0
        _x2 = 0
        _x5 = 4294967295
        _x8 = 98
        _sys_result = <optimized out>
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x5tmp = <optimized out>
        _x4tmp = <optimized out>
        _x3tmp = <optimized out>
        _x2tmp = <optimized out>
        _x1tmp = <optimized out>
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        _x1 = <optimized out>
        _x2 = <optimized out>
        _x3 = <optimized out>
        _x4 = <optimized out>
        _x5 = <optimized out>
        _x8 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaab06af4b0 <thread_pool+176>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x0000ffff829859a4 in do_futex_wait (sem=sem@entry=0xaaaab06af4b0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x0000ffff82985a5c in __new_sem_wait_slow64 (sem=0xaaaab06af4b0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0xffff82985970 <__sem_wait_cleanup>, __arg = 0xaaaab06af4b0 <thread_pool+176>, __canceltype = -1335595012, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x0000aaaab0646ffc in request_handler_thread (arg=0xaaaace4634c0) at src/main/threads.c:870
        self = 0xaaaace4634c0
#6  0x0000ffff8297d5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472794448128, 281474145862304, 281474145862302, 8448320, 281474145862303, 0, 281472785997824, 8448320, 281472880116800, 281472785997824, 281472794445856, 4723273040259771725, 0, 4723273043561312625, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#7  0x0000ffff829e5edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 5 (Thread 0xffff7eef5100 (LWP 146)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:57
        _x3tmp = 0
        _x0tmp = 187650080961712
        _x0 = 187650080961712
        _x3 = 0
        _x4tmp = 0
        _x1tmp = <optimized out>
        _x1 = 393
        _x4 = 0
        _x5tmp = 4294967295
        _x2tmp = 0
        _x2 = 0
        _x5 = 4294967295
        _x8 = 98
        _sys_result = <optimized out>
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x5tmp = <optimized out>
        _x4tmp = <optimized out>
        _x3tmp = <optimized out>
        _x2tmp = <optimized out>
        _x1tmp = <optimized out>
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        _x1 = <optimized out>
        _x2 = <optimized out>
        _x3 = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        _x4 = <optimized out>
        _x5 = <optimized out>
        _x8 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaab06af4b0 <thread_pool+176>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x0000ffff829859a4 in do_futex_wait (sem=sem@entry=0xaaaab06af4b0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x0000ffff82985a5c in __new_sem_wait_slow64 (sem=0xaaaab06af4b0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0xffff82985970 <__sem_wait_cleanup>, __arg = 0xaaaab06af4b0 <thread_pool+176>, __canceltype = -1335595012, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x0000aaaab0646ffc in request_handler_thread (arg=0xaaaace4635c0) at src/main/threads.c:870
        self = 0xaaaace4635c0
#6  0x0000ffff8297d5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472811356416, 281474145862304, 281474145862302, 8448320, 281474145862303, 0, 281472802906112, 8448320, 281472880116800, 281472802906112, 281472811354144, 4723273040259771725, 0, 4723273043544404337, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#7  0x0000ffff829e5edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 4 (Thread 0xffff7ff15100 (LWP 144)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:57
        _x3tmp = 0
        _x0tmp = 187650080961712
        _x0 = 187650080961712
        _x3 = 0
        _x4tmp = 0
        _x1tmp = <optimized out>
        _x1 = 393
        _x4 = 0
        _x5tmp = 4294967295
        _x2tmp = 0
        _x2 = 0
        _x5 = 4294967295
        _x8 = 98
        _sys_result = <optimized out>
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x5tmp = <optimized out>
        _x4tmp = <optimized out>
        _x3tmp = <optimized out>
        _x2tmp = <optimized out>
        _x1tmp = <optimized out>
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        _x1 = <optimized out>
        _x2 = <optimized out>
        _x3 = <optimized out>
        _x4 = <optimized out>
        _x5 = <optimized out>
        _x8 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaab06af4b0 <thread_pool+176>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x0000ffff829859a4 in do_futex_wait (sem=sem@entry=0xaaaab06af4b0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x0000ffff82985a5c in __new_sem_wait_slow64 (sem=0xaaaab06af4b0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0xffff82985970 <__sem_wait_cleanup>, __arg = 0xaaaab06af4b0 <thread_pool+176>, __canceltype = -1335595012, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x0000aaaab0646ffc in request_handler_thread (arg=0xaaaace463690) at src/main/threads.c:870
        self = 0xaaaace463690
#6  0x0000ffff8297d5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472828264704, 281474145862304, 281474145862302, 8448320, 281474145862303, 0, 281472819814400, 8448320, 281472880116800, 281472819814400, 281472828262432, 4723273040259771725, 0, 4723273043529068913, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#7  0x0000ffff829e5edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

--Type <RET> for more, q to quit, c to continue without paging--
Thread 3 (Thread 0xffff7e6e5100 (LWP 147)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:57
        _x3tmp = 0
        _x0tmp = 187650080961712
        _x0 = 187650080961712
        _x3 = 0
        _x4tmp = 0
        _x1tmp = <optimized out>
        _x1 = 393
        _x4 = 0
        _x5tmp = 4294967295
        _x2tmp = 0
        _x2 = 0
        _x5 = 4294967295
        _x8 = 98
        _sys_result = <optimized out>
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x5tmp = <optimized out>
        _x4tmp = <optimized out>
        _x3tmp = <optimized out>
        _x2tmp = <optimized out>
        _x1tmp = <optimized out>
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        _x1 = <optimized out>
        _x2 = <optimized out>
        _x3 = <optimized out>
        _x4 = <optimized out>
        _x5 = <optimized out>
        _x8 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaab06af4b0 <thread_pool+176>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaab06af4b0 <thread_pool+176>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x0000ffff829859a4 in do_futex_wait (sem=sem@entry=0xaaaab06af4b0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x0000ffff82985a5c in __new_sem_wait_slow64 (sem=0xaaaab06af4b0 <thread_pool+176>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0xffff82985970 <__sem_wait_cleanup>, __arg = 0xaaaab06af4b0 <thread_pool+176>, __canceltype = -1335595012, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x0000aaaab0646ffc in request_handler_thread (arg=0xaaaace458080) at src/main/threads.c:870
        self = 0xaaaace458080
#6  0x0000ffff8297d5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472802902272, 281474145862304, 281474145862302, 8448320, 281474145862303, 0, 281472794451968, 8448320, 281472880116800, 281472794451968, 281472802900000, 4723273040259771725, 0, 4723273043536081265, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#7  0x0000ffff829e5edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 2 (Thread 0xffff7f705100 (LWP 145)):
#0  __GI___libc_write (nbytes=60, buf=0xaaaad6b32690, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/write.c:26
        _x2tmp = 60
        _x0tmp = <optimized out>
        _x0 = 60
        _x2 = 60
        _x8 = 64
        _x1tmp = 187650723227280
        _x1 = 187650723227280
        _sys_result = <optimized out>
        sc_ret = <optimized out>
        sc_cancel_oldtype = 0
        _x0tmp = <optimized out>
        _x0 = <optimized out>
        sc_ret = <optimized out>
        sc_ret = <optimized out>
        _sys_result = <optimized out>
        _x1tmp = <optimized out>
        _x1 = <optimized out>
        _x2tmp = <optimized out>
        _x2 = <optimized out>
        _x8 = <optimized out>
#1  __GI___libc_write (fd=<optimized out>, buf=0xaaaad6b32690, nbytes=60) at ../sysdeps/unix/sysv/linux/write.c:24
No locals.
#2  0x0000aaaab066edb8 in tls_socket_write (listener=0xaaaad0c1fab0) at src/main/tls_listen.c:115
        rcode = 0
        sock = 0xaaaaceba0290
#3  0x0000aaaab0671680 in dual_tls_send (listener=0xaaaad0c1fab0, request=0xaaaad0e9f690) at src/main/tls_listen.c:1022
        sock = 0xaaaaceba0290
#4  0x0000aaaab064c408 in request_finish (request=0xaaaad0e9f690, action=1) at src/main/process.c:1669
--Type <RET> for more, q to quit, c to continue without paging--
        vp = 0x0
#5  0x0000aaaab064c7d8 in request_running (request=0xaaaad0e9f690, action=1) at src/main/process.c:1798
        rcode = 0
        __FUNCTION__ = "request_running"
#6  0x0000aaaab0647174 in request_handler_thread (arg=0xaaaace463420) at src/main/threads.c:941
        self = 0xaaaace463420
#7  0x0000ffff8297d5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = 0x0
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {281472819810560, 281474145862304, 281474145862302, 8448320, 281474145862303, 0, 281472811360256, 8448320, 281472880116800, 281472811360256, 281472819808288, 4723273040259771725, 0, 4723273043520745841, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#8  0x0000ffff829e5edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
No locals.

Thread 1 (Thread 0xffff83088440 (LWP 142)):
#0  __pthread_kill_implementation (threadid=281472880116800, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
        tid = 142
        ret = 0
        pd = 0xffff83088440
        old_mask = {__val = {281472872458380, 281474145861072, 281472872646004, 281472873907448, 281474145861184, 281472873610280, 0, 281472873589424, 281472873897984, 281474145861440, 281474145861440, 281474145861392, 18446743528248704976, 38678256952, 281472873610280, 0}}
        ret = <optimized out>
#1  0x0000ffff8297f254 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
No locals.
#2  0x0000ffff8293a67c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#3  0x0000ffff82927130 in __GI_abort () at ./stdlib/abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0xffff82a4eab0, sa_sigaction = 0xffff82a4eab0}, sa_mask = {__val = {281472873897984, 0, 281474145863320, 4222124650659840, 17298308644996116495, 17298308644996116495, 17298308644996116495, 281474145861392, 281472872648304, 281472873907448, 15381545684843038464, 281474145861456, 281472872431564, 281472880332800, 281472873579856, 281472873589232}}, sa_flags = -2103130800, sa_restorer = 0xffff82a4e9f0}
        sigs = {__val = {32, 18446743528248704976, 281474145861456, 281472872431424, 281474145861440, 281474145861440, 281474145861392, 18446743528248704976, 2987131702627757358, 281474145861440, 281474145861376, 281472872799364, 281472880118784, 64, 0, 94}}
#4  0x0000ffff82933fd0 in __assert_fail_base (fmt=0xffff82a4c550 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xffff82a4eab0 "mutex->__data.__owner == 0", file=file@entry=0xffff82a4e9f0 "pthread_mutex_lock.c", line=line@entry=94, function=function@entry=0xffff82a4ec38 <__PRETTY_FUNCTION__.0> "___pthread_mutex_lock") at ./assert/assert.c:92
        str = 0xaaaacfc7b8c0 "{\374\254\252\n"
        total = 4096
#5  0x0000ffff82934040 in __GI___assert_fail (assertion=0xffff82a4eab0 "mutex->__data.__owner == 0", file=0xffff82a4e9f0 "pthread_mutex_lock.c", line=94, function=0xffff82a4ec38 <__PRETTY_FUNCTION__.0> "___pthread_mutex_lock") at ./assert/assert.c:101
No locals.
#6  0x0000ffff829808a8 in ___pthread_mutex_lock (mutex=0xaaaaceba03b8) at ./nptl/pthread_mutex_lock.c:94
        type = <optimized out>
        __PRETTY_FUNCTION__ = "___pthread_mutex_lock"
        id = <optimized out>
#7  0x0000aaaab066fe54 in tls_socket_recv (listener=0xaaaad0c1fab0) at src/main/tls_listen.c:497
        doing_init = false
        already_read = false
        rcode = 187650639681232
        data_len = 3927437169
        packet = 0xffffce7a4100
        request = 0xaaaacf25e1d0
        sock = 0xaaaaceba0290
        status = FR_TLS_INVALID
#8  0x0000aaaab0670b34 in dual_tls_recv (listener=0xaaaad0c1fab0) at src/main/tls_listen.c:770
        packet = 0xaaaab06a78f0
        fun = 0x0
        sock = 0xaaaaceba0290
        client = 0xaaaace4a2ff0
        rbio = 0x3
#9  0x0000aaaab0655200 in event_socket_handler (xel=0xaaaace412b20, fd=33, ctx=0xaaaad0c1fab0) at src/main/process.c:5415
        listener = 0xaaaad0c1fab0
#10 0x0000ffff82ffcca4 in fr_event_loop (el=0xaaaace412b20) at src/lib/event.c:721
        ef = 0xaaaace412d78
        i = 9
        rcode = 1
        when = {tv_sec = 1737998624, tv_usec = 665702}
        wake = 0xffffce7a4318
        read_fds = {fds_bits = {8589934592, 0 <repeats 15 times>}}
        write_fds = {fds_bits = {0 <repeats 16 times>}}
#11 0x0000aaaab06576cc in radius_event_process () at src/main/process.c:6682
No locals.
#12 0x0000aaaab063da7c in main (argc=3, argv=0xffffce7a4678) at src/main/radiusd.c:653
        rcode = 0
        status = 0
        argval = -1
        spawn_flag = true
        display_version = false
        flag = 0
        from_child = {-1, -1}
        p = 0x0
        state = 0xaaaab06aed88 <global_state>
        autofree = 0xaaaace087300
(gdb) 

[alandekok]

It might be an issue local to the system? See comments here: https://debian-bugs-dist.debian.narkive.com/IttE8Mma/bug-479952-libc6-s390-pthread-mutex-lock-assertion-mutex-data-owner-0-failed

Otherwise, perhaps building it with sanitizer flags would help:

check out and configure as normal, then edit Make.inc, and add the following to CFLAGS and LDFLAGS:

-fsanitize=undefined,leak,thread

not all of those are available on all versions of GCC. But they should be available on many recent ones.

The outcome of that will be a binary which may run more slowly, but which will complain loudly and clearly if anything goes wrong. The sanitizer output will produce stack traces of where the memory was modified, and not just where it crashed.

[gk-fschubert]

It might be an issue local to the system?

I've cleaned up my local state of images and also made some other adjustments to be sure that there is no caching mechanism in place which could break the resulting binary and I've also build the server on a dedicated VM to ensure that there is no issue in translating everything from ARM to x86_64.

And then the server was running and running. So i though you're right and the issue is my local build system.
But then it crashed again. It now needs ~20 minutes (in the most cases, in rare cases ~5) to get the server to crash.

I've then switched the build mashine to our git runners in order to see if it's reproducible with them. And yes it is.

The following errors are from images which where not build on my machine.

So if the error can be traced back to a error on the build system, it should be a issue with the build system used in ubuntu:22.04 docker images.

Regarding the sanitizer flags.

The build with -fsanitize=undefined,leak,thread failed as 'thread' and 'leak' seems to be incompatible

#8 1.215 cc1: error: '-fsanitize=leak' is incompatible with '-fsanitize=thread'
#8 1.215 cc1: error: '-fsanitize=leak' is incompatible with '-fsanitize=thread'
#8 1.216 make: *** [scripts/boiler.mk:643: build/objs/src/lib/cbuff.lo] Error 1
#8 1.216 make: *** Waiting for unfinished jobs....
#8 1.217 make: *** [scripts/boiler.mk:643: build/objs/src/lib/cursor.lo] Error 1
#8 1.218 cc1: error: '-fsanitize=leak' is incompatible with '-fsanitize=thread'
#8 1.219 make: *** [scripts/boiler.mk:643: build/objs/src/lib/debug.lo] Error 1

Run with build sanitizer flags -fsanitize=undefined,thread (running with fxxx` that you can the the flags)

root@radius-786fd7c7c9-fmkxm:~# radiusd -fxxxl stdout
Fri Jan 31 23:44:04 2025 : Debug: Server was built with: 
Fri Jan 31 23:44:04 2025 : Debug:   accounting                : yes
Fri Jan 31 23:44:04 2025 : Debug:   authentication            : yes
Fri Jan 31 23:44:04 2025 : Debug:   ascend-binary-attributes  : yes
Fri Jan 31 23:44:04 2025 : Debug:   coa                       : yes
Fri Jan 31 23:44:04 2025 : Debug:   recv-coa-from-home-server : no
Fri Jan 31 23:44:04 2025 : Debug:   control-socket            : yes
Fri Jan 31 23:44:04 2025 : Debug:   detail                    : yes
Fri Jan 31 23:44:04 2025 : Debug:   dhcp                      : yes
Fri Jan 31 23:44:04 2025 : Debug:   dynamic-clients           : yes
Fri Jan 31 23:44:04 2025 : Debug:   osfc2                     : no
Fri Jan 31 23:44:04 2025 : Debug:   proxy                     : yes
Fri Jan 31 23:44:04 2025 : Debug:   regex-pcre                : yes
Fri Jan 31 23:44:04 2025 : Debug:   regex-posix               : no
Fri Jan 31 23:44:04 2025 : Debug:   regex-posix-extended      : no
Fri Jan 31 23:44:04 2025 : Debug:   session-management        : yes
Fri Jan 31 23:44:04 2025 : Debug:   stats                     : yes
Fri Jan 31 23:44:04 2025 : Debug:   systemd                   : no
Fri Jan 31 23:44:04 2025 : Debug:   tcp                       : yes
Fri Jan 31 23:44:04 2025 : Debug:   threads                   : yes
Fri Jan 31 23:44:04 2025 : Debug:   tls                       : yes
Fri Jan 31 23:44:04 2025 : Debug:   unlang                    : yes
Fri Jan 31 23:44:04 2025 : Debug:   vmps                      : yes
Fri Jan 31 23:44:04 2025 : Debug:   developer                 : yes
Fri Jan 31 23:44:04 2025 : Debug: Server core libs:
Fri Jan 31 23:44:04 2025 : Debug:   freeradius-server         : 3.2.8
Fri Jan 31 23:44:04 2025 : Debug:   talloc                    : 2.3.*
Fri Jan 31 23:44:04 2025 : Debug:   ssl                       : 3.0.2
Fri Jan 31 23:44:04 2025 : Debug:   pcre                      : 8.39 2016-06-14
Fri Jan 31 23:44:04 2025 : Debug: Endianness:
Fri Jan 31 23:44:04 2025 : Debug:   little
Fri Jan 31 23:44:04 2025 : Debug: Compilation flags:
Fri Jan 31 23:44:04 2025 : Debug:   cppflags : 
Fri Jan 31 23:44:04 2025 : Debug:   cflags   : -I. -Isrc -include src/freeradius-devel/autoconf.h -include src/freeradius-devel/build.h -include src/freeradius-devel/features.h -include src/freeradius-devel/radpaths.h -fno-strict-aliasing -Wno-date-time -g3 -Wall -std=c99 -D_GNU_SOURCE -D_REENTRANT -D_POSIX_PTHREAD_SEMANTICS -DOPENSSL_NO_KRB5 -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -W -Wredundant-decls -Wundef -Wformat-y2k -Wno-format-extra-args -Wno-format-zero-length -Wno-cast-align -Wformat-nonliteral -Wformat-security -Wformat=2 -DWITH_VERIFY_PTR=1 -fsanitize=undefined,thread -DIS_MODULE=1
Fri Jan 31 23:44:04 2025 : Debug:   ldflags  :  -fsanitize=undefined,thread
Fri Jan 31 23:44:04 2025 : Debug:   libs     : -lcrypto -lssl -ltalloc -latomic -lpcre -lnsl -lresolv -ldl -lpthread 
Fri Jan 31 23:44:04 2025 : Debug:   
Fri Jan 31 23:44:04 2025 : Info: FreeRADIUS Version 3.2.8
Fri Jan 31 23:44:04 2025 : Info: Copyright (C) 1999-2023 The FreeRADIUS server project and contributors
Fri Jan 31 23:44:04 2025 : Info: There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
Fri Jan 31 23:44:04 2025 : Info: PARTICULAR PURPOSE
Fri Jan 31 23:44:04 2025 : Info: You may redistribute copies of FreeRADIUS under the terms of the
Fri Jan 31 23:44:04 2025 : Info: GNU General Public License
Fri Jan 31 23:44:04 2025 : Info: For more information about these matters, see the file named COPYRIGHT
Fri Jan 31 23:44:04 2025 : Info: Starting - reading configuration files ...
Fri Jan 31 23:44:04 2025 : Debug: including dictionary file /usr/local/share/freeradius/dictionary
Fri Jan 31 23:44:05 2025 : Debug: including dictionary file /usr/local/share/freeradius/dictionary.dhcp
Fri Jan 31 23:44:05 2025 : Debug: including dictionary file /usr/local/share/freeradius/dictionary.vqp
Fri Jan 31 23:44:05 2025 : Debug: including dictionary file /usr/local/etc/raddb/dictionary
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/radiusd.conf
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/proxy.conf
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/clients.conf
Fri Jan 31 23:44:05 2025 : Debug: including files in directory /usr/local/etc/raddb/mods-enabled/
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/detail
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/mschap
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/radutmp
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/files
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/unix
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/detail.log
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/realm
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/attr_filter
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/proxy_rate_limit
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/soh
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/dynamic_clients
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/expiration
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/passwd
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/always
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/chap
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/eap
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/ntlm_auth
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/totp
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/date
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/expr
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/replicate
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/echo
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/pap
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/sradutmp
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/digest
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/unpack
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/preprocess
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/logintime
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/linelog
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/utf8
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/mods-enabled/exec
Fri Jan 31 23:44:05 2025 : Debug: including files in directory /usr/local/etc/raddb/policy.d/
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/filter
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/cui
Fri Jan 31 23:44:05 2025 : Debug: OPTIMIZING (${policy.cui_require_operator_name} == yes) --> FALSE
Fri Jan 31 23:44:05 2025 : Debug: OPTIMIZING (no == yes) --> FALSE
Fri Jan 31 23:44:05 2025 : Debug: OPTIMIZING (${policy.cui_require_operator_name} == yes) --> FALSE
Fri Jan 31 23:44:05 2025 : Debug: OPTIMIZING (no == yes) --> FALSE
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/operator-name
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/canonicalization
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/eap
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/accounting
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/control
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/abfab-tr
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/rfc7542
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/moonshot-targeted-ids
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/debug
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/policy.d/dhcp
Fri Jan 31 23:44:05 2025 : Debug: including files in directory /usr/local/etc/raddb/sites-enabled/
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/sites-enabled/inner-tunnel
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/sites-enabled/default
Fri Jan 31 23:44:05 2025 : Debug: including configuration file /usr/local/etc/raddb/sites-enabled/tls
Fri Jan 31 23:44:05 2025 : Debug: main {
    Fri Jan 31 23:44:05 2025 : Debug:  security {
    Fri Jan 31 23:44:05 2025 : Debug:  allow_core_dumps = yes
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[588]: The item 'max_attributes' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[605]: The item 'reject_delay' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[625]: The item 'status_server' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[723]: The item 'require_message_authenticator' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[814]: The item 'limit_proxy_state' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[824]: The item 'allow_vulnerable_openssl' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug: name = "radiusd"
Fri Jan 31 23:44:05 2025 : Debug: prefix = "/usr/local"
Fri Jan 31 23:44:05 2025 : Debug: localstatedir = "/usr/local/var"
Fri Jan 31 23:44:05 2025 : Debug: logdir = "/usr/local/var/log/radius"
Fri Jan 31 23:44:05 2025 : Debug: run_dir = "/usr/local/var/run/radiusd"
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[97]: The item 'confdir' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[104]: The item 'db_dir' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[138]: The item 'libdir' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[149]: The item 'pidfile' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[203]: The item 'max_request_time' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[222]: The item 'cleanup_delay' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[259]: The item 'hostname_lookups' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[434]: The item 'checkrad' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[843]: The item 'proxy_requests' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Debug: }
Fri Jan 31 23:44:05 2025 : Info: Core dumps are enabled
Fri Jan 31 23:44:05 2025 : Debug: main {
    Fri Jan 31 23:44:05 2025 : Debug: name = "radiusd"
Fri Jan 31 23:44:05 2025 : Debug: prefix = "/usr/local"
Fri Jan 31 23:44:05 2025 : Debug: localstatedir = "/usr/local/var"
Fri Jan 31 23:44:05 2025 : Debug: sbindir = "/usr/local/sbin"
Fri Jan 31 23:44:05 2025 : Debug: logdir = "/usr/local/var/log/radius"
Fri Jan 31 23:44:05 2025 : Debug: run_dir = "/usr/local/var/run/radiusd"
Fri Jan 31 23:44:05 2025 : Debug: libdir = "/usr/local/lib"
Fri Jan 31 23:44:05 2025 : Debug: radacctdir = "/usr/local/var/log/radius/radacct"
Fri Jan 31 23:44:05 2025 : Debug: hostname_lookups = no
Fri Jan 31 23:44:05 2025 : Debug: max_request_time = 30
Fri Jan 31 23:44:05 2025 : Debug: proxy_dedup_window = 1
Fri Jan 31 23:44:05 2025 : Debug: cleanup_delay = 5
Fri Jan 31 23:44:05 2025 : Debug: max_requests = 16384
Fri Jan 31 23:44:05 2025 : Debug: max_fds = 512
Fri Jan 31 23:44:05 2025 : Debug: postauth_client_lost = no
Fri Jan 31 23:44:05 2025 : Debug: pidfile = "/usr/local/var/run/radiusd/radiusd.pid"
Fri Jan 31 23:44:05 2025 : Debug: checkrad = "/usr/local/sbin/checkrad"
Fri Jan 31 23:44:05 2025 : Debug: debug_level = 0
Fri Jan 31 23:44:05 2025 : Debug: proxy_requests = yes
Fri Jan 31 23:44:05 2025 : Debug:  log {
    Fri Jan 31 23:44:05 2025 : Debug:  stripped_names = no
Fri Jan 31 23:44:05 2025 : Debug:  auth = no
Fri Jan 31 23:44:05 2025 : Debug:  auth_badpass = no
Fri Jan 31 23:44:05 2025 : Debug:  auth_goodpass = no
Fri Jan 31 23:44:05 2025 : Debug:  colourise = yes
Fri Jan 31 23:44:05 2025 : Debug:  msg_denied = "You are already logged in - access denied"
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[330]: The item 'destination' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[347]: The item 'file' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[355]: The item 'syslog_facility' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  resources {
    Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  security {
    Fri Jan 31 23:44:05 2025 : Debug:  max_attributes = 200
Fri Jan 31 23:44:05 2025 : Debug:  reject_delay = 1.000000
Fri Jan 31 23:44:05 2025 : Debug:  status_server = yes
Fri Jan 31 23:44:05 2025 : Debug:  require_message_authenticator = "auto"
Fri Jan 31 23:44:05 2025 : Debug:  limit_proxy_state = "auto"
Fri Jan 31 23:44:05 2025 : Debug:  allow_vulnerable_openssl = "no"
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  unlang {
    Fri Jan 31 23:44:05 2025 : Debug:  group_stop_return = no
Fri Jan 31 23:44:05 2025 : Debug:  policy_stop_return = no
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[97]: The item 'confdir' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/radiusd.conf[104]: The item 'db_dir' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Debug: }
Fri Jan 31 23:44:05 2025 : Debug: radiusd: #### Loading Realms and Home Servers ####
Fri Jan 31 23:44:05 2025 : Debug:  proxy server {
    Fri Jan 31 23:44:05 2025 : Debug:  retry_delay = 5
Fri Jan 31 23:44:05 2025 : Debug:  retry_count = 3
Fri Jan 31 23:44:05 2025 : Debug:  default_fallback = no
Fri Jan 31 23:44:05 2025 : Debug:  dead_time = 120
Fri Jan 31 23:44:05 2025 : Debug:  wake_all_if_all_dead = no
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  home_server localhost {
    Fri Jan 31 23:44:05 2025 : Debug:  nonblock = no
Fri Jan 31 23:44:05 2025 : Debug:  ipaddr = 127.0.0.1
Fri Jan 31 23:44:05 2025 : Debug:  port = 1812
Fri Jan 31 23:44:05 2025 : Debug:  type = "auth"
Fri Jan 31 23:44:05 2025 : Debug:  secret = "testing123"
Fri Jan 31 23:44:05 2025 : Debug:  response_window = 20.000000
Fri Jan 31 23:44:05 2025 : Debug:  response_timeouts = 1
Fri Jan 31 23:44:05 2025 : Debug:  max_outstanding = 65536
Fri Jan 31 23:44:05 2025 : Debug:  zombie_period = 40
Fri Jan 31 23:44:05 2025 : Debug:  status_check = "status-server"
Fri Jan 31 23:44:05 2025 : Debug:  ping_interval = 30
Fri Jan 31 23:44:05 2025 : Debug:  check_interval = 30
Fri Jan 31 23:44:05 2025 : Debug:  check_timeout = 4
Fri Jan 31 23:44:05 2025 : Debug:  num_answers_to_alive = 3
Fri Jan 31 23:44:05 2025 : Debug:  revive_interval = 120
Fri Jan 31 23:44:05 2025 : Debug:   limit {
    Fri Jan 31 23:44:05 2025 : Debug:   max_connections = 16
Fri Jan 31 23:44:05 2025 : Debug:   max_requests = 0
Fri Jan 31 23:44:05 2025 : Debug:   lifetime = 0
Fri Jan 31 23:44:05 2025 : Debug:   idle_timeout = 0
Fri Jan 31 23:44:05 2025 : Debug:   }
Fri Jan 31 23:44:05 2025 : Debug:   coa {
    Fri Jan 31 23:44:05 2025 : Debug:   irt = 2
Fri Jan 31 23:44:05 2025 : Debug:   mrt = 16
Fri Jan 31 23:44:05 2025 : Debug:   mrc = 5
Fri Jan 31 23:44:05 2025 : Debug:   mrd = 30
Fri Jan 31 23:44:05 2025 : Debug:   }
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  home_server tls {
    Fri Jan 31 23:44:05 2025 : Debug:  nonblock = no
Fri Jan 31 23:44:05 2025 : Debug:  ipaddr = 127.0.0.1
Fri Jan 31 23:44:05 2025 : Debug:  port = 2083
Fri Jan 31 23:44:05 2025 : Debug:  type = "auth"
Fri Jan 31 23:44:05 2025 : Debug:  proto = "tcp"
Fri Jan 31 23:44:05 2025 : Debug:  secret = "radsec"
Fri Jan 31 23:44:05 2025 : Debug:  response_window = 30.000000
Fri Jan 31 23:44:05 2025 : Debug:  response_timeouts = 1
Fri Jan 31 23:44:05 2025 : Debug:  max_outstanding = 65536
Fri Jan 31 23:44:05 2025 : Debug:  zombie_period = 40
Fri Jan 31 23:44:05 2025 : Debug:  status_check = "none"
Fri Jan 31 23:44:05 2025 : Debug:  ping_interval = 30
Fri Jan 31 23:44:05 2025 : Debug:  check_timeout = 4
Fri Jan 31 23:44:05 2025 : Debug:  num_answers_to_alive = 3
Fri Jan 31 23:44:05 2025 : Debug:  revive_interval = 300
Fri Jan 31 23:44:05 2025 : Debug:   limit {
    Fri Jan 31 23:44:05 2025 : Debug:   max_connections = 16
Fri Jan 31 23:44:05 2025 : Debug:   max_requests = 0
Fri Jan 31 23:44:05 2025 : Debug:   lifetime = 0
Fri Jan 31 23:44:05 2025 : Debug:   idle_timeout = 0
Fri Jan 31 23:44:05 2025 : Debug:   }
Fri Jan 31 23:44:05 2025 : Debug:   coa {
    Fri Jan 31 23:44:05 2025 : Debug:   irt = 2
Fri Jan 31 23:44:05 2025 : Debug:   mrt = 16
Fri Jan 31 23:44:05 2025 : Debug:   mrc = 5
Fri Jan 31 23:44:05 2025 : Debug:   mrd = 30
Fri Jan 31 23:44:05 2025 : Debug:   }
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:   tls {
    Fri Jan 31 23:44:05 2025 : Debug:   verify_depth = 0
Fri Jan 31 23:44:05 2025 : Debug:   ca_path = "/usr/local/etc/raddb/certs"
Fri Jan 31 23:44:05 2025 : Debug:   pem_file_type = yes
Fri Jan 31 23:44:05 2025 : Debug:   private_key_file = "/usr/local/etc/raddb/certs/client.pem"
Fri Jan 31 23:44:05 2025 : Debug:   certificate_file = "/usr/local/etc/raddb/certs/client.pem"
Fri Jan 31 23:44:05 2025 : Debug:   ca_file = "/usr/local/etc/raddb/certs/ca.pem"
Fri Jan 31 23:44:05 2025 : Debug:   private_key_password = "whatever"
Fri Jan 31 23:44:05 2025 : Debug:   fragment_size = 8192
Fri Jan 31 23:44:05 2025 : Debug:   include_length = yes
Fri Jan 31 23:44:05 2025 : Debug:   check_crl = no
Fri Jan 31 23:44:05 2025 : Debug:   cipher_list = "DEFAULT"
Fri Jan 31 23:44:05 2025 : Debug:   ca_path_reload_interval = 0
Fri Jan 31 23:44:05 2025 : Debug:   ecdh_curve = "prime256v1"
Fri Jan 31 23:44:05 2025 : Debug:   tls_min_version = "1.2"
Fri Jan 31 23:44:05 2025 : Warning: /usr/local/etc/raddb/sites-enabled/tls[741]: The item 'connect_timeout' is defined, but is unused by the configuration
Fri Jan 31 23:44:05 2025 : Debug:   }
Fri Jan 31 23:44:05 2025 : Debug:   connect_timeout = 30
Fri Jan 31 23:44:05 2025 : Debug:  home_server_pool my_auth_failover {
    Fri Jan 31 23:44:05 2025 : Debug: type = fail-over
Fri Jan 31 23:44:05 2025 : Debug: home_server = localhost
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  realm example.com {
    Fri Jan 31 23:44:05 2025 : Debug: auth_pool = my_auth_failover
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  realm LOCAL {
    Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  home_server_pool tls {
    Fri Jan 31 23:44:05 2025 : Debug: type = fail-over
Fri Jan 31 23:44:05 2025 : Debug: home_server = tls
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug:  realm tls {
    Fri Jan 31 23:44:05 2025 : Debug: auth_pool = tls
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Debug: radiusd: #### Loading Clients ####
Fri Jan 31 23:44:05 2025 : Debug:  client localhost {
    Fri Jan 31 23:44:05 2025 : Debug:  ipaddr = 127.0.0.1
Fri Jan 31 23:44:05 2025 : Debug:  secret = "testing123"
Fri Jan 31 23:44:05 2025 : Debug:  nas_type = "other"
Fri Jan 31 23:44:05 2025 : Debug:  proto = "*"
Fri Jan 31 23:44:05 2025 : Debug:   limit {
    Fri Jan 31 23:44:05 2025 : Debug:   max_connections = 16
Fri Jan 31 23:44:05 2025 : Debug:   lifetime = 0
Fri Jan 31 23:44:05 2025 : Debug:   idle_timeout = 900
Fri Jan 31 23:44:05 2025 : Debug:   }
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Warning: Shared secret for client localhost is short, and likely can be broken by an attacker.
Fri Jan 31 23:44:05 2025 : Debug: Adding client 127.0.0.1/32 (127.0.0.1) to prefix tree 32
Fri Jan 31 23:44:05 2025 : Debug:  client localhost_ipv6 {
    Fri Jan 31 23:44:05 2025 : Debug:  ipv6addr = ::1
Fri Jan 31 23:44:05 2025 : Debug:  secret = "testing123"
Fri Jan 31 23:44:05 2025 : Debug:   limit {
    Fri Jan 31 23:44:05 2025 : Debug:   max_connections = 16
Fri Jan 31 23:44:05 2025 : Debug:   lifetime = 0
Fri Jan 31 23:44:05 2025 : Debug:   idle_timeout = 30
Fri Jan 31 23:44:05 2025 : Debug:   }
Fri Jan 31 23:44:05 2025 : Debug:  }
Fri Jan 31 23:44:05 2025 : Warning: Shared secret for client localhost_ipv6 is short, and likely can be broken by an attacker.
Fri Jan 31 23:44:05 2025 : Debug: Adding client ::1/128 (::1) to prefix tree 128
Fri Jan 31 23:44:05 2025 : Info: Found debugger attached
Fri Jan 31 23:44:05 2025 : Info: Configuration version: 5a62-1fc9-e6fd-90d5
Fri Jan 31 23:44:05 2025 : Debug:  # Creating Auth-Type = mschap
Fri Jan 31 23:44:05 2025 : Debug:  # Creating Auth-Type = eap
Fri Jan 31 23:44:05 2025 : Debug:  # Creating Auth-Type = PAP
Fri Jan 31 23:44:05 2025 : Debug:  # Creating Auth-Type = CHAP
Fri Jan 31 23:44:05 2025 : Debug:  # Creating Auth-Type = MS-CHAP
Fri Jan 31 23:44:05 2025 : Debug:  # Creating Auth-Type = digest
Fri Jan 31 23:44:05 2025 : Debug:  # Creating Autz-Type = New-TLS-Connection
Fri Jan 31 23:44:05 2025 : Debug: radiusd: #### Instantiating modules ####
Fri Jan 31 23:44:05 2025 : Debug:  modules {
    ==135==You are trying to dlopen a /usr/local/lib/rlm_detail.so shared library with RTLD_DEEPBIND flag which is incompatible with sanitizer runtime (see https://github.com/google/sanitizers/issues/611 for details). If you want to run /usr/local/lib/rlm_detail.so library under sanitizers please remove RTLD_DEEPBIND from dlopen flags.

Run with build sanitizer flags `-fsanitize=undefined,leak

(59494) (TLS) Checking connection to see if it is authorized.
(59494) # Executing group from file /usr/local/etc/raddb/sites-enabled/default
(59494)   Autz-Type New-TLS-Connection {
    (59494)     [ok] = ok
(59494)   } # Autz-Type New-TLS-Connection = ok
(59494) (TLS) Connection is authorized
(0) (TLS) We already have 14 bytes of application data, processing it.
(59494) Sent Access-Accept Id 0 from 0.0.0.0:2083 to 10.244.5.183:46030 length 38
radiusd: pthread_mutex_lock.c:94: ___pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
(0) (TLS) RADIUS/TLS - send TLS 1.3 Handshake, Finished
Aborted (core dumped)
root@radius-57f47b5676-m7286:/etc/raddb/certs# gdb $(which radiusd) core.133 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/radiusd...
[New LWP 133]
[New LWP 140]
[New LWP 137]
[New LWP 139]
[New LWP 138]
[New LWP 136]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `radiusd -fxxl stdout'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140581393331072) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7fdba802f380 (LWP 133))]
(gdb) info threads
  Id   Target Id                       Frame 
* 1    Thread 0x7fdba802f380 (LWP 133) __pthread_kill_implementation (no_tid=0, signo=6, threadid=140581393331072) at ./nptl/pthread_kill.c:44
  2    Thread 0x7fdba2a6b640 (LWP 140) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, 
    futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:57
  3    Thread 0x7fdba426e640 (LWP 137) fr_utf8_char (str=0x7fdba425adb1 "59494) Finished request", inlen=-1) at src/lib/print.c:36
  4    Thread 0x7fdba326c640 (LWP 139) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, 
    futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:57
  5    Thread 0x7fdba3a6d640 (LWP 138) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, 
    futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:57
  6    Thread 0x7fdba4a6f640 (LWP 136) __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, 
    futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:57
(gdb) thread apply all bt full

Thread 6 (Thread 0x7fdba4a6f640 (LWP 136)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x56088876c6a8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007fdba859dbdf in do_futex_wait (sem=sem@entry=0x56088876c6a8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x00007fdba859dc78 in __new_sem_wait_slow64 (sem=0x56088876c6a8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0x7fdba859dbb0 <__sem_wait_cleanup>, __arg = 0x56088876c6a8 <thread_pool+168>, __canceltype = 0, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x00005608885f37a3 in request_handler_thread (arg=0x604000006400) at src/main/threads.c:870
        self = 0x604000006400
#6  0x00007fdba8595ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140725851035168, 8144444783962029356, 140581336970816, 0, 140581398992848, 140725851035520, -8164559736531387092, -8164549125600903892}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#7  0x00007fdba8627850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.

Thread 5 (Thread 0x7fdba3a6d640 (LWP 138)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x56088876c6a8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007fdba859dbdf in do_futex_wait (sem=sem@entry=0x56088876c6a8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x00007fdba859dc78 in __new_sem_wait_slow64 (sem=0x56088876c6a8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0x7fdba859dbb0 <__sem_wait_cleanup>, __arg = 0x56088876c6a8 <thread_pool+168>, __canceltype = 0, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x00005608885f37a3 in request_handler_thread (arg=0x604000006540) at src/main/threads.c:870
        self = 0x604000006540
#6  0x00007fdba8595ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140725851035168, 8144444783962029356, 140581320185408, 0, 140581398992848, 140725851035520, -8164561947365802708, -8164549125600903892}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#7  0x00007fdba8627850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.

Thread 4 (Thread 0x7fdba326c640 (LWP 139)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x56088876c6a8 <thread_pool+168>) at .--Type <RET> for more, q to quit, c to continue without paging--
/nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x56088876c6a8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007fdba859dbdf in do_futex_wait (sem=sem@entry=0x56088876c6a8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x00007fdba859dc78 in __new_sem_wait_slow64 (sem=0x56088876c6a8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0x7fdba859dbb0 <__sem_wait_cleanup>, __arg = 0x56088876c6a8 <thread_pool+168>, __canceltype = 0, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x00005608885f37a3 in request_handler_thread (arg=0x6040000061c0) at src/main/threads.c:870
        self = 0x6040000061c0
#6  0x00007fdba8595ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140725851035168, 8144444783962029356, 140581311792704, 0, 140581398992848, 140725851035520, -8164560848391045844, -8164549125600903892}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#7  0x00007fdba8627850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.

Thread 3 (Thread 0x7fdba426e640 (LWP 137)):
#0  fr_utf8_char (str=0x7fdba425adb1 "59494) Finished request", inlen=-1) at src/lib/print.c:36
No locals.
#1  0x00007fdba9579d6c in vradlog (type=L_DBG, msg=0x7fdba95d2e91 "(%u) %.*s%s%s", ap=0x7fdba425d600) at src/main/log.c:450
        clen = 1
        p = 0x7fdba425adb1 "59494) Finished request"
        buffer = "(59494) Finished request\000zed.\n", '\000' <repeats 2906 times>...
        unsan = 0x7fdba425adb0 "(59494) Finished request"
        len = 24
        colourise = 0
#2  0x00007fdba957a3eb in radlog_always (type=L_DBG, msg=0x7fdba95d2e91 "(%u) %.*s%s%s") at src/main/log.c:557
        ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7fdba425d6e8, reg_save_area = 0x7fdba425d620}}
        r = 0
#3  0x00007fdba957b2ce in vradlog_request (type=L_DBG, lvl=L_DBG_LVL_2, request=0x6250000960e0, msg=0x56088869af63 "Finished request", ap=0x7fdba4260040) at src/main/log.c:765
        len = 0
        filename = 0x0
        fp = 0x0
        buffer = "Finished request\000zed.\000\000\000\324Xa\250\333\177\000\000\004\000\000\000\000\000\000\000\070\000\000\000\000\000\000\000\200\330%\244\333\177\000\000\070\000\000\000\000\000\000\000\240\000&\244\333\177\000\000\326\241W\251\333\177\000\000\000\000\000\000\000\000\000\000\320\000&\244\333\177\000\000x\227i\210\bV\000\000\000\000\000\000\020", '\000' <repeats 15 times>, "\001", '\000' <repeats 11 times>, "\267\330%\244\333\177\000\000\067\000\000\000\000\000\000\000\200\330%\244\333\177\000\000Thread 2 handling request 59494, (11899 "...
        p = 0x7fdba4260010 "\020\001&\244\333\177"
        extra = 0x7fdba95d2c2b ""
        indent = 0 '\000'
        aq = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fdba4260120, reg_save_area = 0x7fdba4260060}}
#4  0x00007fdba957b51c in radlog_request (type=L_DBG, lvl=L_DBG_LVL_2, request=0x6250000960e0, msg=0x56088869af63 "Finished request") at src/main/log.c:787
        ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fdba4260120, reg_save_area = 0x7fdba4260060}}
#5  0x000056088860421f in request_finish (request=0x6250000960e0, action=1) at src/main/process.c:1673
--Type <RET> for more, q to quit, c to continue without paging--
        vp = 0x0
#6  0x0000560888604fbf in request_running (request=0x6250000960e0, action=1) at src/main/process.c:1798
        rcode = 0
        __FUNCTION__ = "request_running"
#7  0x00005608885f3e83 in request_handler_thread (arg=0x604000006600) at src/main/threads.c:941
        self = 0x604000006600
#8  0x00007fdba8595ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140725851035168, 8144444783962029356, 140581328578112, 0, 140581398992848, 140725851035520, -8164558637556630228, -8164549125600903892}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#9  0x00007fdba8627850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.

Thread 2 (Thread 0x7fdba2a6b640 (LWP 140)):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:57
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x56088876c6a8 <thread_pool+168>) at ./nptl/futex-internal.c:87
        err = <optimized out>
        clockbit = 256
        op = 393
        err = <optimized out>
        clockbit = <optimized out>
        op = <optimized out>
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x56088876c6a8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
No locals.
#3  0x00007fdba859dbdf in do_futex_wait (sem=sem@entry=0x56088876c6a8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
        err = <optimized out>
#4  0x00007fdba859dc78 in __new_sem_wait_slow64 (sem=0x56088876c6a8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
        _buffer = {__routine = 0x7fdba859dbb0 <__sem_wait_cleanup>, __arg = 0x56088876c6a8 <thread_pool+168>, __canceltype = 0, __prev = 0x0}
        err = <optimized out>
        d = 17179869184
#5  0x00005608885f37a3 in request_handler_thread (arg=0x604000006440) at src/main/threads.c:870
        self = 0x604000006440
#6  0x00007fdba8595ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140725851035168, 8144444783962029356, 140581303400000, 0, 140581398992848, 140725851035520, -8164564145315316436, -8164549125600903892}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#7  0x00007fdba8627850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.


Thread 1 (Thread 0x7fdba802f380 (LWP 133)):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140581393331072) at ./nptl/pthread_kill.c:44
        tid = <optimized out>
        ret = 0
        pd = 0x7fdba802f380
        old_mask = {__val = {106721347483308, 106721347483008, 106721347483308, 0, 0, 0, 0, 0, 551003396064, 94594647908089, 5448501262943977472, 0, 108576774026624, 13359400107790121472, 18446744073709551615, 13359400107790121472}}
        ret = <optimized out>
        pd = <optimized out>
        old_mask = <optimized out>
        ret = <optimized out>
        tid = <optimized out>
        ret = <optimized out>
        resultvar = <optimized out>
        resultvar = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
        __futex = <optimized out>
        resultvar = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
        __futex = <optimized out>
        __private = <optimized out>
        __oldval = <optimized out>
        result = <optimized out>
#1  __pthread_kill_internal (signo=6, threadid=140581393331072) at ./nptl/pthread_kill.c:78
No locals.
#2  __GI___pthread_kill (threadid=140581393331072, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
No locals.
#3  0x00007fdba8543476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#4  0x00007fdba85297f3 in __GI_abort () at ./stdlib/abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0x607000050800, sa_sigaction = 0x607000050800}, sa_mask = {__val = {320, 140725851032224, 30, 4, 140581400319210, 0, 2288174152, 140725851034688, 94594648434776, 0, 140581415662876, 5, 140581400356240, 0, 68719476739, 140581400319210}}, sa_flags = -1468938592, sa_restorer = 0x7fdba7803000}
        sigs = {__val = {32, 140581400594080, 140581400323438, 94, 140581400323481, 140581422641216, 140581393331072, 140581398903022, 206158430256, 140725851034912, 206158430232, 140725851034912, 140725851034704, 13359400107790121472, 140725851037899, 140581400319650}}
#5  0x00007fdba852971b in __assert_fail_base (fmt=0x7fdba86de130 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7fdba86da599 "mutex->__data.__owner == 0", file=0x7fdba86da56e "pthread_mutex_lock.c", line=94, function=<optimized out>) at ./assert/assert.c:92
        str = 0x607000050800 "radiusd: pthread_mutex_lock.c:94: ___pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.\n"
        total = 4096
#6  0x00007fdba853ae96 in __GI___assert_fail (assertion=0x7fdba86da599 "mutex->__data.__owner == 0", file=0x7fdba86da56e "pthread_mutex_lock.c", line=94, fun--Type <RET> for more, q to quit, c to continue without paging--
ction=0x7fdba86e2840 <__PRETTY_FUNCTION__.0> "___pthread_mutex_lock") at ./assert/assert.c:101
No locals.
#7  0x00007fdba85990d0 in ___pthread_mutex_lock (mutex=<optimized out>) at ./nptl/pthread_mutex_lock.c:94
        type = <optimized out>
        __PRETTY_FUNCTION__ = "___pthread_mutex_lock"
        id = <optimized out>
#8  0x0000560888678e80 in tls_socket_recv (listener=0x6130000130e0) at src/main/tls_listen.c:497
        doing_init = false
        already_read = false
        rcode = 0
        data_len = 32
        packet = 0x60b000131e00
        request = 0x6130000321e0
        sock = 0x614000072e60
        status = FR_TLS_OK
#9  0x000056088867d62d in dual_tls_recv (listener=0x6130000130e0) at src/main/tls_listen.c:770
        packet = 0x7ffd4a5c9740
        fun = 0x0
        sock = 0x614000072e60
        client = 0x61900000eb60
        rbio = 0xb9661f9ded253a00
#10 0x000056088862c670 in event_socket_handler (xel=0x629000000060, fd=108, ctx=0x6130000130e0) at src/main/process.c:5415
        listener = 0x6130000130e0
#11 0x00007fdba9484035 in fr_event_loop (el=0x629000000060) at src/lib/event.c:721
        ef = 0x629000000378
        i = 15
        rcode = 1
        when = {tv_sec = 1738368604, tv_usec = 113581}
        wake = 0x7ffd4a5c98f0
        read_fds = {fds_bits = {0, 17592186044416, 0 <repeats 14 times>}}
        write_fds = {fds_bits = {0 <repeats 16 times>}}
#12 0x0000560888631c15 in radius_event_process () at src/main/process.c:6682
No locals.
#13 0x00005608885d0d13 in main (argc=3, argv=0x7ffd4a5c9c08) at src/main/radiusd.c:653
        rcode = 0
        status = 0
        argval = -1
        spawn_flag = true
        display_version = false
        flag = 0
        from_child = {-1, -1}
        p = 0x0
        state = 0x56088876bf80 <global_state>
        autofree = 0x606000000060