IPC segfault with 32345 + 29409 + custom client

[ryanofsky]

Originally posted by @darosior in bitcoin/bitcoin#32345 (comment)

I tested this PR by rebasing #29409 on top of it and running my (updated) core_bdk_wallet against it.

I found that bitcoin-node will (still) segfault if i try to stop it while the client is running. The steps i followed were to 1) start bitcoin-node 2) start my PoC which will connect to it and stall with the open connection for 6 seconds 3) before the 6 seconds have elapsed, stop the bitcoin-node process (in this specific case by sending it a SIGINT). Here are the logs of the bitcoin-node process:

Click for `bitcoin-node` logs before segfault

[...]
2025-05-20T18:44:44Z init message: Starting network threads…
2025-05-20T18:44:44Z DNS seeding disabled
2025-05-20T18:44:44Z init message: Done loading
2025-05-20T18:44:44Z initload thread start
2025-05-20T18:44:44Z msghand thread start
2025-05-20T18:44:44Z Loading 0 mempool transactions from file...
2025-05-20T18:44:44Z Imported mempool transactions from file: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
2025-05-20T18:44:44Z opencon thread start
2025-05-20T18:44:44Z addcon thread start
2025-05-20T18:44:44Z net thread start
2025-05-20T18:44:44Z initload thread exit
2025-05-20T18:44:44Z New manual v2 peer connected: version: 70016, blocks=113, peer=0
2025-05-20T18:44:44Z Synchronizing blockheaders, height: 113 (~0.47%)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  #1 Init.construct$Params ()
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response #1 Init.construct$Results (threadMap = <external capability>)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  #2 Init.makeChain$Params (context = (thread = <external capability>))
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  #2 {bitcoin-node-948798/b-capnp-loop-948823 (from )}
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response #2 Init.makeChain$Results (result = <external capability>)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  #3 Chain.initMessage$Params (context = (thread = <external capability>), message = "Oxydation of the Bitcoin Core wallet in progress..")
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  #3 {bitcoin-node-948798/b-capnp-loop-948823 (from )}
2025-05-20T18:44:45Z init message: Oxydation of the Bitcoin Core wallet in progress..
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response #3 Chain.initMessage$Results ()
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  #4 Chain.showProgress$Params (context = (thread = <external capability>), title = "BDK Core startup", progress = 1, resumePossible = false)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  #4 {bitcoin-node-948798/b-capnp-loop-948823 (from )}
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response #4 Chain.showProgress$Results ()
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  #5 Chain.getHeight$Params (context = (thread = <external capability>))
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  #5 {bitcoin-node-948798/b-capnp-loop-948823 (from )}
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response #5 Chain.getHeight$Results (result = 113, hasResult = true)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  #6 Chain.getBlockHash$Params (context = (thread = <external capability>), height = 113)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  #6 {bitcoin-node-948798/b-capnp-loop-948823 (from )}
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response #6 Chain.getBlockHash$Results (result = "H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD")
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  #7 Chain.showProgress$Params (context = (thread = <external capability>), title = "BDK Core startup", progress = 100, resumePossible = true)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  #7 {bitcoin-node-948798/b-capnp-loop-948823 (from )}
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response #7 Chain.showProgress$Results ()
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server recv request  #8 Chain.handleNotifications$Params (context = (thread = <external capability>), notifications = <external capability>)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server post request  #8 {bitcoin-node-948798/b-capnp-loop-948823 (from )}
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server send response #8 Chain.handleNotifications$Results (result = <external capability>)
2025-05-20T18:44:45Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages7HandlerEEE
^C2025-05-20T18:44:47Z Shutdown: In progress...
2025-05-20T18:44:47Z addcon thread exit
2025-05-20T18:44:47Z opencon thread exit
2025-05-20T18:44:47Z net thread exit
2025-05-20T18:44:47Z msghand thread exit
2025-05-20T18:44:47Z scheduler thread exit
2025-05-20T18:44:47Z Writing 0 mempool transactions to file...
2025-05-20T18:44:47Z Writing 0 unbroadcast transactions to file.
2025-05-20T18:44:47Z Dumped mempool: 0.000s to copy, 0.003s to dump, 27 bytes dumped to file
2025-05-20T18:44:47Z Flushed fee estimates to fee_estimates.dat.
2025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-shutoff-948798} IPC client first request from current thread, constructing waiter
2025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-shutoff-948798} IPC client send ChainNotifications.chainStateFlushed$Params (context = (thread = <external capability>, callbackThread = <external capability>), role = 0, locator = "\\200\\021\\001\\000\\022H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD\\tGm\\237]YbZDV9\\254\\252f\\205\\001\\323K\\375!\\340\\t\\367\\375\\346\\264W\\207\\000\\2101\\035:\\314\\233\\246\\275\\207\\v\\\\\\\\\\343\\202\\026\\274\\037\\213\\314\\303V\\2116<*+\\352\\255\\026\\204M\\332\\340\\nT\\306\\360Pa\\304\\002\\024\\356\\247\\220\\263\\257l\\230\\350\\362\\027Yl\\300\\316lk\\232\\f\\333\\235,\\234\\336\\204+JE\\263>\\237\\177\\305z\\017\\252\\311@\\341e\\026\\300dk\\305\\212V\\334\\377\\004cb\\217p\\n\\307w\\0046\\2261|\\024\\\\N\\336\\223G\\263~n\\271\\256\\264\\346\\227\\301\\032\\301\\030\\bw\\241Z.\\302\\261\\027 \\005\\374B\\357R\\231\\237Ct\\324\\232I\\322\\370\\303T\\253\\205\\354\\233\\337\\031C\\3215O\\3746\\016\\360b &\\300\\201\\244U\\221\\306 Pql+c\\212\\2524\\314a\\t\\216\\275=\\230b\\207ouv[\\365(\\3...
2025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-shutoff-948798} IPC client recv ChainNotifications.chainStateFlushed$Results ()
2025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
2025-05-20T18:44:47Z [ipc] {bitcoin-node-948798/b-capnp-loop-948800} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
Segmentation fault

[ryanofsky]

So far could not reproduce this combining bitcoin/bitcoin#32345 + bitcoin/bitcoin#29409 + bitcoin/bitcoin#30437 and shutting down node with control-c while client is still connected.

Code tested was: ryanofsky/bitcoin@42bf3ec (branch) and testing steps are in the commit description

One difference between the client I tested with and the custom client reported in this bug is that apparently from the logs the custom client is calling Chain.handleNotifcations and registering for notifications. So maybe the node is running into problems if the notifications ChainNotifications object is being destroyed too late, or something like that.

It would help to have a stack trace from the segfault. It should be easy to get a stack trace by running the node with:

gdb -ex run --args build/bin/bitcoin-node <...node arguments here...>

and typing bt (backtrace) after the segfault happens

[darosior]

Here is the backtrace

Logs when sending `SIGINT` from withing gdb + backtrace

2025-05-21T15:22:20Z init message: Done loading
2025-05-21T15:22:20Z msghand thread start
2025-05-21T15:22:20Z New manual v2 peer connected: version: 70016, blocks=113, peer=0
2025-05-21T15:22:20Z Synchronizing blockheaders, height: 113 (~0.47%)
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  #1 Init.construct$Params ()
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response #1 Init.construct$Results (threadMap = <external capability>)
[New Thread 0x7fffa8ffa6c0 (LWP 1160824)]
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  #2 Init.makeChain$Params (context = (thread = <external capability>))
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  #2 {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response #2 Init.makeChain$Results (result = <external capability>)
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  #3 Chain.initMessage$Params (context = (thread = <external capability>), message = "Oxydation of the Bitcoin Core wallet in progress..")
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  #3 {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
2025-05-21T15:22:21Z init message: Oxydation of the Bitcoin Core wallet in progress..
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response #3 Chain.initMessage$Results ()
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  #4 Chain.showProgress$Params (context = (thread = <external capability>), title = "BDK Core startup", progress = 1, resumePossible = false)
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  #4 {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response #4 Chain.showProgress$Results ()
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  #5 Chain.getHeight$Params (context = (thread = <external capability>))
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  #5 {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response #5 Chain.getHeight$Results (result = 113, hasResult = true)
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  #6 Chain.getBlockHash$Params (context = (thread = <external capability>), height = 113)
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  #6 {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response #6 Chain.getBlockHash$Results (result = "H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD")
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  #7 Chain.showProgress$Params (context = (thread = <external capability>), title = "BDK Core startup", progress = 100, resumePossible = true)
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  #7 {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response #7 Chain.showProgress$Results ()
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server recv request  #8 Chain.handleNotifications$Params (context = (thread = <external capability>), notifications = <external capability>)
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server post request  #8 {bitcoin-node-1160793/b-capnp-loop-1160824 (from )}
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server send response #8 Chain.handleNotifications$Results (result = <external capability>)
2025-05-21T15:22:21Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages7HandlerEEE
^C
Thread 1 "bitcoin-node" received signal SIGINT, Interrupt.
__GI___libc_read (nbytes=1, buf=0x7fffffffd747, fd=3) at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) signal 2
Continuing with signal SIGINT.
2025-05-21T15:22:24Z Shutdown: In progress...
2025-05-21T15:22:24Z addcon thread exit
2025-05-21T15:22:24Z opencon thread exit
2025-05-21T15:22:24Z net thread exit
[Thread 0x7fffa9ffc6c0 (LWP 1160819) exited]
2025-05-21T15:22:24Z msghand thread exit
[Thread 0x7fffaa7fd6c0 (LWP 1160818) exited]
[Thread 0x7fffaaffe6c0 (LWP 1160817) exited]
[Thread 0x7fffa97fb6c0 (LWP 1160820) exited]
2025-05-21T15:22:24Z scheduler thread exit
[Thread 0x7ffff71ff6c0 (LWP 1160797) exited]
2025-05-21T15:22:24Z Writing 0 mempool transactions to file...
2025-05-21T15:22:24Z Writing 0 unbroadcast transactions to file.
2025-05-21T15:22:24Z Dumped mempool: 0.000s to copy, 0.001s to dump, 27 bytes dumped to file
2025-05-21T15:22:24Z Flushed fee estimates to fee_estimates.dat.
2025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-shutoff-1160793} IPC client first request from current thread, constructing waiter
2025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-shutoff-1160793} IPC client send ChainNotifications.chainStateFlushed$Params (context = (thread = <external capability>, callbackThread = <external capability>), role = 0, locator = "\\200\\021\\001\\000\\022H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD\\tGm\\237]YbZDV9\\254\\252f\\205\\001\\323K\\375!\\340\\t\\367\\375\\346\\264W\\207\\000\\2101\\035:\\314\\233\\246\\275\\207\\v\\\\\\\\\\343\\202\\026\\274\\037\\213\\314\\303V\\2116<*+\\352\\255\\026\\204M\\332\\340\\nT\\306\\360Pa\\304\\002\\024\\356\\247\\220\\263\\257l\\230\\350\\362\\027Yl\\300\\316lk\\232\\f\\333\\235,\\234\\336\\204+JE\\263>\\237\\177\\305z\\017\\252\\311@\\341e\\026\\300dk\\305\\212V\\334\\377\\004cb\\217p\\n\\307w\\0046\\2261|\\024\\\\N\\336\\223G\\263~n\\271\\256\\264\\346\\227\\301\\032\\301\\030\\bw\\241Z.\\302\\261\\027 \\005\\374B\\357R\\231\\237Ct\\324\\232I\\322\\370\\303T\\253\\205\\354\\233\\337\\031C\\3215O\\3746\\016\\360b &\\300\\201\\244U\\221\\306 Pql+c\\212\\2524\\314a\\t\\216\\275=\\230b\\207ouv[\\365(\\3...
2025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-shutoff-1160793} IPC client recv ChainNotifications.chainStateFlushed$Results ()
[Thread 0x7fffbeffd6c0 (LWP 1160814) exited]
[Thread 0x7fffbe7fc6c0 (LWP 1160815) exited]
[Thread 0x7fffbf7fe6c0 (LWP 1160813) exited]
[Thread 0x7fffbffff6c0 (LWP 1160812) exited]
[Thread 0x7fffecff96c0 (LWP 1160811) exited]
[Thread 0x7fffed7fa6c0 (LWP 1160810) exited]
[Thread 0x7fffedffb6c0 (LWP 1160809) exited]
[Thread 0x7fffee7fc6c0 (LWP 1160808) exited]
[Thread 0x7fffeeffd6c0 (LWP 1160807) exited]
[Thread 0x7fffef7fe6c0 (LWP 1160806) exited]
[Thread 0x7ffff48d56c0 (LWP 1160805) exited]
[Thread 0x7ffff50d66c0 (LWP 1160804) exited]
[Thread 0x7ffff58d76c0 (LWP 1160803) exited]
[Thread 0x7ffff60d86c0 (LWP 1160802) exited]
2025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
[Thread 0x7ffff68d96c0 (LWP 1160801) exited]
[Thread 0x7fffa8ffa6c0 (LWP 1160824) exited]
2025-05-21T15:22:24Z [ipc] {bitcoin-node-1160793/b-capnp-loop-1160799} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
[New Thread 0x7fffa8ffa6c0 (LWP 1160831)]
2025-05-21T15:22:24Z Shutdown: done

Thread 25 "b-capnp-loop" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffa8ffa6c0 (LWP 1160831)]
___pthread_mutex_lock (mutex=0x5550033b99fb) at ./nptl/pthread_mutex_lock.c:80
80      ./nptl/pthread_mutex_lock.c: No such file or directory.
(gdb) bt
#0  ___pthread_mutex_lock (mutex=0x5550033b99fb) at ./nptl/pthread_mutex_lock.c:80
#1  0x00005555559c35e3 in __gthread_mutex_lock (__mutex=0x5550033b99fb) at /usr/include/x86_64-linux-gnu/c++/12/bits/gthr-default.h:749
#2  std::mutex::lock (this=0x5550033b99fb) at /usr/include/c++/12/bits/std_mutex.h:100
#3  std::unique_lock<std::mutex>::lock (this=<synthetic pointer>) at /usr/include/c++/12/bits/unique_lock.h:139
#4  UniqueLock<AnnotatedMixin<std::mutex> >::Enter (nLine=<optimized out>, pszFile=<optimized out>, pszName=<optimized out>, this=<synthetic pointer>) at ./sync.h:163
#5  UniqueLock<AnnotatedMixin<std::mutex> >::UniqueLock (pszName=<optimized out>, pszFile=<optimized out>, nLine=<optimized out>, fTry=false, mutexIn=..., this=<synthetic pointer>) at ./sync.h:182
#6  ValidationSignalsImpl::Unregister (callbacks=0x7fff94001ee0, this=0x5550033b99fb) at ./validationinterface.cpp:59
#7  ValidationSignals::UnregisterValidationInterface (this=<optimized out>, callbacks=0x7fff94001ee0) at ./validationinterface.cpp:130
#8  0x00005555559c37e4 in ValidationSignals::UnregisterSharedValidationInterface (this=<optimized out>, callbacks=...) at ./validationinterface.cpp:125
#9  0x00005555557e4d10 in node::(anonymous namespace)::NotificationsHandlerImpl::disconnect (this=0x7fff94003a70) at ./node/interfaces.cpp:497
#10 node::(anonymous namespace)::NotificationsHandlerImpl::~NotificationsHandlerImpl (this=<optimized out>, __in_chrg=<optimized out>) at ./node/interfaces.cpp:493
#11 node::(anonymous namespace)::NotificationsHandlerImpl::~NotificationsHandlerImpl (this=0x7fff94003a70, __in_chrg=<optimized out>) at ./node/interfaces.cpp:493
#12 0x0000555555b53ed9 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fff940035e0) at /usr/include/c++/12/bits/shared_ptr_base.h:346
#13 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fff940035e0) at /usr/include/c++/12/bits/shared_ptr_base.h:317
#14 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1071
#15 std::__shared_ptr<interfaces::Handler, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1524
#16 std::__shared_ptr<interfaces::Handler, (__gnu_cxx::_Lock_policy)2>::reset (this=0x7fffe800c6d0) at /usr/include/c++/12/bits/shared_ptr_base.h:1642
#17 mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}::operator()() (__closure=0x7fffe800c6d0) at ./ipc/libmultiprocess/include/mp/proxy-io.h:478
#18 std::__invoke_impl<void, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&>(std::__invoke_other, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&) (__f=...) at /usr/include/c++/12/bits/invoke.h:61
#19 std::__invoke_r<void, mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&>(mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}&) (__fn=...) at /usr/include/c++/12/bits/invoke.h:111
#20 std::_Function_handler<void (), mp::ProxyServerBase<ipc::capnp::messages::Handler, interfaces::Handler>::~ProxyServerBase()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/12/bits/std_function.h:290
#21 0x0000555555f403c2 in std::function<void ()>::operator()() const (this=0x7fffa8ff9d20) at /usr/include/c++/12/bits/std_function.h:591
#22 mp::Unlock<mp::Lock, std::function<void ()> const&>(mp::Lock&, std::function<void ()> const&) (callback=..., lock=...) at ./ipc/libmultiprocess/include/mp/util.h:201
#23 operator() (__closure=<optimized out>) at ./ipc/libmultiprocess/src/mp/proxy.cpp:282
#24 std::__invoke_impl<void, mp::EventLoop::startAsyncThread()::<lambda()> > (__f=...) at /usr/include/c++/12/bits/invoke.h:61
#25 std::__invoke<mp::EventLoop::startAsyncThread()::<lambda()> > (__fn=...) at /usr/include/c++/12/bits/invoke.h:96
#26 std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::<lambda()> > >::_M_invoke<0> (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:252
#27 std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::<lambda()> > >::operator() (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:259
#28 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::EventLoop::startAsyncThread()::<lambda()> > > >::_M_run(void) (this=0x7fffe800cc00) at /usr/include/c++/12/bits/std_thread.h:210
#29 0x00007ffff78d44a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#30 0x00007ffff75c81f5 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#31 0x00007ffff764889c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[ryanofsky]

Thanks! I also just reproduced this and got a similar backtrace with: ryanofsky/bitcoin@7d6a108 (branch)

[ryanofsky]

So commit ryanofsky/bitcoin@c93c985 (branch) fixes the problem for me.

It should be possible to cherry-pick it or just apply the diff https://github.com/ryanofsky/bitcoin/commit/c93c9852d6408a3022e346f457c730c08c175b02.diff to test.

I'm not very satisfied with this fix because it is not really compatible with the way bitcoin-gui works in bitcoin/bitcoin#10102 (details in commit description), but assuming it solves current problems, it is a clean fix.

[darosior]

Can confirm it's fixed. Are we sure it only happens in this specific instance and there should be no other way the order of destruction can lead to a crash? Can i continue testing with this commit applied or should i wait till you've updated your PR?

2025-05-21T17:13:00Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages7HandlerEEE
^C2025-05-21T17:13:02Z Shutdown: In progress...
2025-05-21T17:13:02Z addcon thread exit
2025-05-21T17:13:02Z opencon thread exit
2025-05-21T17:13:02Z net thread exit
2025-05-21T17:13:02Z msghand thread exit
2025-05-21T17:13:02Z scheduler thread exit
2025-05-21T17:13:02Z Writing 0 mempool transactions to file...
2025-05-21T17:13:02Z Writing 0 unbroadcast transactions to file.
2025-05-21T17:13:02Z Dumped mempool: 0.000s to copy, 0.003s to dump, 27 bytes dumped to file
2025-05-21T17:13:02Z Flushed fee estimates to fee_estimates.dat.
2025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-shutoff-1168991} IPC client first request from current thread, constructing waiter
2025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-shutoff-1168991} IPC client send ChainNotifications.chainStateFlushed$Params (context = (thread = <external capability>, callbackThread = <external capability>), role = 0, locator = "\\200\\021\\001\\000\\022H\\3678\\307#\\020|\\020\\017\\340\\2469\\330\\356\\335\\277\\341dOB{\\220\\327\\354[\\361\\036\\233\\301\\035DD\\tGm\\237]YbZDV9\\254\\252f\\205\\001\\323K\\375!\\340\\t\\367\\375\\346\\264W\\207\\000\\2101\\035:\\314\\233\\246\\275\\207\\v\\\\\\\\\\343\\202\\026\\274\\037\\213\\314\\303V\\2116<*+\\352\\255\\026\\204M\\332\\340\\nT\\306\\360Pa\\304\\002\\024\\356\\247\\220\\263\\257l\\230\\350\\362\\027Yl\\300\\316lk\\232\\f\\333\\235,\\234\\336\\204+JE\\263>\\237\\177\\305z\\017\\252\\311@\\341e\\026\\300dk\\305\\212V\\334\\377\\004cb\\217p\\n\\307w\\0046\\2261|\\024\\\\N\\336\\223G\\263~n\\271\\256\\264\\346\\227\\301\\032\\301\\030\\bw\\241Z.\\302\\261\\027 \\005\\374B\\357R\\231\\237Ct\\324\\232I\\322\\370\\303T\\253\\205\\354\\233\\337\\031C\\3215O\\3746\\016\\360b &\\300\\201\\244U\\221\\306 Pql+c\\212\\2524\\314a\\t\\216\\275=\\230b\\207ouv[\\365(\\3...
2025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-shutoff-1168991} IPC client recv ChainNotifications.chainStateFlushed$Results ()
2025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages5ChainEEE
2025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} IPC server destroy N2mp11ProxyServerIN3ipc5capnp8messages4InitEEE
2025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} EventLoop::loop done, cancelling event listeners.
2025-05-21T17:13:02Z [ipc] {bitcoin-node-1168991/b-capnp-loop-1168993} EventLoop::loop bye.
2025-05-21T17:13:02Z Shutdown: done

[ryanofsky]

Are we sure it only happens in this specific instance and there should be no other way the order of destruction can lead to a crash? Can i continue testing with this commit applied or should i wait till you've updated your PR?

This fix should be a pretty general fix because it just waits for all objects associated with any IPC connection to be destroyed before freeing resources they could be referencing. So I think if you continued to test it you would not find similar problems. To be clear, IPC connections using the Chain interface are pretty trusted, so there are probably other things IPC clients could do using it to crash the node. But the fix should address any shutdown issues.

I'm not sure yet if I will update bitcoin/bitcoin#32345 with this specific fix or a more general one that could work better with later changes in bitcoin/bitcoin#10102.

The problem in bitcoin/bitcoin#10102 is that bitcoin-node can be spawned by a parent bitcoin-gui process there. To work correctly, it should be able to keep its connection to the bitcoin-gui process alive while destroying other connections. I think to allow this, Connection objects in libmultiprocess could have a new bool m_controlling field indicating whether they are "controlling" connections that should not be disconnected. Then in addition to the EventLoop::m_num_cients reference count incremented by normal connections, there could be an EventLoop::m_num_cients_controlling reference count incremented by controlling connections. Then, when bitcoin-node is shutting down, it could disconnect all non-controlling connections, wait for m_num_cients == m_num_cients_controlling to be true, free all resources the non-controlling connections could have been referencing, and the finally disconnect the controlling connection. Implementing this should not be too complicated and it should be behave same way as the current fix, while being compatible with bitcoin/bitcoin#10102

[ryanofsky]

Connection objects in libmultiprocess could have a new bool m_controlling field indicating whether they are "controlling" connections that should not be disconnected.

Thinking about it more, a different and more general approach would probably be simpler.

Instead of creating a distinction between controlling and non-controlling connections, and adding way to wait for objects associated with non-controlling connections to be freed after they are disconnected, it would make more sense to just add a general way to wait for objects associated with any connection to be freed after disconnection.

This should be pretty straightforward to implement. Instead of having EventLoop::m_num_clients and EventLoop::m_num_clients_controlling reference counts there could be EventLoop::m_num_clients and Connection::m_num_clients reference counts so objects associated with each connection are tracked separately. This will require another change to make Connection objects outlive all the client and server objects associated with them which is not currently the case. But that should not be too hard to implement.