~message dead lock

[AndreyBarmaley]

I don’t know for sure how it succeeds, but I caught the lock here:

#0  0x00007f83fd3f7c10 in __lll_lock_wait () from /lib64/libc.so.6
#1  0x00007f83fd3fe21d in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6
#2  0x00007f83fd8e3273 in sdbus::internal::SdBus::sd_bus_message_unref(sd_bus_message*) () from /lib64/libsdbus-c++.so.1
#3  0x00007f83fd8d6faa in sdbus::Message::~Message() () from /lib64/libsdbus-c++.so.1
#4  0x000000000042eef2 in sdbus::MethodCall::~MethodCall (this=0x7f83fafc18e0, __in_chrg=<optimized out>) at /usr/include/sdbus-c++/Message.h:170
#5  0x00000000004758ac in std::__invoke_impl<void, sdbus::MethodRegistrator::implementedAs<LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}>(LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}&&)::{lambda(sdbus::MethodCall)#1}&, sdbus::MethodCall>(std::__invoke_other, sdbus::MethodRegistrator::implementedAs<LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}>(LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}&&)::{lambda(sdbus::MethodCall)#1}&, sdbus::MethodCall&&) (__f=...)
    at /usr/include/c++/12/bits/invoke.h:61
#6  0x000000000046e862 in std::__invoke_r<void, sdbus::MethodRegistrator::implementedAs<LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}>(LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}&&)::{lambda(sdbus::MethodCall)#1}&, sdbus::MethodCall>(sdbus::MethodRegistrator::implementedAs<LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}>(LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}&&)::{lambda(sdbus::MethodCall)#1}&, sdbus::MethodCall&&) (__fn=...) at /usr/include/c++/12/bits/invoke.h:111
#7  0x0000000000468185 in std::_Function_handler<void (sdbus::MethodCall), sdbus::MethodRegistrator::implementedAs<LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}>(LTSM::Manager::Service_adaptor::Service_adaptor(sdbus::IObject&)::{lambda(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#7}&&)::{lambda(sdbus::MethodCall)#1}>::_M_invoke(std::_Any_data const&, sdbus::MethodCall&&) (__functor=..., __args#0=...) at /usr/include/c++/12/bits/std_function.h:290
#8  0x00007f83fd8db700 in sdbus::internal::Object::sdbus_method_callback(sd_bus_message*, void*, sd_bus_error*) () from /lib64/libsdbus-c++.so.1
#9  0x00007f83fd2c2b3c in object_find_and_run.lto_priv () from /lib64/libsystemd.so.0
#10 0x00007f83fd2d17df in bus_process_internal () from /lib64/libsystemd.so.0
#11 0x00007f83fd8e33bb in sdbus::internal::SdBus::sd_bus_process(sd_bus*, sd_bus_message**) () from /lib64/libsdbus-c++.so.1
#12 0x00007f83fd8d3bb2 in sdbus::internal::Connection::processPendingRequest() () from /lib64/libsdbus-c++.so.1
#13 0x00007f83fd8d4ba8 in sdbus::internal::Connection::enterEventLoop() () from /lib64/libsdbus-c++.so.1

sdbus-cpp version: 1.2.0, systemd-250.8-1.fc36.x86_64

[AndreyBarmaley]

Hi. I'm sure, I got this after the fork system call.

[sangelovic]

I cannot imagine any way this could happen normally. The fact that #11 0x00007f83fd8e33bb in sdbus::internal::SdBus::sd_bus_process succeeds means the (recursive) mutex has been locked. And then is re-locked at #2 0x00007f83fd8e3273 in sdbus::internal::SdBus::sd_bus_message_unref. The only explanation I have for this deadlock is what you've written -- using forks unsafely. forks have specific behavior regarding threads. Citing from fork specification:

A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called.

Are you abiding to this specification? From what I see it looks like you're accessing in a child process the mutex that has been locked before, in the parent process.

[sangelovic]

See also https://elias.rhi.hi.is/libc/Threads-and-Fork.html or http://www.doublersolutions.com/docs/dce/osfdocs/htmls/develop/appdev/Appde193.htm