[CameraServer] segfault during static destruction #7685
Open
Description
Describe the bug
See PhotonVision/photonvision#1718 - during gtest teardown, we randomly see segfaults.
**Additional context**
Running a photonlibTest from https://github.com/PhotonVision/photonvision/pull/1714 :
Running a photonlibTest from https://github.com/PhotonVision/photonvision/pull/1714 :
Failure 1:
[==========] 52 tests from 7 test suites ran. (1505 ms total)
[ PASSED ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==77094==ERROR: AddressSanitizer: SEGV on unknown address 0x000069800042 (pc 0x155550999d79 bp 0x1555499ff5a0 sp 0x1555499fed18 T9)
==77094==The signal is caused by a READ memory access.
#0 0x155550999d79 (/lib/x86_64-linux-gnu/libc.so.6+0x199d79)
#1 0x155554a942e9 in MemcmpInterceptorCommon(void*, int (*)(void const*, void const*, unsigned long), void const*, void const*, unsigned long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:881
#2 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:892
#3 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:887
#4 0x15555549ac00 in std::char_traits<char>::compare(char const*, char const*, unsigned long) /usr/include/c++/11/bits/char_traits.h:389
#5 0x15555549ac00 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const /usr/include/c++/11/bits/basic_string.tcc:1444
#6 0x15555549ac00 in bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*) /usr/include/c++/11/bits/basic_string.h:6250
#7 0x15555549ac00 in GetSinkStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:163
#8 0x15555549ac00 in UpdateStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:228
#9 0x15555549ac00 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:464
#10 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
#11 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
#12 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
#13 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
#14 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
#15 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
#16 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
#17 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
#18 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
#19 0x1555514dc252 (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
#20 0x155550894ac2 in start_thread nptl/pthread_create.c:442
#21 0x15555092684f (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x199d79)
Thread T9 created by T0 here:
#0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)
==77094==ABORTING
Failure 2:
[==========] 52 tests from 7 test suites ran. (1515 ms total)
[ PASSED ] 52 tests.
terminate called after throwing an instance of 'std::bad_function_call'
what(): bad_function_call
./run-photonlibpy-to-crash.sh: line 31: 97288 Aborted "$PROGRAM"
(.venv) matt@photonvision:~/Documents/GitHub/photonvision$
Failure 3:
[==========] 52 tests from 7 test suites ran. (1529 ms total)
[ PASSED ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==74081==ERROR: AddressSanitizer: SEGV on unknown address 0x00006980003b (pc 0x155554a2cc37 bp 0x00006980004b sp 0x1555499fed20 T9)
==74081==The signal is caused by a WRITE memory access.
#0 0x155554a2cc37 in bool __sanitizer::atomic_compare_exchange_strong<__sanitizer::atomic_uint8_t>(__sanitizer::atomic_uint8_t volatile*, __sanitizer::atomic_uint8_t::Type*, __sanitizer::atomic_uint8_t::Type, __sanitizer::memory_order) ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h:80
#1 0x155554a2cc37 in __asan::Allocator::AtomicallySetQuarantineFlagIfAllocated(__asan::AsanChunk*, void*, __sanitizer::BufferedStackTrace*) ../../../../src/libsanitizer/asan/asan_allocator.cpp:621
#2 0x155554a2cc37 in __asan::Allocator::Deallocate(void*, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) ../../../../src/libsanitizer/asan/asan_allocator.cpp:697
#3 0x155554ab720c in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:172
#4 0x155555498f70 in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /usr/include/c++/11/ext/new_allocator.h:145
#5 0x155555498f70 in std::allocator<char>::deallocate(char*, unsigned long) /usr/include/c++/11/bits/allocator.h:199
#6 0x155555498f70 in std::allocator_traits<std::allocator<char> >::deallocate(std::allocator<char>&, char*, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:496
#7 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy(unsigned long) /usr/include/c++/11/bits/basic_string.h:245
#8 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() /usr/include/c++/11/bits/basic_string.h:240
#9 0x155555498f70 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() /usr/include/c++/11/bits/basic_string.h:672
#10 0x155555498f70 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:88
#11 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:149
#12 0x155555498f70 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:163
#13 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/c++/11/bits/stl_construct.h:196
#14 0x155555498f70 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&) /usr/include/c++/11/bits/alloc_traits.h:848
#15 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~vector() /usr/include/c++/11/bits/stl_vector.h:680
#16 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_move_assign(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, std::integral_constant<bool, true>) /usr/include/c++/11/bits/stl_vector.h:1821
#17 0x155555498f70 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::operator=(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&) /usr/include/c++/11/bits/stl_vector.h:714
#18 0x155555498f70 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:463
#19 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
#20 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
#21 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
#22 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
#23 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
#24 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
#25 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
#26 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
#27 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
#28 0x1555514dc252 (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
#29 0x155550894ac2 in start_thread nptl/pthread_create.c:442
#30 0x15555092684f (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h:80 in bool __sanitizer::atomic_compare_exchange_strong<__sanitizer::atomic_uint8_t>(__sanitizer::atomic_uint8_t volatile*, __sanitizer::atomic_uint8_t::Type*, __sanitizer::atomic_uint8_t::Type, __sanitizer::memory_order)
Thread T9 created by T0 here:
#0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)
==74081==ABORTING
Failure 4:
[ PASSED ] 52 tests.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==89459==ERROR: AddressSanitizer: SEGV on unknown address 0x000069800050 (pc 0x155550999d79 bp 0x1555499ff5a0 sp 0x1555499fed18 T9)
==89459==The signal is caused by a READ memory access.
#0 0x155550999d79 (/lib/x86_64-linux-gnu/libc.so.6+0x199d79)
#1 0x155554a942e9 in MemcmpInterceptorCommon(void*, int (*)(void const*, void const*, unsigned long), void const*, void const*, unsigned long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:881
#2 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:892
#3 0x155554a94bc6 in __interceptor_memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:887
#4 0x15555549ac00 in std::char_traits<char>::compare(char const*, char const*, unsigned long) /usr/include/c++/11/bits/char_traits.h:389
#5 0x15555549ac00 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const /usr/include/c++/11/bits/basic_string.tcc:1444
#6 0x15555549ac00 in bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*) /usr/include/c++/11/bits/basic_string.h:6250
#7 0x15555549ac00 in GetSinkStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:163
#8 0x15555549ac00 in UpdateStreamValues /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:228
#9 0x15555549ac00 in operator() /work/cameraserver/src/main/native/cpp/cameraserver/CameraServer.cpp:464
#10 0x155554591e9e in std::function<void (cs::RawEvent const&)>::operator()(cs::RawEvent const&) const /usr/include/c++/11/bits/std_function.h:590
#11 0x155554591e9e in cs::impl::NotifierThread::DoCallback(std::function<void (cs::RawEvent const&)>, cs::RawEvent const&) /work/cscore/src/main/native/cpp/Notifier.h:51
#12 0x155554591e9e in wpi::CallbackThread<cs::impl::NotifierThread, cs::RawEvent, cs::impl::ListenerData, cs::RawEvent>::Main() /work/wpiutil/src/main/native/include/wpi/CallbackManager.h:165
#13 0x1555537578da in operator() /work/wpiutil/src/main/native/cpp/SafeThread.cpp:81
#14 0x1555537578da in __invoke_impl<void, wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:61
#15 0x1555537578da in __invoke<wpi::detail::SafeThreadOwnerBase::Start(std::shared_ptr<wpi::SafeThreadBase>)::<lambda()> > /usr/include/c++/11/bits/invoke.h:96
#16 0x1555537578da in _M_invoke<0> /usr/include/c++/11/bits/std_thread.h:259
#17 0x1555537578da in operator() /usr/include/c++/11/bits/std_thread.h:266
#18 0x1555537578da in _M_run /usr/include/c++/11/bits/std_thread.h:211
#19 0x1555514dc252 (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
#20 0x155550894ac2 in start_thread nptl/pthread_create.c:442
#21 0x15555092684f (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x199d79)
Thread T9 created by T0 here:
#0 0x155554a58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x1555514dc328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)
==89459==ABORTING
(.venv) matt@photonvision:~/Documents/GitHub/photonvision$
Based on the above asan logs, here, loos suspiscious to me. We're iterating over m_addresses from a function called by NotifierThread.
The cameraserver::instance could only come from this magic static here. The callback comes from m_videoListener here. The callback does capture this. Not sure if this setup could produce a race at destruction, since it seems like the VideoListener destructor deregisters the callback.
To Reproduce
Steps to reproduce the behavior:
- Checkout PhotonVision from 2025/01/12
- Run photon-lib unit tests repeatedly (for me, dozens of times per failure)
- Given sufficient runs, one will segfault eventually (implying this is a race)?