All network threads stuck in FUTEX_WAIT_PRIVATE

[jmott]

I'm migrating multiple instances on multiple servers.
Small depots work fine. But on larger depots it doesn't take long before all the network threads are in FUTEX_WAIT_PRIVATE with CPU at 0%. I gave up waiting after 16 hours in that state.

Here's a sample command line:
build/p4-fusion/p4-fusion --path //3rdParty/... --user "$P4USER" --port "$P4PORT" --client "$P4CLIENT" --src clones/3rdParty/.git --networkThreads 10 --printBatch 100 --lookAhead 2000 --retries 10

For a sample depot:
=== Depot: 3rdParty === - Total: 160.4G
//3rdParty/... 891863 files 160.4G bytes

Any suggestions?

Thanks!

[jmott]

$ gdb -p 3040257
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-20.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 3040257
[New LWP 3040268]
[New LWP 3040269]
[New LWP 3040270]
[New LWP 3040271]
[New LWP 3040272]
[New LWP 3040273]
[New LWP 3040274]
[New LWP 3040275]
[New LWP 3040276]
[New LWP 3040277]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-251.el8_10.25.x86_64 libgcc-8.5.0-28.el8_10.x86_64 libstdc++-8.5.0-28.el8_10.x86_64 openssl-libs-1.1.1k-14.el8_6.x86_64 zlib-1.2.11-25.el8.x86_64
(gdb) thread apply all bt

Thread 11 (Thread 0x7fb59aa5f700 (LWP 3040277)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256,
__lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10b96fe8)
at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...)
at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10b96fe8) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10b96fe8) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10b96fe0) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fb59b260700 (LWP 3040276)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256,
__lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
--Type for more, q to quit, c to continue without paging--c
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10b96df8) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10b96df8) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10b96df8) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10b96df0) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fb59ba61700 (LWP 3040275)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256, __lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10b96a88) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10b96a88) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10b96a88) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10b96a80) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fb59c262700 (LWP 3040274)):
#0 0x00007fb5a09ce4cd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fb5a09c7ac9 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x0000000000465b38 in __gthread_mutex_lock (__mutex=0x7fb5687c6910) at /bnr2/lude/soft/gcc-8.1.0/run/default/x86_64_pc_linux_centos66/gcc/8.1.0/include/c++/8.1.0/x86_64-pc-linux/bits/gthr-default.h:748
#3 std::mutex::lock (this=) at /bnr2/lude/soft/gcc-8.1.0/run/default/x86_64_pc_linux_centos66/gcc/8.1.0/include/c++/8.1.0/bits/std_mutex.h:103
#4 std::lock_guardstd::mutex::lock_guard (__m=..., this=) at /bnr2/lude/soft/gcc-8.1.0/run/default/x86_64_pc_linux_centos66/gcc/8.1.0/include/c++/8.1.0/bits/std_mutex.h:162
#5 ClientUser::SetVarList (this=0x7fb59c261b60, l=0x10713720) at ../p4/client/clientuser.cc:1153
#6 0x0000000000468579 in Client::RunTag (this=0x10713720, func=0x61c8c9 "print", u=0x7fb59c261b60) at ../p4/client/client.cc:305
#7 0x00000000004689ec in Client::Run (this=0x10713720, func=, u=0x7fb59c261b60) at ../p4/client/client.cc:261
#8 0x0000000000444c7e in P4API::RunEx (this=0x1004ba70, command=0x61c8c9 "print", stringArguments=std::vector of length 100, capacity 128 = {...}, commandRetries=10) at /home/scmteam/p4-fusion/p4-fusion/p4_api.h:135
#9 0x000000000043eab5 in P4API::Run (this=0x1004ba70, command=0x61c8c9 "print", stringArguments=std::vector of length 100, capacity 128 = {...}) at /home/scmteam/p4-fusion/p4-fusion/p4_api.h:178
#10 0x000000000043d776 in P4API::PrintFiles (this=0x1004ba70, fileRevisions=std::vector of length 100, capacity 128 = {...}) at /home/scmteam/p4-fusion/p4-fusion/p4_api.cc:321
#11 0x000000000041b915 in ChangeList::<lambda(P4API*)>::operator()(P4API ) const (__closure=0x7fb5685bb8f0, p4=0x1004ba70) at /home/scmteam/p4-fusion/p4-fusion/commands/change_list.cc:121
#12 0x000000000041c22d in std::_Function_handler<void(P4API), ChangeList::Flush(std::shared_ptr<std::vector<std::__cxx11::basic_string > >, std::shared_ptr<std::vector<FileData*> >)::<lambda(P4API*)> >::_M_invoke(const std::_Any_data &, P4API &&) (__functor=..., __args#0=@0x7fb59c261c60: 0x1004ba70) at /usr/include/c++/8/bits/std_function.h:297
#13 0x0000000000449d83 in std::function<void (P4API)>::operator()(P4API*) const (this=0x7fb59c261ca0, __args#0=0x1004ba70) at /usr/include/c++/8/bits/std_function.h:687
#14 0x0000000000448b17 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10b96928) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:127
#15 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#16 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#17 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10b96928) at /usr/include/c++/8/thread:244
#18 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10b96928) at /usr/include/c++/8/thread:253
#19 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10b96920) at /usr/include/c++/8/thread:196
#20 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#21 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#22 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fb59ca63700 (LWP 3040273)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256, __lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10b967c8) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10b967c8) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10b967c8) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10b967c0) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fb59d264700 (LWP 3040272)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256, __lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10cc36f8) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10cc36f8) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10cc36f8) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10cc36f0) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fb59da65700 (LWP 3040271)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256, __lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10cc2e58) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10cc2e58) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10cc2e58) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10cc2e50) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fb59e266700 (LWP 3040270)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256, __lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10cc3628) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10cc3628) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10cc3628) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10cc3620) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fb59ea67700 (LWP 3040269)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256, __lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10cc3548) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10cc3548) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10cc3548) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10cc3540) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fb59f268700 (LWP 3040268)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000448ec6 in std::condition_variable::wait<ThreadPool::Initialize(int)::<lambda()>::<lambda()> >(std::unique_lockstd::mutex &, ThreadPool::<lambda()>::<lambda()>) (this=0x9bc040 ThreadPool::GetSingleton()::singleton+256, __lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x0000000000448a92 in ThreadPool::<lambda()>::operator()(void) const (__closure=0x10cc3528) at /home/scmteam/p4-fusion/p4-fusion/thread_pool.cc:113
#4 0x00000000004490ae in std::__invoke_impl<void, ThreadPool::Initialize(int)::<lambda()> >(std::__invoke_other, ThreadPool::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#5 0x0000000000448eeb in std::__invoke<ThreadPool::Initialize(int)::<lambda()> >(ThreadPool::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#6 0x00000000004492de in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x10cc3528) at /usr/include/c++/8/thread:244
#7 0x00000000004492b4 in std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > >::operator()(void) (this=0x10cc3528) at /usr/include/c++/8/thread:253
#8 0x0000000000449298 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ThreadPool::Initialize(int)::<lambda()> > > >::_M_run(void) (this=0x10cc3520) at /usr/include/c++/8/thread:196
#9 0x00007fb5a06eab23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#10 0x00007fb5a09c51ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007fb59fcf08d3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fb5a1784840 (LWP 3040257)):
#0 0x00007fb5a09cb371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fb5a06e4870 in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x000000000041bf54 in std::condition_variable::wait<ChangeList::WaitForDownload()::<lambda()> >(std::unique_lockstd::mutex &, ChangeList::<lambda()>) (this=0x10120a30, __lock=..., __p=...) at /usr/include/c++/8/condition_variable:99
#3 0x000000000041bc80 in ChangeList::WaitForDownload (this=0x7fb57d4ab128) at /home/scmteam/p4-fusion/p4-fusion/commands/change_list.cc:142
#4 0x0000000000433f30 in Main (argc=19, argv=0x7ffff94f45f8) at /home/scmteam/p4-fusion/p4-fusion/main.cc:374
#5 0x00000000004359c0 in main (argc=19, argv=0x7ffff94f45f8) at /home/scmteam/p4-fusion/p4-fusion/main.cc:503
(gdb)
(gdb) info symbol 0x9bc068
ThreadPool::GetSingleton()::singleton + 296 in section .bss of /home/scmteam/p4-fusion/build/p4-fusion/p4-fusion

[twaritwaikar]

This is a new bug I have been experiencing as well. My only idea here is to try to downgrade the P4API version that you are using until this problem goes away. Some known good release numbers are listed in the README for the Helix Core C++ API

[twaritwaikar]

Here - https://github.com/salesforce/p4-fusion#build

[jmott]

Thanks @twaritwaikar I'll try that and report back.

[jmott]

Unfortunately downgrading from 25.1 to 24.1 (which is on the list of verified builds) didn't help.

[sstepashka]

I think, I was able to find out a possible issues:

ClientUser::~ClientUser()
{
	delete transfer;
	delete ssoHandler;
# ifdef HAS_CPP11
	delete (std::mutex *)setterGuard;
# endif
}

All types which are coming via P4API::RunEx are inherited from the ClientUser and the retry logic in P4API::RunEx leads to the copy of the raw pointer and hence double-delete of the mutex. I can guess, that the double free hangs because it's UB? I tried locally converting the code to return std::unique_ptr<T> and hence fixed the rest of the usages and it seems to solve the issue.

It might be hard to reproduce if we never trigger retry logic.

[sstepashka]

In more details about the retry logic:

        T clientUser; // creates a default instance of, let's say `TestResult`
...
	while (m_ClientAPI.Dropped() || clientUser->GetError().IsError())
	{
		if (retries == 0)
		{
			break;
		}
                 ...
                 clientUser = T(); // creates a new instance and assign by copy

during the copy the temporary olbject still holds pointer to the setterGuard mutex and deletes it. But, later, the new instance keep the pointer to the already deleted mutex.

It seems that the ClientUser subclasses must be never copied or moved because they would keep the dangling pointer to the mutex and would likely double-free it.

Locally, I tried to convert every instance of ClientUser into unique_ptr<T> and fix all the API usage, and it seems to resolve the issue.

[twaritwaikar]

@sstepashka Amazing stuff! Would it be possible for you to open a PR?

[sstepashka]

@twaritwaikar Yep, I am on it!

[sstepashka]

@twaritwaikar PTAL: #95

[sstepashka]

@twaritwaikar PTAL?