Segfault in TestDQMOnlineClient-visualization

[makortel]

The test TestDQMOnlineClient-visualization crashed in CMSSW_13_2_NONLTO_X_2023-06-24-1100 on el8_amd64_gcc11 with

Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 4 at 24-Jun-2023 18:59:42.015 CEST

Thread 9 (Thread 0x14b8941fe700 (LWP 1590063) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f472e2c4 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8941f7cb0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8941f7cb0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::address_context>::wait (this=0x14b8941f7c80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::commit_wait (node=..., this=0x14b8f4761460 <_ZN3tbb6detail2r1L20address_waiter_tableE.lto_priv.0+52320>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::address_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::address_context>, tbb::detail::d1::delegate_base&> (node=..., pred=..., this=0x14b8f4761460 <_ZN3tbb6detail2r1L20address_waiter_tableE.lto_priv.0+52320>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::wait_on_address (address=<optimized out>, predicate=..., context=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/address_waiter.cpp:71
#12 0x000014b8f473e95c in tbb::detail::d1::waitable_atomic<int>::wait_until (order=<optimized out>, context=787, expected=<optimized out>, this=0x14b8f1dbb098) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_waitable_atomic.h:79
#13 tbb::detail::r1::market::adjust_demand (this=0x14b8f1dcb580, a=..., delta=-7, mandatory=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:586
#14 0x000014b8f4740934 in tbb::detail::r1::arena::is_out_of_work() [clone .isra.0] (this=0x14b8f1dbaf00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:333
#15 0x000014b8f4730bad in tbb::detail::r1::waiter_base::pause (this=0x14b8941f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:36
#16 tbb::detail::r1::outermost_worker_waiter::pause (this=0x14b8941f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:69
#17 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=0x14b8f1dbbc00, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#18 0x000014b8f473cac0 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#20 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#21 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#22 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#24 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#25 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14b8965fe700 (LWP 1590062) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff12b in sched_yield () from /lib64/libc.so.6
#5  0x000014b8f4730b8a in __gthread_yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/bits/std_thread.h:329
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x14b8965f7f38) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x14b8965f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::outermost_worker_waiter::pause (this=0x14b8965f7f30) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/waiters.h:69
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=0x14b8f1dbbc80, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x000014b8f473cac0 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbc80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#18 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14b894bff700 (LWP 1590061) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24de20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14b895bfd700 (LWP 1590060) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24df20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14b8971ff700 (LWP 1590059) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24dea0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14b89839d700 (LWP 1590057) "cmsRun"):
#0  0x000014b8f35bf868 in nanosleep () from /lib64/libc.so.6
#1  0x000014b8f35bf76e in sleep () from /lib64/libc.so.6
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff9bd in syscall () from /lib64/libc.so.6
#5  0x000014b8f473ee9e in tbb::detail::r1::futex_wait (comparand=2, futex=0x14b8ec24e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14b8ec24e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14b8ec24e120) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#11 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14b898d9e700 (LWP 1590056) "cmsRun"):
#3  0x000014b8eb4b901b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x000014b8f5f9b01a in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#7  0x000014b8f473c96f in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x14b87f42bc00, waiter=..., this=0x14b8f1dbbb80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14b8f1dbbb80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x000014b8f473ebb5 in tbb::detail::r1::rml::private_worker::run (this=0x14b8ec24e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14b8ec24e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000014b8f38931ca in start_thread () from /lib64/libpthread.so.0
#14 0x000014b8f34ffe73 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14b8f2a23640 (LWP 1589014) "cmsRun"):
#2  0x000014b8eb4b5590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014b8f34ff12b in sched_yield () from /lib64/libc.so.6
#5  0x000014b8f473e917 in __gthread_yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc11/external/gcc/11.4.1-30ebdc301ebd200f2ae0e3d880258e65/include/c++/11.4.1/bits/std_thread.h:329
#7  tbb::detail::d0::timed_spin_wait_until<tbb::detail::d1::waitable_atomic<int>::wait_until(int, unsigned long, std::memory_order)::{lambda()#1}>(tbb::detail::d1::waitable_atomic<int>::wait_until(int, unsigned long, std::memory_order)::{lambda()#1}) (condition=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:129
#8  tbb::detail::d1::waitable_atomic<int>::wait_until (order=<optimized out>, context=788, expected=<optimized out>, this=0x14b8f1dbb098) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_waitable_atomic.h:74
#9  tbb::detail::r1::market::adjust_demand (this=0x14b8f1dcb580, a=..., delta=7, mandatory=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/market.cpp:586
#10 0x000014b8f474273b in tbb::detail::r1::arena::advertise_new_work<(tbb::detail::r1::arena::new_work_type)0> (this=0x14b8f1dbaf00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.h:547
#11 0x000014b8f5fcd8c8 in edm::EventProcessor::endUnfinishedLumi(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#12 0x000014b8f5fcd987 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#13 0x000014b8f5fd26bd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02790/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_NONLTO_X_2023-06-24-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so
#14 0x000000000040a1bd in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014b8f4731877 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.9.0-b1d97183c007769a478c02aa8504b8cb/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x000000000040b009 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040971c in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_13_2_NONLTO_X_2023-06-24-1100/src/DQM/Integration/test/runtest.sh: line 19: 1589014 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization had ERRORS

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_2_NONLTO_X_2023-06-24-1100/unitTestLogs/DQM/Integration#/

[makortel]

assign core

FYI @cms-sw/dqm-l2

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

We updated TBB to 2021.9.0 in CMSSW_13_2_X_2023-06-20-2300 (with cms-sw/cmsdist#8553), it may be related.

[makortel]

Another one in CMSSW_13_3_X_2023-09-05-1100 on slc7_amd64_gcc11

Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 4 at 05-Sep-2023 14:24:24.917 CEST

Thread 10 (Thread 0x14d71e3fc700 (LWP 1862305) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115df24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115df20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115df00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14d71edfd700 (LWP 1862304) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115de24) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115de20) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115de00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x14d71f9fe700 (LWP 1862303) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115dea4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115dea0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115de80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x14d7203ff700 (LWP 1862302) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e024) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e024) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e020) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14d7215ff700 (LWP 1862301) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115dfa4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115dfa4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115dfa0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115df80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14d7225f6700 (LWP 1862300) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e124) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e120) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14d722ff7700 (LWP 1862299) "cmsRun"):
#2  0x000014d7802d0b10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d788646e29 in syscall () from /lib64/libc.so.6
#5  0x000014d789759eae in tbb::detail::r1::futex_wait (comparand=2, futex=0x14d78115e0a4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14d78115e0a4) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14d78115e0a0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14d78115e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d78115e080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14d76065a700 (LWP 1862126) "cmsRun"):
#0  0x000014d78892b1d9 in waitpid () from /lib64/libpthread.so.0
#1  0x000014d7802d0e87 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000014d7802d381a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000014d788f28b04 in std::execute_native_thread_routine (__p=0x14d76d992530) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014d788923ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x000014d78864cb0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14d78678c780 (LWP 1861056) "cmsRun"):
#3  0x000014d7802d434b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x000014d78aed69d9 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#7  0x000014d78975fa7d in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x14d6e1939d00, this=0x14d785bbb900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x14d785bbb900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#10 0x000014d78aeea42d in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#11 0x000014d78aef6153 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#12 0x000014d78aef6213 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#13 0x000014d78aef83fd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02801/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_3_X_2023-09-03-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so
#14 0x000000000040c17d in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014d78974c887 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.9.0-5caec19d753f5e33a53af823382dc5ea/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x00000000004101e0 in main::{lambda()#1}::operator()() const ()
#17 0x0000000000407d7c in main ()

Current Modules:
Module: none

---> test TestDQMOnlineClient-visualization_secondInstance had ERRORS

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc11/CMSSW_13_3_X_2023-09-05-1100/unitTestLogs/DQM/Integration#/

[aandvalenzuela]

Another one in CMSSW_14_1_X_2024-02-21-2300 on el8_amd64_gcc12:

Thread 3 (Thread 0x148615bcb700 (LWP 2694561) "cmsRun"):
#0  0x00001486631d3918 in nanosleep () from /lib64/libc.so.6
#1  0x00001486631d381e in sleep () from /lib64/libc.so.6
#2  0x000014865cd15cc0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001486631139bd in syscall () from /lib64/libc.so.6
#5  0x000014866436adf2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x14865e26d0a4) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x14865e26d0a4) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x14865e26d0a0) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x14865e26d080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14865e26d080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001486634a71ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000148663113e73 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x14863a3ff700 (LWP 2694273) "cmsRun"):
#0  0x00001486634b1672 in waitpid () from /lib64/libpthread.so.0
#1  0x000014865cd15e17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014865cd18cea in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000148663b3ba73 in std::execute_native_thread_routine (__p=0x14863d4504e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00001486634a71ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148663113e73 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x148662631640 (LWP 2692422) "cmsRun"):
#0  0x00001486631fe301 in poll () from /lib64/libc.so.6
#1  0x000014865cd646af in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014865cd18dbc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014865cd19720 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#6  0x0000148665b20928 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000148664371281 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148661b2b900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148661b2b900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#10 0x0000148665b30cdb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x0000148665b40428 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x0000148665b34f63 in edm::(anonymous namespace)::RunResources::~RunResources() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x0000148665b3aaf6 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02825/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-02-21-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x00000000004074ef in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x000014866435d9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2_SKYLAKEAVX512-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2_SKYLAKEAVX512-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5109e373398eaf79a8268d6a86a2f6e2/tbb-v2021.9.0/src/tbb/arena.cpp:688
#16 0x0000000000408ed2 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040517c in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_X_2024-02-21-2300/src/DQM/Integration/test/runtest.sh: line 28: 2692422 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization had ERRORS

[aandvalenzuela]

Hello, new segfault occurrence in CMSSW_14_1_X_2024-07-17-2300:

%MSG
Begin processing the 98th record. Run 355380, Event 26705277, LumiSection 19 on stream 4 at 18-Jul-2024 02:49:28.143 CEST
Begin processing the 99th record. Run 355380, Event 26732456, LumiSection 19 on stream 1 at 18-Jul-2024 02:49:28.672 CEST
Begin processing the 100th record. Run 355380, Event 26785259, LumiSection 19 on stream 0 at 18-Jul-2024 02:49:29.123 CEST


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thu Jul 18 02:49:36 CEST 2024
Thread 26 (Thread 0x1506d83ff700 (LWP 1304338) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x1506d8dff700 (LWP 1304337) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x1506d92de700 (LWP 1304336) "cmsRun"):
#0  0x000015075f66c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001507264dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00001507264e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00001507264dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000150715878472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#6  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x150705bf1700 (LWP 1304251) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x150705df2700 (LWP 1304250) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x150705ff3700 (LWP 1304249) "cmsRun"):
#0  0x000015075f66edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x000015075f66eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x000015075779d706 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x000015075779d7b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x1507061f4700 (LWP 1304248) "cmsRun"):
#0  0x000015075f6701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000150759019f08 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x000015075771203c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000150757712179 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x1507063f5700 (LWP 1304247) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x1507065f6700 (LWP 1304246) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x1507067f7700 (LWP 1304245) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x1507069f8700 (LWP 1304244) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x150706bf9700 (LWP 1304243) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x150706dfa700 (LWP 1304242) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x150706ffb700 (LWP 1304241) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x1507071fc700 (LWP 1304240) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x1507073fd700 (LWP 1304239) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x1507075fe700 (LWP 1304238) "cmsRun"):
#0  0x000015075f3c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000150759014282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00001507590101ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000150759019617 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-07-17-2300/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x1507081ff700 (LWP 1304237) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4e24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4e24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4e20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x1507095fd700 (LWP 1304236) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4f24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4f24) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4f20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x150709ffe700 (LWP 1304235) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d5024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d5024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d5020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x15070a9ff700 (LWP 1304234) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4ea4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4ea4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4ea0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x15070b76b700 (LWP 1304233) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c141d in syscall () from /lib64/libc.so.6
#5  0x00001507613d3fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x15075a0d4fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#11 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x15070c890700 (LWP 1304232) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000015075f2c140d in syscall () from /lib64/libc.so.6
#5  0x00001507613d346b in tbb::detail::r1::futex_wakeup_one (futex=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:109
#6  tbb::detail::r1::binary_semaphore::V (this=0x15075a0d4fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/semaphore.h:262
#7  tbb::detail::r1::rml::internal::thread_monitor::notify (this=0x15075a0d4fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:230
#8  tbb::detail::r1::rml::private_worker::wake_or_launch (this=0x15075a0d4f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:292
#9  tbb::detail::r1::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:412
#10 0x00001507613d3e49 in tbb::detail::r1::rml::private_server::propagate_chain_reaction (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:159
#11 tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:275
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#14 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x15070d291700 (LWP 1304231) "cmsRun"):
#0  0x000015075f390098 in nanosleep () from /lib64/libc.so.6
#1  0x000015075f38ff9e in sleep () from /lib64/libc.so.6
#2  0x00001507599dfa10 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000150762032377 in edm::Worker::reset() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000150762032dc4 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000150761f5d10d in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000150761f6dedd in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x00001507613d1b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x150708769100, waiter=..., this=0x15075ddbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#9  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x15075ddbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#10 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#11 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/market.cpp:599
#12 0x00001507613d3cee in tbb::detail::r1::rml::private_worker::run (this=0x15075a0d5080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x15075a0d5080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#14 0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#15 0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x1507358cd700 (LWP 1304112) "cmsRun"):
#0  0x000015075f6706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x00001507599dfb67 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001507599e299a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000015075fcd8a73 in std::execute_native_thread_routine (__p=0x150738c0cdb0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000015075f6661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000015075f2c18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1507612e1680 (LWP 1303317) "cmsRun"):
#0  0x000015075f3baac1 in poll () from /lib64/libc.so.6
#1  0x00001507599e2bb7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001507599e2db4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000150761f2d438 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x00001507613da3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x15075ddbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x15075ddbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x0000150761f433ab in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x0000150761f502f6 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x0000150761f5164d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02846/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-07-17-1200/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x0000000000408410 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x00001507613c69ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-df2a8868ba5d9111864e1a63085d7a4b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a102 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_X_2024-07-17-2300/src/DQM/Integration/test/runtest.sh: line 28: 1303317 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

---> test TestDQMOnlineClient-visualization_secondInstance had ERRORS

[iarspider]

New occurance: CMSSW_14_1_X_2024-08-12-1100

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon Aug 12 18:43:19 CEST 2024
Thread 26 (Thread 0x148ea05ff700 (LWP 3757517) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x148ea0fff700 (LWP 3757516) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x148ea1525700 (LWP 3757515) "cmsRun"):
#0  0x0000148f2766c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000148eee2dfba6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x0000148eee2e01a5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x0000148eee2dd968 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x0000148edd678472 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x148ecd9f2700 (LWP 3757385) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x148ecdbf3700 (LWP 3757384) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x148ecddf4700 (LWP 3757383) "cmsRun"):
#0  0x0000148f2766edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000148f2766eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000148f1f5ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f5ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x148ecdff5700 (LWP 3757382) "cmsRun"):
#0  0x0000148f276701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000148f218e47e8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f1f53f37c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000148f1f53f4b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x148ece1f6700 (LWP 3757381) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x148ece3f7700 (LWP 3757380) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x148ece5f8700 (LWP 3757379) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x148ece7f9700 (LWP 3757378) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x148ece9fa700 (LWP 3757377) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x148ecebfb700 (LWP 3757376) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x148ecedfc700 (LWP 3757375) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x148eceffd700 (LWP 3757374) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x148ecf1fe700 (LWP 3757373) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x148ecf3ff700 (LWP 3757372) "cmsRun"):
#0  0x0000148f273c7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000148f218deb62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000148f218daacd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000148f218e3ef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x148ed03ff700 (LWP 3757371) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c141d in syscall () from /lib64/libc.so.6
#5  0x0000148f2936b46b in tbb::detail::r1::futex_wakeup_one (futex=0x148f22030024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:109
#6  tbb::detail::r1::binary_semaphore::V (this=0x148f22030024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:262
#7  tbb::detail::r1::rml::internal::thread_monitor::notify (this=0x148f22030020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:230
#8  tbb::detail::r1::rml::private_worker::wake_or_launch (this=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:292
#9  tbb::detail::r1::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:412
#10 0x0000148f2936be49 in tbb::detail::r1::rml::private_server::propagate_chain_reaction (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:159
#11 tbb::detail::r1::rml::private_worker::run (this=0x148f2202fe00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:275
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202fe00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x148ed13fd700 (LWP 3757370) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f956 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88c00, waiter=..., this=0x148f25dbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202ff00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202ff00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x148ed1dfe700 (LWP 3757369) "cmsRun"):
#0  0x0000148f273baac1 in poll () from /lib64/libc.so.6
#1  0x0000148f21a3f857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000148f21a3fa54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000148f29f2e5e8 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88900, waiter=..., this=0x148f25dbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x148ed27ff700 (LWP 3757368) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f33f in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b89100, waiter=..., this=0x148f25dbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202fe80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202fe80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x148ed37ff700 (LWP 3757367) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5f33f in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f6ee37 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148eb6b88300, waiter=..., this=0x148f25dbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f2202ff80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f2202ff80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x148ed474c700 (LWP 3757366) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f29f5cf4d in std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2>::__weak_ptr(std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2> const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5  0x0000148f29f5f9f6 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6  0x0000148f29f6eead in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x0000148f29369b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x148e8c7d4600, waiter=..., this=0x148f25dbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x148ed514d700 (LWP 3757365) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c0b8b in sched_yield () from /lib64/libc.so.6
#5  0x0000148f2935dc73 in __gthread_yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/std_thread.h:353
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x148ed5146ef8) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x148ed5146ef0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::outermost_worker_waiter::pause (this=0x148ed5146ef0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:69
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::outermost_worker_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x0000148f29369cb2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x148f25dbba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x0000148f2936bcee in tbb::detail::r1::rml::private_worker::run (this=0x148f22030080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x148f22030080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#18 0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x148efd5ff700 (LWP 3757295) "cmsRun"):
#0  0x0000148f276706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000148f21a3c327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000148f21a3f63a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000148f27cd8a73 in std::execute_native_thread_routine (__p=0x148f00f2be70) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000148f276661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000148f272c18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x148f289fe680 (LWP 3756508) "cmsRun"):
#0  0x0000148f27390098 in nanosleep () from /lib64/libc.so.6
#1  0x0000148f2738ff9e in sleep () from /lib64/libc.so.6
#2  0x0000148f21a3c1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000148f272c0b8b in sched_yield () from /lib64/libc.so.6
#5  0x0000148f29370686 in __gthread_yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693
#6  std::this_thread::yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/std_thread.h:353
#7  tbb::detail::r1::stealing_loop_backoff::pause (this=0x7ffeeb9dc0e8) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/scheduler_common.h:266
#8  tbb::detail::r1::waiter_base::pause (this=0x7ffeeb9dc0e0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:35
#9  tbb::detail::r1::external_waiter::pause (this=0x7ffeeb9dc0e0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:138
#10 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::external_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#11 0x0000148f293724e2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x148f25dbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x148f25dbb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#14 0x0000148f29f445fb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x0000148f29f51296 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x0000148f29f525ed in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-08-12-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#18 0x0000148f2935e9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#19 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#20 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none

[iarspider]

Another occurance: el8_amd64_gcc12/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Sun Aug 18 13:15:09 CEST 2024
Thread 26 (Thread 0x1539aa5ff700 (LWP 3435629) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x1539aafff700 (LWP 3435628) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x1539ab524700 (LWP 3435627) "cmsRun"):
#0  0x0000153a3206c48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001539f688d9a6 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#2  0x00001539f688dfa5 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#3  0x00001539f688b728 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_cc.so.2
#4  0x00001539e569cbf2 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/scram_x86-64-v3/libtensorflow_framework.so.2
#5  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#6  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x1539d7bf2700 (LWP 3435618) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x1539d7df3700 (LWP 3435617) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x1539d7ff4700 (LWP 3435616) "cmsRun"):
#0  0x0000153a3206edd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000153a3206eec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000153a285ce7c6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a285ce879 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x1539d81f5700 (LWP 3435615) "cmsRun"):
#0  0x0000153a320701b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000153a2a7bd7e8 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2853f37c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000153a2853f4b9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x1539d83f6700 (LWP 3435614) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x1539d85f7700 (LWP 3435613) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x1539d87f8700 (LWP 3435612) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x1539d89f9700 (LWP 3435611) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x1539d8bfa700 (LWP 3435610) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x1539d8dfb700 (LWP 3435609) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x1539d8ffc700 (LWP 3435608) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x1539d91fd700 (LWP 3435607) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x1539d93fe700 (LWP 3435606) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x1539d95ff700 (LWP 3435605) "cmsRun"):
#0  0x0000153a31dc7307 in epoll_wait () from /lib64/libc.so.6
#1  0x0000153a2a7b7b62 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000153a2a7b3acd in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000153a2a7bcef7 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x1539da5ff700 (LWP 3435604) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964a0e in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996721d00, waiter=..., this=0x153a307bbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766e00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x1539db5fc700 (LWP 3435603) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964285 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a3497419d in tbb::detail::d1::function_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#3}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x1539966e3200, waiter=..., this=0x153a307bbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766f00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x1539dbffd700 (LWP 3435602) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a349642ae in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x1539968b0a00, waiter=..., this=0x153a307bba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bba80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x1539dd3ff700 (LWP 3435601) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34961fed in std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2>::__weak_ptr(std::__weak_ptr<edm::serviceregistry::ServicesManager, (__gnu_cxx::_Lock_policy)2> const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34964ab5 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#7  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996860c00, waiter=..., this=0x153a307bb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bb980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b766e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x1539dc9fe700 (LWP 3435600) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a31cc141d in syscall () from /lib64/libc.so.6
#5  0x0000153a33d09fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x153a2b766fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x153a2b766fa4) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x153a2b766fa0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x153a2b766f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b766f80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x1539de7c7700 (LWP 3435599) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a348f1c10 in tbb::detail::r1::spawn(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&)@plt () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34964ade in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#7  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996862100, waiter=..., this=0x153a307bbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bbb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#9  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#10 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#11 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#13 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#14 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x1539df1c8700 (LWP 3435598) "cmsRun"):
#0  0x0000153a31d90098 in nanosleep () from /lib64/libc.so.6
#1  0x0000153a31d8ff9e in sleep () from /lib64/libc.so.6
#2  0x0000153a2b171f90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000153a34964285 in edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#5  0x0000153a34974127 in tbb::detail::d1::enqueue_task<edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d07b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x153996746a00, waiter=..., this=0x153a307bba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x153a307bba00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#9  tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#10 0x0000153a33d09cee in tbb::detail::r1::rml::private_worker::run (this=0x153a2b767080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x153a2b767080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#12 0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#13 0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x153a085a9700 (LWP 3435567) "cmsRun"):
#0  0x0000153a320706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x0000153a2b1720e7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#2  0x0000153a2b1754ca in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  0x0000153a326d8a73 in std::execute_native_thread_routine (__p=0x153a0a5cfe80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000153a320661ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000153a31cc18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x153a3371c680 (LWP 3434931) "cmsRun"):
#0  0x0000153a31dbaac1 in poll () from /lib64/libc.so.6
#1  0x0000153a2b1756e7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#2  0x0000153a2b1758e4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  tbb::detail::r1::spawn (t=..., ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:36
#5  0x0000153a349329f8 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#6  0x0000153a33d103e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x153a307bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#7  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x153a307bb900) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#8  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#9  0x0000153a34948f7b in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#10 0x0000153a34955f96 in edm::EventProcessor::endUnfinishedRun(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#11 0x0000153a3495734d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_MULTIARCHS_X_2024-08-18-0000/lib/el8_amd64_gcc12/scram_x86-64-v3/libFWCoreFramework.so
#12 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x0000153a33cfc9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#15 0x0000000000405100 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none

Doesn't reproduce when ran locally