Invalid reads in LHAPDF::LogBicubicInterpolator::_interpolateXQ2()

[makortel]

This issue spins off the discovery in #44271 (comment) that running step1 of workflow 9.0 in CMSSW_14_1_X_2024-03-04-1100 with valgrind shows many

==4984== Invalid read of size 8
==4984==    at 0x426BC33C: LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray const&, double, unsigned long, double, unsigned long, int) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x426A82E9: LHAPDF::PDF::xfxQ2(int, double, double) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x4272B0EB: structm_ (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x404581CE: pypdfu_ (pypdfu.f:356)
==4984==    by 0x404A053B: pysigh_ (pysigh.f:420)
==4984==    by 0x40439FF7: pymign_ (pymign.f:80)
==4984==    by 0x404267E3: pyinit_ (pyinit.f:287)
==4984==    by 0x4025A94E: gen::Pythia6Hadronizer::initializeForInternalPartons() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4025B92E: edm::GeneratorFilter<gen::Pythia6Hadronizer, gen::ExternalDecayDriver>::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4C62378: edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4C4C8CF: edm::WorkerT<edm::one::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B9F95E: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4BA36ED: edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x42131CF: tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-0000/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:322)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:458)
==4984==    by 0x63F9290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==4984==    by 0x4B61CBA: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6B629: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6BB80: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/bin/slc7_amd64_gcc12/cmsRun)
==4984==    by 0x63E59BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==4984==  Address 0x7b6e53f8 is 8 bytes before a block of size 400 alloc'd
==4984==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4984==    by 0x5285FFE: std::vector<double, std::allocator<double> >::_M_default_append(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/lcg/root/6.30.05-09dd1c56883e4f434d3cfa5dffc18d6e/lib/libMathCore.so)
==4984==    by 0x426BA5C2: LHAPDF::KnotArray::fillLogKnots() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x426B68EC: LHAPDF::GridPDF::_loadData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x426CACDE: LHAPDF::mkPDF(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x427287EB: (anonymous namespace)::PDFSetHandler::loadMember(int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x4272E0F4: pdfset_ (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x404264A5: pyinit_ (pyinit.f:115)
==4984==    by 0x4025A94E: gen::Pythia6Hadronizer::initializeForInternalPartons() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4025B92E: edm::GeneratorFilter<gen::Pythia6Hadronizer, gen::ExternalDecayDriver>::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4C62378: edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4C4C8CF: edm::WorkerT<edm::one::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B9F95E: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4BA36ED: edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x42131CF: tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-0000/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:322)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:458)
==4984==    by 0x63F9290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==4984==    by 0x4B61CBA: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6B629: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6BB80: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{

and

==4984== Conditional jump or move depends on uninitialised value(s)
==4984==    at 0x404C0CED: pystrf_ (pystrf.f:145)
==4984==    by 0x4041C539: pyexec_ (pyexec.f:134)
==4984==    by 0x40419BB3: pyevnw_ (pyevnw.f:431)
==4984==    by 0x40259FB9: gen::Pythia6Hadronizer::generatePartonsAndHadronize() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4025D950: edm::GeneratorFilter<gen::Pythia6Hadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4C612FD: edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4C4C63D: edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4BDE3D8: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4BDF4AE: edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x42131CF: tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-0000/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:322)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:458)
==4984==    by 0x63F9290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==4984==    by 0x4B61CBA: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6B629: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6BB80: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/bin/slc7_amd64_gcc12/cmsRun)
==4984==    by 0x63E59BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==4984==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/bin/slc7_amd64_gcc12/cmsRun)
==4984==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/bin/slc7_amd64_gcc12/cmsRun)

[makortel]

assign generators

[cmsbuild]

New categories assigned: generators

@alberto-sanchez,@bbilin,@GurpreetSinghChahal,@mkirsano,@menglu21,@SiewYan you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@antoniovilela, @rappoccio, @sextonkennedy, @makortel, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[iarspider]

Possibly related crash in CMSSW_14_1_DEVEL_X_2024-08-06-2300:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Aug  7 03:57:09 CEST 2024
Thread 5 (Thread 0x146fe05ff700 (LWP 2186482) "cmsRun"):
#0  0x0000147016faf098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147016faef9e in sleep () from /lib64/libc.so.6
#2  0x00001470126af1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147016ee041d in syscall () from /lib64/libc.so.6
#5  0x0000147018149fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x1470130a2024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x1470130a2024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x1470130a2020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x1470130a2000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1470130a2000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001470172851ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000147016ee08d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x146fe15a8700 (LWP 2186480) "cmsRun"):
#0  0x0000147016faf098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147016faef9e in sleep () from /lib64/libc.so.6
#2  0x00001470126af1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147016ee041d in syscall () from /lib64/libc.so.6
#5  0x0000147018149fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x1470130a2124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x1470130a2124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x1470130a2120) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x1470130a2100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1470130a2100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001470172851ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000147016ee08d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x146fe1fa9700 (LWP 2186479) "cmsRun"):
#0  0x0000147016fd9ac1 in poll () from /lib64/libc.so.6
#1  0x00001470126b2857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001470126b2a54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000146fddc7533c in LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray const&, double, unsigned long, double, unsigned long, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#5  0x0000146fddc612ea in LHAPDF::PDF::xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#6  0x0000146fb8d39b26 in Pythia8::LHAPDF6::xfUpdate(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8lhapdf6.so
#7  0x0000146fde34d5b6 in Pythia8::PDF::xf(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#8  0x0000146fde30b884 in Pythia8::MultipartonInteractions::upperEnvelope() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#9  0x0000146fde31c807 in Pythia8::MultipartonInteractions::init(bool, int, Pythia8::BeamParticle*, Pythia8::BeamParticle*, std::shared_ptr<Pythia8::PartonVertex>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#10 0x0000146fde363e5e in Pythia8::PartonLevel::init(std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::SpaceShower>, Pythia8::RHadrons*, std::shared_ptr<Pythia8::MergingHooks>, std::shared_ptr<Pythia8::PartonVertex>, std::shared_ptr<Pythia8::StringInteractions>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#11 0x0000146fde3f98e2 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#12 0x0000146fe0ab3eef in Pythia8Hadronizer::initializeForExternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so
#13 0x0000146fe0ad66ba in edm::ConcurrentHadronizerFilter<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>::initLumi(edm::gen::StreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>*, edm::LuminosityBlock const&, edm::EventSetup const&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so
#14 0x0000146fe0ac4dd8 in edm::ConcurrentHadronizerFilter<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>::globalBeginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so
#15 0x0000147019a5e2fb in edm::global::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x0000147019a55aa0 in edm::WorkerT<edm::global::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x00001470199961ca in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x0000147019985e8b in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00001470199445de in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000147018147b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x147015720b00, waiter=..., this=0x1470157c9400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#21 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x1470157c9400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#22 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#23 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#24 0x0000147018149cee in tbb::detail::r1::rml::private_worker::run (this=0x1470130a2080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1470130a2080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#26 0x00001470172851ca in start_thread () from /lib64/libpthread.so.0
#27 0x0000147016ee08d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x146ff03ff700 (LWP 2186458) "cmsRun"):
#0  0x000014701728f6a2 in waitpid () from /lib64/libpthread.so.0
#1  0x00001470126af327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001470126b263a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000147017919a73 in std::execute_native_thread_routine (__p=0x146ff19636b0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00001470172851ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000147016ee08d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1470163fd680 (LWP 2186334) "cmsRun"):
#0  0x0000147016faf098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147016faef9e in sleep () from /lib64/libc.so.6
#2  0x00001470126af1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147016ee041d in syscall () from /lib64/libc.so.6
#5  0x000014701814eca4 in tbb::detail::r1::futex_wait (comparand=2, futex=0x7fffb8ddbb50) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x7fffb8ddbb50) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0x7fffb8ddbb20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x1470157cb598) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=<synthetic pointer>..., this=0x1470157cb598) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=<optimized out>, wakeup_condition=..., uniq_tag=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::external_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 0x00001470181504e2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x1470157c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x1470157c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x000014701995a5fb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00001470199680ef in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x00001470199685a1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x000014701813c9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000405100 in main ()

Current Modules:

Module: Pythia8ConcurrentHadronizerFilter:generator (crashed)
Module: none
Module: none
Module: none

[iarspider]

New occurance: link

Thread 3 (Thread 0x150df77ad700 (LWP 2456340) "cmsRun"):
#0  0x0000150e2e1baac1 in poll () from /lib64/libc.so.6
#1  0x0000150e2a5dacd7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-08-18-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000150e2a5e07f3 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-08-18-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000150df436633c in LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray const&, double, unsigned long, double, unsigned long, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#5  0x0000150df43522ea in LHAPDF::PDF::xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#6  0x0000150dcd14db26 in Pythia8::LHAPDF6::xfUpdate(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8lhapdf6.so
#7  0x0000150df323e5b6 in Pythia8::PDF::xf(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#8  0x0000150df31fc884 in Pythia8::MultipartonInteractions::upperEnvelope() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#9  0x0000150df320d807 in Pythia8::MultipartonInteractions::init(bool, int, Pythia8::BeamParticle*, Pythia8::BeamParticle*, std::shared_ptr<Pythia8::PartonVertex>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#10 0x0000150df3254e5e in Pythia8::PartonLevel::init(std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::SpaceShower>, Pythia8::RHadrons*, std::shared_ptr<Pythia8::MergingHooks>, std::shared_ptr<Pythia8::PartonVertex>, std::shared_ptr<Pythia8::StringInteractions>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#11 0x0000150df32ea8e2 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#12 0x0000150df6268abc in Pythia8Hadronizer::initializeForExternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-08-18-0000/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so

[lviliani]

Is this still relevant?

[makortel]

Last recorded occurrence was two weeks ago, so yes, looks still relevant.

[lviliani]

Sorry, didn't realize the last messages are so recent.
Anyway I don't seem to be able to open the links above.

[makortel]

The IB logs are cleaned up after about two weeks. In order to reproduce I'd suggest to run one of the reported jobs through valgrind (which is how the problem first showed up), i.e. one of

• workflow 9.0 step 1
• workflow 515.0 step 2
• workflow 518.0 step 2

We have also other LHAPDF issues that may be related or not: #35082, #35251, #43526. Actually #43526 (comment) seems to show these same symptoms (in workflow 561.0 step 1)

[dan131riley]

Seen again, CMSSW_15_1_ROOT6_X_2025-04-01-2300, el8_amd64_gcc12, wf 161.4 step1:

#2  0x0000146ad32fef54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000146aa308a33c in LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray const&, double, unsigned long, double, unsigned long, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#5  0x0000146aa30762ea in LHAPDF::PDF::xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#6  0x0000146a95a52b26 in Pythia8::LHAPDF6::xfUpdate(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/external/el8_amd64_gcc12/lib/libpythia8lhapdf6.so
#7  0x0000146aa2a3e5b6 in Pythia8::PDF::xf(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#8  0x0000146aa29fc7d7 in Pythia8::MultipartonInteractions::upperEnvelope() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#9  0x0000146aa2a0d807 in Pythia8::MultipartonInteractions::init(bool, int, Pythia8::BeamParticle*, Pythia8::BeamParticle*, std::shared_ptr<Pythia8::PartonVertex>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#10 0x0000146aa2a54e5e in Pythia8::PartonLevel::init(std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::SpaceShower>, Pythia8::RHadrons*, std::shared_ptr<Pythia8::MergingHooks>, std::shared_ptr<Pythia8::PartonVertex>, std::shared_ptr<Pythia8::StringInteractions>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#11 0x0000146aa2aea8e2 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#12 0x0000146aa51a100d in Pythia8Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02883/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_ROOT6_X_2025-04-01-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so