Skip to content

L1 tracking update #47067

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Jun 1, 2025
Merged

L1 tracking update #47067

merged 18 commits into from
Jun 1, 2025

Conversation

tschuh
Copy link
Contributor

@tschuh tschuh commented Jan 9, 2025

This PR updates mainly the modules

  • L1Trigger/TrackFindingTracklet
  • L1Trigger/TrackFindingTMTT
  • L1Trigger/TrackTrigger
  • L1Trigger/TrackerDTC
  • L1Trigger/TrackerTFP
  • DataFormats/L1TrackTrigger
  • SimTracker/TrackTriggerAssociation

in order to sync s/w emulation of L1 tracking with current f/w, which means to make the s/w more realistic. Some name changes have also been propagated to additional modules.

some specific added features:

  • KF simulator callable in KF emulator chain
  • KF starts state updating process from 2-layer states instead of 0-layer states
  • 5 parameter fit simulation added to KF emulator as an option
  • DR uses 11 layer scheme
  • DR receives now full 10b stub identifier for all stubs
  • KFout module splitted into track qulaity and final output formatter
  • DRin renamed in TrackMultiplexer, merges now all seed type channel into one
  • L1FPGATrackProducer produces now emulator output in one step rendering module TBout obsolete

This PR is based of CMSSW_14_2_0_pre4 rebased to master on Jan 9.

The code in this PR passed the git CI tests of the L1 track group, seen in the CI results. The git CI tests run the baseline L1 tracking algorithm (both for prompt and displaced tracking) on ttbar events with zero pileup (to save CPU). The CI checks require that compared with reference, the tracking efficiency does not fall significantly, the number of reconstructed tracks does not increase significantly, and that the z0 resolution does not get significantly worse. (The exact cuts can be found here).

@cmsbuild
Copy link
Contributor

cmsbuild commented Jan 9, 2025

cms-bot internal usage

@cmsbuild
Copy link
Contributor

cmsbuild commented Jan 9, 2025

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47067/43234

@cmsbuild
Copy link
Contributor

cmsbuild commented Jan 9, 2025

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47067/43235

@cmsbuild
Copy link
Contributor

cmsbuild commented Jan 9, 2025

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47067/43238

@civanch
Copy link
Contributor

civanch commented May 31, 2025

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @mandrenguyen, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

@mandrenguyen
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 13a0ef8 into cms-sw:master Jun 1, 2025
11 checks passed
@smuzaffar
Copy link
Contributor

@tschuh , there are many relvals failing in IBs with error [a] . Looks like the change here is triggering these failures. Can you please look in to it and provide a fix?

[a]

----- Begin Fatal Exception 02-Jun-2025 02:05:07 CEST-----------------------
An exception of category 'RunTimeError.' occurred while
   [0] Processing  Event run: 1 lumi: 64 event: 6301 stream: 0
   [1] Running path 'L1TrackTrigger_step'
   [2] Calling method for module trackerDTC::ProducerDTC/'ProducerDTC'
   [3] TTBV::checkU
Exception Message:
Value 1 does not fit into a -3b binary.
----- End Fatal Exception -------------------------------------------------

@tschuh
Copy link
Contributor Author

tschuh commented Jun 3, 2025

I had a look and struggle to reproduce the error. Looks like it is a feature for CMSSW_13_2_0_pre1 samples using a Geometry (Extended2026) which doesn't exist anymore. I can't run this sample with CMSSW_15_1_X therefore, but I am also getting compilation errors when I try to compile this branch in CMSSW_13_2_0.

@tschuh
Copy link
Contributor Author

tschuh commented Jun 3, 2025

Looks for me however that we have again a case where the workflow tests L1 tracking with unsupported configuration. In the past I did detect such cases inside my EDProducers and let the produce function immediately return without executing the tracking. I found it however to be too cumbersome to maintain this detection of valid configurations. Is it possible to tune the tests to not run the tracking when it is too outdated?

@makortel
Copy link
Contributor

makortel commented Jun 3, 2025

I can't run this sample with CMSSW_15_1_X therefore

The files are available in the IB EOS cache. E.g. on lxplus run runTheMatrix.py -l 24834.0 --ibeos.

Is it possible to tune the tests to not run the tracking when it is too outdated?

I suppose this question would be for @cms-sw/upgrade-l2.

@tschuh
Copy link
Contributor Author

tschuh commented Jun 3, 2025

So I did run runTheMatrix.py -l 24834.0 --ibeos and got this output

processing relval_standard
processing relval_highstats
processing relval_pileup
processing relval_generator
processing relval_extendedgen
processing relval_production
processing relval_ged
ignoring relval_upgrade from default matrix
ignoring relval_cleanedupgrade from default matrix
ignoring relval_gpu from default matrix
processing relval_2017
processing relval_Run4
ignoring relval_identity from default matrix
processing relval_machine
processing relval_premix
processing relval_nano
ignoring relval_data_highstats from default matrix
Running up to 4 concurrent jobs, each with 1 thread per process

Preparing to run 24834.0 TTbar_14TeV+Run4D98

# in: /scratch/tschuh/CMSSW_15_1_X_2025-06-01-2300/src going to execute cd 24834.0_TTbar_14TeV+Run4D98
 cmsDriver.py TTbar_14TeV_TuneCP5_cfi  -s GEN,SIM -n 10 --conditions auto:phase2_realistic_T25 --beamspot DBrealisticHLLHC --datatier GEN-SIM --eventcontent FEVTDEBUG --geometry ExtendedRun4D98 --era Phase2C17I13M9 --relval 9000,100 --fileout file:step1.root  > step1_TTbar_14TeV+Run4D98.log  2>&1
 

# in: /scratch/tschuh/CMSSW_15_1_X_2025-06-01-2300/src going to execute cd 24834.0_TTbar_14TeV+Run4D98
 cmsDriver.py step2  -s DIGI:pdigi_valid,L1TrackTrigger,L1,L1P2GT,DIGI2RAW,HLT:@relvalRun4 --conditions auto:phase2_realistic_T25 --datatier GEN-SIM-DIGI-RAW -n 10 --eventcontent FEVTDEBUGHLT --geometry ExtendedRun4D98 --era Phase2C17I13M9 --filein  file:step1.root  --fileout file:step2.root  > step2_TTbar_14TeV+Run4D98.log  2>&1
 

# in: /scratch/tschuh/CMSSW_15_1_X_2025-06-01-2300/src going to execute cd 24834.0_TTbar_14TeV+Run4D98
 cmsDriver.py step3  -s RAW2DIGI,RECO,RECOSIM,PAT,VALIDATION:@phase2Validation+@miniAODValidation,DQM:@phase2+@miniAODDQM --conditions auto:phase2_realistic_T25 --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO -n 10 --eventcontent FEVTDEBUGHLT,MINIAODSIM,DQM --geometry ExtendedRun4D98 --era Phase2C17I13M9 --filein  file:step2.root  --fileout file:step3.root  > step3_TTbar_14TeV+Run4D98.log  2>&1
 

# in: /scratch/tschuh/CMSSW_15_1_X_2025-06-01-2300/src going to execute cd 24834.0_TTbar_14TeV+Run4D98
 cmsDriver.py step4  -s HARVESTING:@phase2Validation+@phase2+@miniAODValidation+@miniAODDQM --conditions auto:phase2_realistic_T25 --mc  --geometry ExtendedRun4D98 --scenario pp --filetype DQM --era Phase2C17I13M9 -n 100  --filein file:step3_inDQM.root --fileout file:step4.root  > step4_TTbar_14TeV+Run4D98.log  2>&1
 

# in: /scratch/tschuh/CMSSW_15_1_X_2025-06-01-2300/src going to execute cd 24834.0_TTbar_14TeV+Run4D98
 cmsDriver.py step5  -s ALCA:SiPixelCalSingleMuonLoose+SiPixelCalSingleMuonTight+TkAlMuonIsolated+TkAlMinBias+MuAlOverlaps+EcalESAlign+TkAlZMuMu+TkAlDiMuonAndVertex+HcalCalHBHEMuonProducerFilter+TkAlUpsilonMuMu+TkAlJpsiMuMu --conditions auto:phase2_realistic_T25 --datatier ALCARECO -n 10 --eventcontent ALCARECO --geometry ExtendedRun4D98 --era Phase2C17I13M9 --filein file:step3.root --fileout file:step5.root  > step5_TTbar_14TeV+Run4D98.log  2>&1
 
24834.0_TTbar_14TeV+Run4D98 Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED  - time date Tue Jun  3 15:28:23 2025-date Tue Jun  3 14:56:23 2025; exit: 0 0 0 0 0
1 1 1 1 1 tests passed, 0 0 0 0 0 failed

which looks good to me, what am I doing wrong?

@Dr15Jones
Copy link
Contributor

The problem doesn't happen consistently. E.g. in the 2025-06-02-2300 IB 24834.0 did not fail but
24834.911 did fail. See all the 'OtherCMS' failures in

https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_15_1/2025-06-02-2300?selectedArchs=el9_amd64_gcc12&selectedFlavors=X&selectedStatus=failed

@makortel
Copy link
Contributor

makortel commented Jun 3, 2025

Could this be a threading problem then? Repeating the step 2 with e.g. 4 threads some number of times could hopefully reproduce the problem.

@dan131riley
Copy link

Could this be a threading problem then? Repeating the step 2 with e.g. 4 threads some number of times could hopefully reproduce the problem.

Thread-related is my guess (I've also been puzzling over the non-replicability), so not surprised that a single threaded test fails to reproduce the failure.

@tschuh
Copy link
Contributor Author

tschuh commented Jun 3, 2025

I see. To explain the error thrown: during tracking emulation we request to put a variable in a binary with negative number of bits, which is nonsense. The bit widths are calculated in an ESProduct (called trackerTFP::Dataformats).

@dan131riley
Copy link

Exception seen in a UBSAN build WF 34434.0 step2, with some possibly helpful UBSAN runtime errors. The first two look like an negative array index (UBSAN hits also have the virtue of line numbers):

/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:61:36: runtime error: index 18446744073709551613 out of bounds for type 'double [65]'
    #0 0x148c62db4afe in std::__array_traits<double, 65ul>::_S_ref(double const (&) [65], unsigned long) /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:61
    #1 0x148c62db4afe in std::array<double, 65ul>::operator[](unsigned long) const /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:219
    #2 0x148c62db4afe in TTBV::iMax() const src/DataFormats/L1TrackTrigger/interface/TTBV.h:387
    #3 0x148c62db4afe in TTBV::checkU(unsigned long long) src/DataFormats/L1TrackTrigger/interface/TTBV.h:394
    #4 0x148c62db4afe in TTBV::checkI(int) src/DataFormats/L1TrackTrigger/interface/TTBV.h:428
    #5 0x148c62da0f1a in TTBV::TTBV(int, int, bool) src/DataFormats/L1TrackTrigger/interface/TTBV.h:49
    #6 0x148c62da0f1a in void trackerTFP::DataFormats::convertStub<double, double, double, TTBV, int, int, int, int, int, int>(trackerTFP::Process, std::tuple<double, double, double, TTBV, int, int, int, int, int, int> const&, std::bitset<64ul>&) const src/L1Trigger/TrackerTFP/interface/DataFormats.h:320
    #7 0x148c62da0f1a in trackerTFP::Stub<double, double, double, TTBV, int, int, int, int, int, int>::Stub(edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&, trackerTFP::DataFormats const*, trackerTFP::Process, double, double, double, TTBV, int, int, int, int, int, int) src/L1Trigger/TrackerTFP/interface/DataFormats.h:437
    #8 0x148c62da0f1a in trackerTFP::StubDTC::StubDTC(edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&, trackerTFP::DataFormats const*, double, double, double, TTBV const&, int, int, int, int, int, int) src/L1Trigger/TrackerTFP/interface/DataFormats.h:477
    #9 0x148c62da0f1a in trackerDTC::Stub::formatTMTT(int) const src/L1Trigger/TrackerDTC/src/Stub.cc:192
    #10 0x148c62db323c in trackerDTC::Stub::frame(int) const src/L1Trigger/TrackerDTC/src/Stub.cc:135
    #11 0x148c62d59913 in operator() src/L1Trigger/TrackerDTC/src/DTC.cc:153
    #12 0x148c62d59913 in transform<std::_Deque_iterator<trackerDTC::Stub*, trackerDTC::Stub* const&, trackerDTC::Stub* const*>, std::back_insert_iterator<std::vector<std::pair<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi> > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi> >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi> > >::FindForDetSetVector>, std::bitset<64> > > >, trackerDTC::DTC::produce(const Stubss&, TTDTC&)::<lambda(trackerDTC::Stub*)> > /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_algo.h:4263
    #13 0x148c62d59913 in trackerDTC::DTC::produce(std::vector<std::deque<trackerDTC::Stub*, std::allocator<trackerDTC::Stub*> >, std::allocator<std::deque<trackerDTC::Stub*, std::allocator<trackerDTC::Stub*> > > > const&, TTDTC&) src/L1Trigger/TrackerDTC/src/DTC.cc:157
    #14 0x148c62d6a045 in trackerDTC::DTC::produce(TTDTC&, TTDTC&) src/L1Trigger/TrackerDTC/src/DTC.cc:81
    #15 0x148c224d535a in trackerDTC::ProducerDTC::produce(edm::Event&, edm::EventSetup const&) src/L1Trigger/TrackerDTC/plugins/ProducerDTC.cc:101
    #16 0x148cc465ce7f in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:83

/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:61:36: runtime error: index 18446744073709551613 out of bounds for type 'double [65]'
    #0 0x148c62db4ac2 in std::__array_traits<double, 65ul>::_S_ref(double const (&) [65], unsigned long) /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:61
    #1 0x148c62db4ac2 in std::array<double, 65ul>::operator[](unsigned long) const /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/array:219
    #2 0x148c62db4ac2 in TTBV::iMax() const src/DataFormats/L1TrackTrigger/interface/TTBV.h:387
    #3 0x148c62db4ac2 in TTBV::checkU(unsigned long long) src/DataFormats/L1TrackTrigger/interface/TTBV.h:394
    #4 0x148c62db4ac2 in TTBV::checkI(int) src/DataFormats/L1TrackTrigger/interface/TTBV.h:428
    #5 0x148c62da0f1a in TTBV::TTBV(int, int, bool) src/DataFormats/L1TrackTrigger/interface/TTBV.h:49
    #6 0x148c62da0f1a in void trackerTFP::DataFormats::convertStub<double, double, double, TTBV, int, int, int, int, int, int>(trackerTFP::Process, std::tuple<double, double, double, TTBV, int, int, int, int, int, int> const&, std::bitset<64ul>&) const src/L1Trigger/TrackerTFP/interface/DataFormats.h:320
    #7 0x148c62da0f1a in trackerTFP::Stub<double, double, double, TTBV, int, int, int, int, int, int>::Stub(edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&, trackerTFP::DataFormats const*, trackerTFP::Process, double, double, double, TTBV, int, int, int, int, int, int) src/L1Trigger/TrackerTFP/interface/DataFormats.h:437
    #8 0x148c62da0f1a in trackerTFP::StubDTC::StubDTC(edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&, trackerTFP::DataFormats const*, double, double, double, TTBV const&, int, int, int, int, int, int) src/L1Trigger/TrackerTFP/interface/DataFormats.h:477
    #9 0x148c62da0f1a in trackerDTC::Stub::formatTMTT(int) const src/L1Trigger/TrackerDTC/src/Stub.cc:192
    #10 0x148c62db323c in trackerDTC::Stub::frame(int) const src/L1Trigger/TrackerDTC/src/Stub.cc:135
    #11 0x148c62d59913 in operator() src/L1Trigger/TrackerDTC/src/DTC.cc:153
    #12 0x148c62d59913 in transform<std::_Deque_iterator<trackerDTC::Stub*, trackerDTC::Stub* const&, trackerDTC::Stub* const*>, std::back_insert_iterator<std::vector<std::pair<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi> > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi> >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi> > >::FindForDetSetVector>, std::bitset<64> > > >, trackerDTC::DTC::produce(const Stubss&, TTDTC&)::<lambda(trackerDTC::Stub*)> > /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_algo.h:4263
    #13 0x148c62d59913 in trackerDTC::DTC::produce(std::vector<std::deque<trackerDTC::Stub*, std::allocator<trackerDTC::Stub*> >, std::allocator<std::deque<trackerDTC::Stub*, std::allocator<trackerDTC::Stub*> > > > const&, TTDTC&) src/L1Trigger/TrackerDTC/src/DTC.cc:157
    #14 0x148c62d6a045 in trackerDTC::DTC::produce(TTDTC&, TTDTC&) src/L1Trigger/TrackerDTC/src/DTC.cc:81
    #15 0x148c224d535a in trackerDTC::ProducerDTC::produce(edm::Event&, edm::EventSetup const&) src/L1Trigger/TrackerDTC/plugins/ProducerDTC.cc:101
    #16 0x148cc465ce7f in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:83

src/L1Trigger/TrackTrigger/interface/Setup.h:277:37: runtime error: load of value 140, which is not a valid value for type 'bool'
    #0 0x148c62d4cd5f in tt::Setup::useHybrid() const src/L1Trigger/TrackTrigger/interface/Setup.h:277
    #1 0x148c62d4cd5f in trackerDTC::Stub::Stub(tt::Setup const*, trackerTFP::DataFormats const*, trackerDTC::LayerEncoding const*, tt::SensorModule const*, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&) src/L1Trigger/TrackerDTC/src/Stub.cc:115
    #2 0x148c62d6e328 in decltype (::new ((void*)(0)) trackerDTC::Stub((declval<tt::Setup const*&>)(), (declval<trackerTFP::DataFormats const*&>)(), (declval<trackerDTC::LayerEncoding const*&>)(), (declval<tt::SensorModule const*&>)(), (declval<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&>)())) std::construct_at<trackerDTC::Stub, tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&>(trackerDTC::Stub*, tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&) /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_construct.h:97
    #3 0x148c62d6e328 in void std::allocator_traits<std::allocator<trackerDTC::Stub> >::construct<trackerDTC::Stub, tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&>(std::allocator<trackerDTC::Stub>&, trackerDTC::Stub*, tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&) /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/alloc_traits.h:518
    #4 0x148c62d6e328 in trackerDTC::Stub& std::vector<trackerDTC::Stub, std::allocator<trackerDTC::Stub> >::emplace_back<tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&>(tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&) /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/vector.tcc:117
    #5 0x148c62d6e328 in trackerDTC::DTC::DTC(tt::Setup const*, trackerTFP::DataFormats const*, trackerDTC::LayerEncoding const*, int, std::vector<std::vector<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::allocator<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> > >, std::allocator<std::vector<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::allocator<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> > > > > const&) src/L1Trigger/TrackerDTC/src/DTC.cc:40
    #6 0x148c224d5344 in trackerDTC::ProducerDTC::produce(edm::Event&, edm::EventSetup const&) src/L1Trigger/TrackerDTC/plugins/ProducerDTC.cc:99
    #7 0x148cc465ce7f in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:83

@dan131riley
Copy link

Another possibly relevant UBSAN hit preceding the others in the previous comment:

src/L1Trigger/TrackTrigger/interface/Setup.h:277:37: runtime error: load of value 6, which is not a valid value for type 'bool'
    #0 0x15084c54cd5f in tt::Setup::useHybrid() const src/L1Trigger/TrackTrigger/interface/Setup.h:277
    #1 0x15084c54cd5f in trackerDTC::Stub::Stub(tt::Setup const*, trackerTFP::DataFormats const*, trackerDTC::LayerEncoding const*, tt::SensorModule const*, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&) src/L1Trigger/TrackerDTC/src/Stub.cc:115
    #2 0x15084c56e328 in decltype (::new ((void*)(0)) trackerDTC::Stub((declval<tt::Setup const*&>)(), (declval<trackerTFP::DataFormats const*&>)(), (declval<trackerDTC::LayerEncoding const*&>)(), (declval<tt::SensorModule const*&>)(), (declval<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&>)())) std::construct_at<trackerDTC::Stub, tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&>(trackerDTC::Stub*, tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&) /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_construct.h:97
    #3 0x15084c56e328 in void std::allocator_traits<std::allocator<trackerDTC::Stub> >::construct<trackerDTC::Stub, tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&>(std::allocator<trackerDTC::Stub>&, trackerDTC::Stub*, tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&) /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/alloc_traits.h:518
    #4 0x15084c56e328 in trackerDTC::Stub& std::vector<trackerDTC::Stub, std::allocator<trackerDTC::Stub> >::emplace_back<tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&>(tt::Setup const*&, trackerTFP::DataFormats const*&, trackerDTC::LayerEncoding const*&, tt::SensorModule const*&, edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> const&) /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/vector.tcc:117
    #5 0x15084c56e328 in trackerDTC::DTC::DTC(tt::Setup const*, trackerTFP::DataFormats const*, trackerDTC::LayerEncoding const*, int, std::vector<std::vector<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::allocator<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> > >, std::allocator<std::vector<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::allocator<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector> > > > > const&) src/L1Trigger/TrackerDTC/src/DTC.cc:40
    #6 0x15080b8d5344 in trackerDTC::ProducerDTC::produce(edm::Event&, edm::EventSetup const&) src/L1Trigger/TrackerDTC/plugins/ProducerDTC.cc:99
    #7 0x1508ada5ce7f in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:83

@tschuh
Copy link
Contributor Author

tschuh commented Jun 4, 2025

ah. I see that we dropped the init of useHybrid_ in tt::Setup by accident. Shall I push a commit to this branch or create a new PR with a fix for that?

@makortel
Copy link
Contributor

makortel commented Jun 4, 2025

You need to make a new PR to fix it (nothing can be done here after the merge).

@tschuh
Copy link
Contributor Author

tschuh commented Jun 4, 2025

I created this PR #48247 which should get rid of the UBSAN runtime errors pointed by Dan.

@BenjaminRS BenjaminRS mentioned this pull request Jun 5, 2025
cmsbuild added a commit that referenced this pull request Jun 9, 2025
@tschuh
Copy link
Contributor Author

tschuh commented Jun 10, 2025

Looks like the uninitiated variable was the problem. Am I seeing it correct that there no further outstanding issues?

@smuzaffar
Copy link
Contributor

@tschuh , yes looks like #48247 has fixed the problem. We still see 4 relvals failing ( not yet visible on IB dashboard though) but I think those are not related to L1 code

@tschuh
Copy link
Contributor Author

tschuh commented Jun 10, 2025

Great, many thanks for the hints and help and sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.