ProductNotFound issue in T0 HLTMonitor processing jobs starting Run 378985

[saumyaphor4252]

Reporting the T0 processing error in jobs for HLTMonitor stream for Run 378985 detailed in
https://cms-talk.web.cern.ch/t/express-paused-jobs-run2024b-productnotfound-error/38544

CRITICAL:root:Error running cmsRun
{‘arguments’: [‘/bin/bash’, ‘/srv/job/WMTaskSpace/cmsRun1/cmsRun1-main.sh’, ‘’, ‘el8_amd64_gcc12’, ‘scramv1’, ‘CMSSW’, ‘CMSSW_14_0_4’, ‘FrameworkJobReport.xml’, ‘cmsRun’, ‘PSet.py’, ’
']}
CMSSW Return code: 8006

CRITICAL:root:Error message: An exception of category ‘ProductNotFound’ occurred while
[0] Processing Event run: 378985 lumi: 224 event: 220047314 stream: 6
[1] Running path ‘dqmoffline_step’
[2] Calling method for module TrackRefitter/‘hltrefittedForPixelDQM’
Exception Message:
RefCore: A request to resolve a reference to a product of type ‘std::vectorreco::TrackExtra’ with ProductID ‘1:3256’
can not be satisfied because the product cannot be found.
Probably the branch containing the product is not stored in the input file.
Additional Info:
[a] If you wish to continue processing events after a ProductNotFound exception,
add “TryToContinue = cms.untracked.vstring(‘ProductNotFound’)” to the “options” PSet in the configuration.

The tarball for the PSet is available at

/afs/cern.ch/user/c/cmst0/public/PausedJobs/Run2024B/ProductNotFound/vocms014.cern.ch-27238-3-log.tar.gz

Somewhat similar symptoms also seem to be for Run378993 with error

type: "Fatal Exception"
details: "An exception of category 'ProductNotFound' occurred while 
   [0] Processing Event run: 378993 lumi: 188 event: 223221502 stream: 1  
   [1] Running path 'dqmoffline_step' 
   [2] Calling method for module TrackingMonitor/'gsfTracksMonitoringHLT' 
Exception Message: 
RefCore: A request to resolve a reference to a product of type 'std::vector<reco::TrackExtra>' with ProductID '1:3118' can not be satisfied because the product cannot be found. 
Probably the branch containing the product is not stored in the input file. 
  Additional Info: 
        [a] If you wish to continue processing events after a ProductNotFound exception, add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration."
exitCode: 8006

FYI @cms-sw/dqm-l2 May also be relevant to the online hlt DQM client crashes starting Run 378981 reported at the DRM today:
https://cmsweb.cern.ch/dqm/dqm-square/api?what=get_logs&id=dqm-source-state-run378981-hostdqmfu-c2b03-45-01-pid3057526&db=production

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @saumyaphor4252.

@makortel, @smuzaffar, @Dr15Jones, @sextonkennedy, @antoniovilela, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[perrotta]

assign dqm

[cmsbuild]

New categories assigned: dqm

@rvenditti,@syuvivida,@tjavaid,@nothingface0,@antoniovagnerini you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mmusich]

just for the record, the PoolOutputModule of the HLTMonitor stream is configured such that it sends the right event content to be consumed at Tier-0:

      'keep *_hltEgammaGsfTracks_*_*',
      'keep *_hltMergedTracksForBTag_*_*',
      'keep *_hltMergedTracks_*_*',
      'keep *_hltPixelTracks_*_*',

thus the recoTrackExtras_*__HLT branches should be present in the input streamer files.
I suspect something along the lines of #39064.

[Pranjal033]

We copied a few LSs from run 378981 (LS=455-465) in the playback region which were affected and reproduced this crash at playback, here is it's log file.

The information about how to reproduce the crash at lxplus can be found here and the streamers have been copied at this path :
/eos/cms/store/group/comm_dqm/Collisions24_tempStreamers/

[syuvivida]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[saumyaphor4252]

Dear all, the issue is now becoming a showstopper for T0 operations with 100s of paused jobs in the HLTMonitor stream and Prompt Reco processing now stopped starting Run 378981: https://cms-talk.web.cern.ch/t/promptreco-is-paused-for-run2024b/38673.

Can the experts please look into it with a high priority.

Thanks and regards,
Saumya (incoming ORM)