NanoAOD VertexException in PromptReco_Run381515_ParkingVBF0 (CMSSW 14_0_7, on AMD arch)

[gpetruc]

A PromptReco job failure in the NanoAOD step was observed at the tier0 with the following error message
cms-talk thead: https://cms-talk.web.cern.ch/t/paused-job-for-promptreco-run381515-parkingvbf0-vertexexception/42163

----- Begin Fatal Exception 11-Jun-2024 10:46:46 CEST-----------------------
An exception of category 'VertexException' occurred while
   [0] Processing  Event run: 381515 lumi: 384 event: 765632765 stream: 0
   [1] Running path 'write_NANOAOD_step'
   [2] Prefetching for module PoolOutputModule/'write_NANOAOD'
   [3] Prefetching for module SimplePATMuonFlatTableProducer/'muonTable'
   [4] Calling method for module MuonBeamspotConstraintValueMapProducer/'muonBSConstrain'
Exception Message:
BasicSingleVertexState::could not invert weight matrix
----- End Fatal Exception -------------------------------------------------

The exception appears to be reproducible running on a single event, but only on AMD: the job fails at Tier0 (AMD EPYC 7763) and on my desktop (AMD Ryzen 9 5950X), but not on another Intel machine I tested (Intel Xeon Silver 4216).

Instructions to reproduce it on an EL8 AMD machine:

export SCRAM_ARCH=el8_amd64_gcc12
cmsrel CMSSW_14_0_7
cd CMSSW_14_0_7/src
cmsenv
cp /afs/cern.ch/user/c/cmst0/public/PausedJobs/Run2024E/vertexException/job/WMTaskSpace/cmsRun1/PSet.pkl .
cat > PSet_one.py <<END
import FWCore.ParameterSet.Config as cms
import pickle
with open('PSet.pkl', 'rb') as handle:
    process = pickle.load(handle)

process.source.eventsToProcess = cms.untracked.VEventRange("381515:384:765632765",)
process.options.wantSummary = cms.untracked.bool(True)
process.options.numberOfThreads = 1
process.options.numberOfStreams = 1
END
cmsRun PSet_one.py 2>&1 | tee PSet_one.log

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @gpetruc.

@Dr15Jones, @rappoccio, @makortel, @smuzaffar, @antoniovilela, @sextonkennedy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

Assign RecoMuon/GlobalTrackingTools

[cmsbuild]

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[jfernan2]

type muon

[gpetruc]

Adding @namapane as I believe the exception comes from the 14_0_X backport of #42646

[namapane]

Adding @namapane as I believe the exception comes from the 14_0_X backport of #42646

TNX, looking into it.

[namapane]

The problem seems to be that this event has a PV which, despite being isValid(), has a ~zero covariance() matrix:

          auto pv = pvHandle->at(0);
	  cout << pv.isValid() << endl << pv.covariance() << endl;

gives:

1
[  6.30659e-07 2.46501e-08-1.17693e-06
   2.46501e-08 5.49215e-07-3.72355e-07
  -1.17693e-06-3.72355e-07 6.69129e-06 ]

This goes through SingleTrackVertexConstraint::constrain() -> KalmanVertexTrackUpdator::update() -> KVFHelper::vertexChi2 which trivially fails at this point.
The event also has a 2.8e9 GeV muon, so there must be something pathologic.

I can't think of a way of adding a simple protection to check for the covariance matrix to be sensible, so I think the easiest solution is to catch the exception in MuonBeamspotConstraintValueMapProducer.
Let me know if you have objections or better suggestions.