[PY312_X] Tests TestConfigDP_X timing out

[iarspider]

In PY312_X IBs, TestConfigDP_X tests are timing out. Log example (full log):

===== Test "TestConfigDP_12" ====
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_15_1_PY312_X_2025-03-19-2300/src/Configuration/DataProcessing/test/RunExpressProcessing.py --scenario trackingOnlyEra_Run2_2018 --global-tag GLOBALTAG --lfn /store/whatever --alcarecos=TkAlMinBias+PromptCalibProdBeamSpotHP
Retrieved Scenario: trackingOnlyEra_Run2_2018
Using Global Tag: GLOBALTAG
Configuring to Write out ALCARECO
entry tobeoverwritten.xyz
Step: RAW2DIGI Spec: 
Step: L1Reco Spec: 
Step: RECO Spec: ['reconstruction_trackingOnly']
Step: ALCAPRODUCER Spec: ['TkAlMinBias']
Step: DQM Spec: ['DQMOfflineTracking']
Step: ENDJOB Spec: 
customising the process with customisePostEra_Run2_2018_express_trackingOnly from Configuration/DataProcessing/RecoTLR
Now do:
cmsRun -e RunExpressProcessingCfg.py
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_15_1_PY312_X_2025-03-19-2300/src/Configuration/DataProcessing/test/RunAlcaSkimming.py --scenario trackingOnlyEra_Run2_2018 --lfn /store/whatever --global-tag GLOBALTAG --skims TkAlMinBias+PromptCalibProdBeamSpotHP
Retrieved Scenario: trackingOnlyEra_Run2_2018
Creating ALCA skimming config with skims:
 => TkAlMinBias
 => PromptCalibProdBeamSpotHP
Step: ALCA Spec: ['PromptCalibProdBeamSpotHP']
Setting numberOfConcurrentLuminosityBlocks=1 because of AlCa sequence PromptCalibProdBeamSpotHP
Step: ALCAOUTPUT Spec: ['TkAlMinBias']
Now do:
cmsRun -e RunAlcaSkimmingCfg.py
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_15_1_PY312_X_2025-03-19-2300/src/Configuration/DataProcessing/test/RunAlcaSkimming.py --scenario trackingOnlyEra_Run2_2018 --lfn /store/whatever --global-tag GLOBALTAG --skims TkAlMinBias+PromptCalibProdBeamSpotHPLowPU
Retrieved Scenario: trackingOnlyEra_Run2_2018
Creating ALCA skimming config with skims:
 => TkAlMinBias
 => PromptCalibProdBeamSpotHPLowPU
Step: ALCA Spec: ['PromptCalibProdBeamSpotHPLowPU']
Setting numberOfConcurrentLuminosityBlocks=1 because of AlCa sequence PromptCalibProdBeamSpotHPLowPU
Step: ALCAOUTPUT Spec: ['TkAlMinBias']
Now do:
cmsRun -e RunAlcaSkimmingCfg.py
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_15_1_PY312_X_2025-03-19-2300/src/Configuration/DataProcessing/test/RunAlcaHarvesting.py --scenario trackingOnlyEra_Run2_2018 --lfn /store/whatever --dataset /A/B/C --global-tag GLOBALTAG --alcapromptdataset=PromptCalibProdBeamSpotHP
Retrieved Scenario: trackingOnlyEra_Run2_2018
Using Global Tag: GLOBALTAG
Dataset: /A/B/C
Step: ALCAHARVEST Spec: ['@PromptCalibProdBeamSpotHP']
Now do:
cmsRun -j FrameworkJobReport.xml RunAlcaHarvestingCfg.py
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_15_1_PY312_X_2025-03-19-2300/src/Configuration/DataProcessing/test/RunAlcaHarvesting.py --scenario trackingOnlyEra_Run2_2018 --lfn /store/whatever --dataset /A/B/C --global-tag GLOBALTAG --alcapromptdataset=PromptCalibProdBeamSpotHPLowPU
Retrieved Scenario: trackingOnlyEra_Run2_2018
Using Global Tag: GLOBALTAG
Dataset: /A/B/C
Step: ALCAHARVEST Spec: ['@PromptCalibProdBeamSpotHPLowPU']
Now do:
cmsRun -j FrameworkJobReport.xml RunAlcaHarvestingCfg.py
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_15_1_PY312_X_2025-03-19-2300/src/Configuration/DataProcessing/test/RunExpressProcessing.py --scenario trackingOnlyEra_Run2_2018_highBetaStar --global-tag GLOBALTAG --lfn /store/whatever --alcarecos=TkAlMinBias+PromptCalibProdBeamSpotHP
Retrieved Scenario: trackingOnlyEra_Run2_2018_highBetaStar
Using Global Tag: GLOBALTAG
Configuring to Write out ALCARECO
entry tobeoverwritten.xyz
Step: RAW2DIGI Spec: 
Step: L1Reco Spec: 
Step: RECO Spec: ['reconstruction_trackingOnly']
Step: ALCAPRODUCER Spec: ['TkAlMinBias']

---> test TestConfigDP_12 had ERRORS
TestTime:3600
^^^^ End Test TestConfigDP_12 ^^^^

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[iarspider]

assign Configuration/DataProcessing

[cmsbuild]

New categories assigned: operations

@antoniovilela,@davidlange6,@fabiocos,@mandrenguyen,@rappoccio you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

In the default IB this test took

---> test TestConfigDP_12 succeeded
TestTime:198
^^^^ End Test TestConfigDP_12 ^^^^

The test itself runs python3 5 x 5 = 25 times, but IIUC the python code being run only generates configuration.

@iarspider Do you know if these timeouts started recently?

Somehow the things being done in this test are slower in python 3.12 than in 3.9. I'm wondering if some "recent" change triggered that, or we just haven't noticed before?

[iarspider]

It is failing since at least February (but it is not known if this test was timing out or failing for some other reason)