2026 RelVal Workflows #49419

AdrianoDee · 2025-11-19T12:40:18Z

PR description:

The 2026 version of #46445.

This PR proposes the addition of 2026 wfs to the matrix. For the moment, they are a copy of what we run currently for 2025 with the idea that once 2026 parameters (conditions, HLT menu) come, we will need to fill the dummy ones (keeping them in sync with each other). For the moment:

@relval2026 is pointing toGRun (@cms-sw/hlt-l2, let me know if you prefer something different there);
auto:phase1_2026_* symbolic GTs are copies of 2025 ones (@cms-sw/alca-l2 let me know what you think about this, I'll updated them following your suggestion);

The era is Era_2026 that, for now, is just a copy of Era_2025 (the same for FastSim, just fyi @cms-sw/simulation-l2). TTbar 2026 noPU/PU wfs are added to the limited matrix to be tested with PRs. The standard PU input for 2026 wfs is the same as 2025 for now. I took the chance to update also the GPU workflows to 2026 (@cms-sw/heterogeneous-l2) so that we avoid running them with the fake menu when 2025 wfs will be switching to that (as happened last year).

PR validation:

New wfs run.

AdrianoDee · 2025-11-19T12:40:32Z

enable gpu

cmsbuild · 2025-11-19T12:40:42Z

cms-bot internal usage

AdrianoDee · 2025-11-19T12:40:52Z

please test
(the new tests are in the limited matrix now)

cmsbuild · 2025-11-19T12:42:45Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49419/46876

Found files with invalid states:
- fatal: bad object 009f087
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Fix FastSim refinement for CHS #49400
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Fix location of alpaka modification for particleFlowClusterHCAL #49295, Update mkFit-related processModifier for LST-seeded phase-2 HLT single tracking iteration #49383, ECAL Phase 2 - Upgrades for Phase 2 development workflow, validation and DQM #49409
- File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): [Do not merge] For cms-bot PR testing #48637, Update mkFit-related processModifier for LST-seeded phase-2 HLT single tracking iteration #49383

cmsbuild · 2025-11-19T12:42:51Z

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

Configuration/AlCa (alca)
Configuration/Eras (operations)
Configuration/HLT (hlt)
Configuration/PyReleaseValidation (pdmv)

@AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @arunhep, @atpathak, @cmsbuild, @davidlange6, @fabiocos, @ftenchini, @mandrenguyen, @miquork, @mmusich, @perrotta can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @mmusich, @rsreds, @slomeo, @tocheng, @yuanchao this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

AdrianoDee · 2025-11-19T12:43:23Z

please test

cmsbuild · 2025-11-19T12:43:48Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49419/46877

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Fix FastSim refinement for CHS #49400
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Fix location of alpaka modification for particleFlowClusterHCAL #49295, Update mkFit-related processModifier for LST-seeded phase-2 HLT single tracking iteration #49383, ECAL Phase 2 - Upgrades for Phase 2 development workflow, validation and DQM #49409
- File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): [Do not merge] For cms-bot PR testing #48637, Update mkFit-related processModifier for LST-seeded phase-2 HLT single tracking iteration #49383

AdrianoDee · 2025-11-19T12:44:54Z

please abort

perrotta · 2025-11-24T11:32:49Z

+alca

mmusich · 2025-11-24T11:44:38Z

Configuration/HLT/python/autoHLT.py

  'relval2022'          : 'Fake2',
  'relval2023'          : 'Fake2',
  'relval2024'          : 'Fake2',
  'relval2025'          : '2025v13',


eventually, now that we have coverage I guess we can demote this to fake (in another PR).

Yes, I'd wait to do that after pre3 is out just to use 2025 for one last round of validations.

mmusich · 2025-11-24T11:44:49Z

+hlt

cmsbuild · 2025-11-24T11:45:13Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

mmusich · 2025-11-24T12:06:48Z

Out of curiosity @cms-sw/dqm-l2 is it expected that in this PR the DQM bin-by-bin comparison is empty: https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-21-1100+fd870d/71938/dqm-histo-comparison-summary.html ?

gabrielmscampos · 2025-11-24T14:30:31Z

Out of curiosity @cms-sw/dqm-l2 is it expected that in this PR the DQM bin-by-bin comparison is empty: cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-21-1100+fd870d/71938/dqm-histo-comparison-summary.html ?

I'd say that is unusual. I was taking a look at signed and merged PRs from last week that target 16_0_X and the cmsdt pages are still available and most of them also are displaying empty DQM bin-by-bin comparison.

On this specific PR, I cannot find the /RelVal_wf25202_0_pr/CMSSW_16_0_X-PRcmssw_49419-XXXXXX/DQMIO dataset in the Development GUI. Maybe the ROOT file was not uploaded at the end of the tests? Can we check Jenkins's logs to check this?

mmusich · 2025-11-24T14:31:35Z

shall we try to test again?

smuzaffar · 2025-11-24T14:38:53Z

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-21-1100+fd870d/71938/dqmBinByBinLog.log shows some root/python errors [a] . Could this be the reason the dqm bin-by-bin comparison were not properly done ? Could it be due to ROOT 6.36 update?

[a]

PR file successfully opened
Traceback (most recent call last):
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 345, in <module>
    create_dif(args.base_file, args.pr_file, args.pr_number, args.test_number, cmssw_version, args.num_processes, args.output_dir)
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 80, in create_dif
    compare(shared_paths, pr_flat_dict, base_flat_dict, paths_to_save_in_pr, paths_to_save_in_base)
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 182, in compare
    pr_array = np.array(pr_item)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 367, in _getitem
    uhi_index = _compute_common_index(self, index)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 223, in _compute_common_index
    return [_compute_uhi_index(self, idx, axis, include_flow_bins) for axis, idx in enumerate(index)]
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 223, in <listcomp>
    return [_compute_uhi_index(self, idx, axis, include_flow_bins) for axis, idx in enumerate(index)]
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 192, in _compute_uhi_index
    return _process_index_for_axis(self, index, axis)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 180, in _process_index_for_axis
    raise IndexError(f"Histogram index {index - 1} out of range for axis {axis}. Valid range: (0,{nbins})")
IndexError: Histogram index 30 out of range for axis 0. Valid range: (0,30)

gabrielmscampos · 2025-11-24T14:46:25Z

cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-21-1100+fd870d/71938/dqmBinByBinLog.log shows some root/python errors [a] . Could this be the reason the dqm bin-by-bin comparison were not properly done ? Could it be due to ROOT 6.36 update?

That is right, at the end of the log we can see that there were no files to be uploaded as well.

shall we try to test again?

I don't think testing again will solve the issue.

mmusich · 2025-11-24T14:52:36Z

Could this be the reason the dqm bin-by-bin comparison were not properly done ? Could it be due to ROOT 6.36 update?

so just to be clear, the issue is not coming specifically from this PR, but it's also present in all other recent PRs?

gabrielmscampos · 2025-11-24T15:10:17Z

so just to be clear, the issue is not coming specifically from this PR, but it's also present in all other recent PRs?

I don't think this is related to thir PR. Two other cases that this happened, before this PR:

Looks like this is not happening on 15_1_X.

smuzaffar · 2025-11-24T15:13:01Z

this might be related to ROOT 6.36 change root-project/root@ff65e3b which is in 16.0.X IBs .

smuzaffar · 2025-11-24T15:17:37Z

Do we need to updates in https://github.com/cms-sw/cmssw/blob/master/DQMServices/FileIO/scripts/compareHistograms.py for ROOT change root-project/root#18735 ?

smuzaffar · 2025-11-24T15:18:14Z

I am opening a new issue to track dqm bin-by-bin failures

smuzaffar · 2025-11-24T15:24:34Z

#49452

mmusich · 2025-11-24T15:52:30Z

I am opening a new issue to track dqm bin-by-bin failures

thanks!

mandrenguyen · 2025-11-24T20:08:08Z

+1

cmsbuild added this to the CMSSW_16_0_X milestone Nov 19, 2025

cmsbuild added hlt-pending alca-pending operations-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending labels Nov 19, 2025

cmsbuild added tests-started and removed tests-pending labels Nov 19, 2025

AdrianoDee force-pushed the relval_2026_wfs branch from f92c857 to 009f087 Compare November 19, 2025 12:42

cmsbuild added tests-pending and removed tests-started labels Nov 19, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 19, 2025

cmsbuild added tests-started and removed tests-pending labels Nov 19, 2025

AdrianoDee force-pushed the relval_2026_wfs branch from 009f087 to 63ca699 Compare November 19, 2025 12:44

cmsbuild added tests-pending and removed tests-started code-checks-approved labels Nov 19, 2025

cmsbuild added the pdmv-approved label Nov 24, 2025

cmsbuild added alca-approved and removed alca-pending labels Nov 24, 2025

mmusich reviewed Nov 24, 2025

View reviewed changes

cmsbuild added hlt-approved fully-signed and removed hlt-pending pending-signatures labels Nov 24, 2025

smuzaffar mentioned this pull request Nov 24, 2025

DQM bin-by-bin test failing since we moved to ROOT 6.3.6 #49452

Closed

cmsbuild added orp-approved and removed orp-pending labels Nov 24, 2025

cmsbuild merged commit b610721 into cms-sw:master Nov 24, 2025
25 checks passed

AdrianoDee deleted the relval_2026_wfs branch November 24, 2025 20:46

akritkbehera mentioned this pull request Nov 25, 2025

[GPU] RelVals 18XXX.4XX failing on AMD_MI300X with SIGABRT #49464

Open

2026 RelVal Workflows #49419

2026 RelVal Workflows #49419

Uh oh!

Conversation

AdrianoDee commented Nov 19, 2025

PR description:

PR validation:

Uh oh!

AdrianoDee commented Nov 19, 2025

Uh oh!

cmsbuild commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdrianoDee commented Nov 19, 2025

Uh oh!

cmsbuild commented Nov 19, 2025

Uh oh!

cmsbuild commented Nov 19, 2025

Uh oh!

AdrianoDee commented Nov 19, 2025

Uh oh!

cmsbuild commented Nov 19, 2025

Uh oh!

AdrianoDee commented Nov 19, 2025

Uh oh!

perrotta commented Nov 24, 2025

Uh oh!

mmusich Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

AdrianoDee Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

mmusich commented Nov 24, 2025

Uh oh!

cmsbuild commented Nov 24, 2025

Uh oh!

mmusich commented Nov 24, 2025

Uh oh!

gabrielmscampos commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented Nov 24, 2025

Uh oh!

smuzaffar commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gabrielmscampos commented Nov 24, 2025

Uh oh!

mmusich commented Nov 24, 2025

Uh oh!

gabrielmscampos commented Nov 24, 2025

Uh oh!

smuzaffar commented Nov 24, 2025

Uh oh!

smuzaffar commented Nov 24, 2025

Uh oh!

smuzaffar commented Nov 24, 2025

Uh oh!

smuzaffar commented Nov 24, 2025

Uh oh!

mmusich commented Nov 24, 2025

Uh oh!

mandrenguyen commented Nov 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

cmsbuild commented Nov 19, 2025 •

edited

Loading

gabrielmscampos commented Nov 24, 2025 •

edited

Loading

smuzaffar commented Nov 24, 2025 •

edited

Loading