Force use of RNTuple in RelVal workflows #47718

Dr15Jones · 2025-03-27T19:46:27Z

PR description:

added obsolete/optional parameters to make RNTuple modules be able to accept the same parameters as their equivalent Pool modules.
Fixed some issues with reading information from the files.
Fixed ProcessHistory handling
Made necessary changes to components called by runTheMatrix.py to accommodate the use of RNTuple.

PR validation:

Did runTheMatrix.py for workflows 1.0, 7.0 and 11834.99.

resolves cms-sw/framework-team#1315, cms-sw/framework-team#1314

cmsbuild · 2025-03-27T19:46:54Z

A new Pull Request was created by @Dr15Jones for CMSSW_15_1_RNTUPLE_X.

It involves the following packages:

Configuration/Applications (operations)
Configuration/PyReleaseValidation (upgrade, pdmv)
FWIO/RNTuple (core)

@AdrianoDee, @Dr15Jones, @Moanwar, @antoniovilela, @cmsbuild, @davidlange6, @DickyChant, @fabiocos, @makortel, @mandrenguyen, @miquork, @rappoccio, @smuzaffar, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @slomeo this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2025-03-27T19:46:55Z

cms-bot internal usage

Dr15Jones · 2025-03-27T19:49:16Z

please test

makortel · 2025-03-27T19:51:56Z

Configuration/PyReleaseValidation/python/MatrixReader.py

@@ -209,6 +209,9 @@ def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None):
            wfName = wfInfo[0]
            stepList = wfInfo[1]
            stepOverrides=wfInfo.overrides
+            #force use of RNTuple
+            stepOverrides['--rntuple_out']=''


@smuzaffar We want to pass the --rntuple_out to runTheMatrix.py for the IB and PR tests of the RNTUPLE_X IB. Is it more convenient to do it in the runTheMatrix code (in the RNTUPLE_X branch) as here, or via the cms-bot?

For this PR, that is not needed as the above line adds --rntuple_out to ALL workflows. In fact, I didn't add an option to runTheMatrix.py to use such a command line option.

@makortel , it will be easier if we do it via runTheMatrix itself. This way it will be enabled for IB and PR tests. May be add it conditionally e.g if 'RNTUPLE_X' in os.environ['CMSSW_VERSION'] then add it?

May be add it conditionally e.g if 'RNTUPLE_X' in os.environ['CMSSW_VERSION'] then add it?

Some kind of conditional will be needed if this logic would enter the master branch (or "eventually" and "when"). This PR targets the RNTUPLE_X branch, so technically unconditional would be ok. Generally I'd like to keep the runTheMatrix and cmsDriver code as close as possible between master and RNTUPLE_X.

I agree that keeping them close is important. That is why that change is in its own commit.

FWIO/RNTuple/plugins/RNTupleSource.cc

cmsbuild · 2025-03-27T20:37:41Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2d6247/45253/summary.html
COMMIT: 29bc7f7
CMSSW: CMSSW_15_1_RNTUPLE_X_2025-03-26-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47718/45253/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@smuzaffar [GCC13] Fix array-bound issue for JetTaggerTableProducer #47710

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2d6247/45253/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2d6247/45253/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test TestConfigurationApplicationsConfigBuilder had ERRORS

RelVals

25202.0

----- Begin Fatal Exception 27-Mar-2025 21:18:39 CET-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling File::sysopen()
Exception Message:
Failed to open the file 'step3_inMINIAODSIM.root'
   Additional Info:
      [a] Input file file:step3_inMINIAODSIM.root could not be opened.
      [b] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

5.1

----- Begin Fatal Exception 27-Mar-2025 21:05:30 CET-----------------------
An exception of category 'StdException' occurred while
   [0] while setting up field SimTrackToTPMap_simHitTPAssocProducer_simTrackToTP_RECO.
Exception Message:
A std::exception was thrown.
the type list for std::unordered_map must have exactly two elements (unchecked RResult access!) (unchecked RResult access!) (unchecked RResult access!)
At:
  ROOT::Experimental::RFieldBase::Create(const std::string&, const std::string&, const std::string&, bool)::<lambda(const std::string&)> [/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/lcg/root/6.35.1-cc9e1c987ea2cc73181ea1f469f472c9/root-6.35.1/tree/ntuple/v7/src/RFieldBase.cxx:308]
  static ROOT::RResult<std::unique_ptr<ROOT::Experimental::RFieldBase> > ROOT::Experimental::RFieldBase::Create(const std::string&, const std::string&, const std::string&, bool) [/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/lcg/root/6.35.1-cc9e1c987ea2cc73181ea1f469f472c9/root-6.35.1/tree/ntuple/v7/src/RFieldBase.cxx:483]
  static ROOT::RResult<std::unique_ptr<ROOT::Experimental::RFieldBase> > ROOT::Experimental::RFieldBase::Create(const std::string&, const std::string&) [/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/lcg/root/6.35.1-cc9e1c987ea2cc73181ea1f469f472c9/root-6.35.1/tree/ntuple/v7/src/RFieldBase.cxx:271]
----- End Fatal Exception -------------------------------------------------

135.4

----- Begin Fatal Exception 27-Mar-2025 21:05:30 CET-----------------------
An exception of category 'StdException' occurred while
   [0] while setting up field SimTrackToTPMap_simHitTPAssocProducer_simTrackToTP_RECO.
Exception Message:
A std::exception was thrown.
the type list for std::unordered_map must have exactly two elements (unchecked RResult access!) (unchecked RResult access!) (unchecked RResult access!)
At:
  ROOT::Experimental::RFieldBase::Create(const std::string&, const std::string&, const std::string&, bool)::<lambda(const std::string&)> [/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/lcg/root/6.35.1-cc9e1c987ea2cc73181ea1f469f472c9/root-6.35.1/tree/ntuple/v7/src/RFieldBase.cxx:308]
  static ROOT::RResult<std::unique_ptr<ROOT::Experimental::RFieldBase> > ROOT::Experimental::RFieldBase::Create(const std::string&, const std::string&, const std::string&, bool) [/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/lcg/root/6.35.1-cc9e1c987ea2cc73181ea1f469f472c9/root-6.35.1/tree/ntuple/v7/src/RFieldBase.cxx:483]
  static ROOT::RResult<std::unique_ptr<ROOT::Experimental::RFieldBase> > ROOT::Experimental::RFieldBase::Create(const std::string&, const std::string&) [/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc12/lcg/root/6.35.1-cc9e1c987ea2cc73181ea1f469f472c9/root-6.35.1/tree/ntuple/v7/src/RFieldBase.cxx:271]
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

makortel · 2025-03-28T14:34:45Z

@smuzaffar Could we actually enable the MaxMemoryPreload (--maxmem_profile argument to cmsDriver.py) for the PRs tested with the RNTUPLE_X IB?

cmsbuild · 2025-03-28T20:37:04Z

Pull request #47718 was updated. @AdrianoDee, @Dr15Jones, @Moanwar, @antoniovilela, @cmsbuild, @davidlange6, @DickyChant, @fabiocos, @makortel, @mandrenguyen, @miquork, @rappoccio, @smuzaffar, @srimanob, @subirsarkar can you please check and sign again.

Dr15Jones · 2025-03-28T20:39:57Z

please test

cmsbuild · 2025-03-28T21:28:37Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2d6247/45276/summary.html
COMMIT: 341d82a
CMSSW: CMSSW_15_1_RNTUPLE_X_2025-03-27-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47718/45276/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

25202.0

----- Begin Fatal Exception 28-Mar-2025 22:06:08 CET-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling File::sysopen()
Exception Message:
Failed to open the file 'step3_inMINIAODSIM.root'
   Additional Info:
      [a] Input file file:step3_inMINIAODSIM.root could not be opened.
      [b] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

1330.0

----- Begin Fatal Exception 28-Mar-2025 21:59:53 CET-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling File::sysopen()
Exception Message:
Failed to open the file 'step3_inMINIAODSIM.root'
   Additional Info:
      [a] Input file file:step3_inMINIAODSIM.root could not be opened.
      [b] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

1000.01000.0_RunMinBias2011A/step3_RunMinBias2011A.log

Expand to see more relval errors ...

smuzaffar · 2025-04-01T08:30:47Z

test parameters:

workflow_command_option = --maxmem_profile

smuzaffar · 2025-04-01T08:31:17Z

please test

This information is now in classes_def.xml.

cmsbuild · 2025-04-14T14:41:18Z

Pull request #47718 was updated. @AdrianoDee, @Dr15Jones, @Moanwar, @cmsbuild, @DickyChant, @makortel, @miquork, @smuzaffar, @srimanob, @subirsarkar can you please check and sign again.

Dr15Jones · 2025-04-14T14:44:25Z

please test

cmsbuild · 2025-04-14T16:40:38Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2d6247/45542/summary.html
COMMIT: ffce48b
CMSSW: CMSSW_15_1_RNTUPLE_X_2025-04-13-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47718/45542/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@mmusich Fix hltInfo utility script and add a unit test #47860

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2d6247/45542/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2d6247/45542/git-merge-result

Comparison Summary

Summary:

You potentially removed 4935 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3915905
DQMHistoTests: Total failures: 683
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3915202
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -3409.04 KiB( 49 files compared)
DQMHistoSizes: changed ( 1001.0 ): -3409.040 KiB AlCaReco/SiStripHitEfficiency
Checked 215 log files, 8 edm output root files, 50 DQM output files
TriggerResults: no differences found

makortel · 2025-04-14T20:17:54Z

+core

Matrix test outputs showed .rntpl files.

makortel · 2025-04-14T20:18:37Z

@cms-sw/pdmv-l2 @cms-sw/upgrade-l2 Please note this PR is for RNTUPLE_X branch, and therefore will not impact any other IB flavor.

AdrianoDee · 2025-04-14T20:35:11Z

+pdmv

Moanwar · 2025-04-15T08:14:19Z

+Upgrade

cmsbuild · 2025-04-15T08:14:44Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_15_1_RNTUPLE_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_15_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @rappoccio, @mandrenguyen, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2025-04-15T11:17:20Z

+1

cmsbuild added core-pending operations-pending pending-signatures tests-pending orp-pending pdmv-pending upgrade-pending labels Mar 27, 2025

cmsbuild added tests-started and removed tests-pending labels Mar 27, 2025

makortel reviewed Mar 27, 2025

View reviewed changes

FWIO/RNTuple/plugins/RNTupleSource.cc Outdated Show resolved Hide resolved

FWIO/RNTuple/plugins/RNTupleSource.cc Show resolved Hide resolved

FWIO/RNTuple/plugins/RNTupleSource.cc Show resolved Hide resolved

cmsbuild added tests-rejected and removed tests-started labels Mar 27, 2025

cmsbuild added tests-pending and removed tests-rejected labels Mar 28, 2025

cmsbuild added tests-started and removed tests-pending labels Mar 28, 2025

cmsbuild added tests-rejected and removed tests-started labels Mar 28, 2025

cmsbuild removed the tests-rejected label Apr 1, 2025

No longer need to specify which data products use streamers

ffce48b

This information is now in classes_def.xml.

Dr15Jones force-pushed the runTheMatrix_rntuple branch from 2f8b5bd to ffce48b Compare April 14, 2025 14:40

cmsbuild added tests-pending and removed tests-approved labels Apr 14, 2025

cmsbuild added tests-started and removed tests-pending labels Apr 14, 2025

cmsbuild added tests-approved and removed tests-started labels Apr 14, 2025

cmsbuild added core-approved and removed core-pending labels Apr 14, 2025

cmsbuild added pdmv-approved and removed pdmv-pending labels Apr 14, 2025

cmsbuild added fully-signed upgrade-approved and removed pending-signatures upgrade-pending labels Apr 15, 2025

cmsbuild added orp-approved and removed orp-pending labels Apr 15, 2025

cmsbuild merged commit 9bae691 into cms-sw:CMSSW_15_1_RNTUPLE_X Apr 15, 2025
10 checks passed

This was referenced Apr 15, 2025

Make RNTupleOutputModule to allow some configuration parameters of PoolOutputModule cms-sw/framework-team#1315

Closed

Add skip events functionality to RNTupleSource cms-sw/framework-team#1322

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Force use of RNTuple in RelVal workflows #47718

Force use of RNTuple in RelVal workflows #47718

Dr15Jones commented Mar 27, 2025 •

edited

Loading

cmsbuild commented Mar 27, 2025

cmsbuild commented Mar 27, 2025 •

edited

Loading

Dr15Jones commented Mar 27, 2025

makortel Mar 27, 2025

Dr15Jones Mar 27, 2025 •

edited

Loading

smuzaffar Mar 27, 2025 •

edited

Loading

makortel Mar 27, 2025

Dr15Jones Mar 27, 2025

cmsbuild commented Mar 27, 2025

makortel commented Mar 28, 2025

cmsbuild commented Mar 28, 2025

Dr15Jones commented Mar 28, 2025

cmsbuild commented Mar 28, 2025

smuzaffar commented Apr 1, 2025

smuzaffar commented Apr 1, 2025

cmsbuild commented Apr 14, 2025

Dr15Jones commented Apr 14, 2025

cmsbuild commented Apr 14, 2025

makortel commented Apr 14, 2025

makortel commented Apr 14, 2025

AdrianoDee commented Apr 14, 2025

Moanwar commented Apr 15, 2025

cmsbuild commented Apr 15, 2025

mandrenguyen commented Apr 15, 2025

Force use of RNTuple in RelVal workflows #47718

Force use of RNTuple in RelVal workflows #47718

Conversation

Dr15Jones commented Mar 27, 2025 • edited Loading

PR description:

PR validation:

cmsbuild commented Mar 27, 2025

cmsbuild commented Mar 27, 2025 • edited Loading

Dr15Jones commented Mar 27, 2025

makortel Mar 27, 2025

Choose a reason for hiding this comment

Dr15Jones Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

smuzaffar Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

makortel Mar 27, 2025

Choose a reason for hiding this comment

Dr15Jones Mar 27, 2025

Choose a reason for hiding this comment

cmsbuild commented Mar 27, 2025

Unit Tests

RelVals

makortel commented Mar 28, 2025

cmsbuild commented Mar 28, 2025

Dr15Jones commented Mar 28, 2025

cmsbuild commented Mar 28, 2025

RelVals

smuzaffar commented Apr 1, 2025

smuzaffar commented Apr 1, 2025

cmsbuild commented Apr 14, 2025

Dr15Jones commented Apr 14, 2025

cmsbuild commented Apr 14, 2025

Comparison Summary

makortel commented Apr 14, 2025

makortel commented Apr 14, 2025

AdrianoDee commented Apr 14, 2025

Moanwar commented Apr 15, 2025

cmsbuild commented Apr 15, 2025

mandrenguyen commented Apr 15, 2025

Dr15Jones commented Mar 27, 2025 •

edited

Loading

cmsbuild commented Mar 27, 2025 •

edited

Loading

Dr15Jones Mar 27, 2025 •

edited

Loading

smuzaffar Mar 27, 2025 •

edited

Loading