Implement a `GenericCloner` test module #47504

fwyzard · 2025-03-04T20:52:59Z

PR description:

Specialise Handle and OrphanHandle for WrapperBase: this lets users produce and consume collections via their wrapper, using their run time type information (e.g. type name, type id, ...) instead of the compile time types. Implement integration test for this specialisations.

Introduce a MemcpyTraits structure, that should be specialised for types that can be safely memcpyed, without requiring a full ROOT streamer de/serialisation.
Specialise the MemcpyTraits for arithmetic types and vectors of arithmetic types, and all PortableObject and PortableCollection types.

Implement edmtest::GenericCloner: this EDProducer will consume and clone all the event products declared by its configuration; a direct memcpy will be used for types that support the MemcpyTraits interface; the ROOT dictionaries will be use to serialise and deserialise the other tyepes.

PR validation:

Unit tests pass.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

May be backported to 15.0.x as part of the MPI work.

This EDProducer will clone all the event products declared by its configuration, using their ROOT dictionaries.

cmsbuild · 2025-03-04T20:53:25Z

cms-bot internal usage

fwyzard · 2025-03-04T20:53:56Z

@makortel what do you think of this approach ?

I don't mind moving stuff around, of course: consider this more of a prototype.

fwyzard · 2025-03-04T20:54:04Z

please test

cmsbuild · 2025-03-04T20:55:03Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/43956

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/43956/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/43956/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2025-03-04T20:58:53Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/43957

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821

cmsbuild · 2025-03-04T20:59:13Z

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

DataFormats/Common (core)
DataFormats/Portable (heterogeneous)
FWCore/Framework (core)
FWCore/Integration (core)
FWCore/TestModules (core)

@Dr15Jones, @cmsbuild, @fwyzard, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@makortel, @missirol, @mmusich, @rovere, @wddgit this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

fwyzard · 2025-03-04T20:59:59Z

FWCore/Framework/interface/WrapperBaseHandle.h

Most of this could actually go directly under DataFormats/Common, only the specialisation of the Event methods need to go in FWCore/Framework.
Let me know if I should split and rearrange this.

fwyzard · 2025-03-04T21:00:30Z

FWCore/Framework/interface/WrapperBaseOrphanHandle.h

Most of this could actually go directly under DataFormats/Common, only the specialisation of the Event methods need to go in FWCore/Framework.
Let me know if I should split and rearrange this.

fwyzard · 2025-03-04T21:02:24Z

DataFormats/Portable/interface/PortableCollection.h

+      return {object->metadata().size()};
+    }
+
+    static std::vector<std::pair<void*, std::size_t>> regions(PortableDeviceCollection<T, TDev>& object) {


To make the specialisation for PortableDeviceCollection work, some kind of pass-through specialisation for edm::DeviceProduct is still needed.

cmsbuild · 2025-03-10T21:36:05Z

Pull request #47504 was updated. @Dr15Jones, @cmsbuild, @fwyzard, @makortel, @smuzaffar can you please check and sign again.

cmsbuild · 2025-03-11T10:49:46Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/44032

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): New constructor for the SoAView and aggregate method for Collections #47306
- File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821, Remove FinalPath from the framework #47546
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/44032/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/44032/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

…ostCollection

fwyzard · 2025-03-11T11:07:11Z

please test

fwyzard · 2025-03-11T11:07:51Z

@makortel @Dr15Jones this is now close enough to what I think I need, that I'd be happy for a first review.

cmsbuild · 2025-03-11T11:08:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/44033

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): New constructor for the SoAView and aggregate method for Collections #47306
- File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821, Remove FinalPath from the framework #47546
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821

cmsbuild · 2025-03-11T11:08:47Z

Pull request #47504 was updated. @Dr15Jones, @fwyzard, @makortel, @smuzaffar can you please check and sign again.

cmsbuild · 2025-03-11T16:01:38Z

+1

Size: This PR adds an extra 44KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7fe69b/44911/summary.html
COMMIT: 4988fc2
CMSSW: CMSSW_15_1_X_2025-03-10-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47504/44911/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 4 lines from the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 57
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920223
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

fwyzard added 3 commits March 4, 2025 13:49

Specialize Handle and OrphanHandle for WrapperBase

acf9d6a

Implement integration tests for WrapperBase specialisations

4ba7794

Implement edmtest::GenericCloner

31d3ae7

This EDProducer will clone all the event products declared by its configuration, using their ROOT dictionaries.

cmsbuild added this to the CMSSW_15_1_X milestone Mar 4, 2025

cmsbuild added core-pending pending-signatures tests-pending orp-pending code-checks-pending heterogeneous-pending labels Mar 4, 2025

cmsbuild added tests-started and removed tests-pending labels Mar 4, 2025

cmsbuild added code-checks-rejected and removed code-checks-pending labels Mar 4, 2025

fwyzard force-pushed the implement_generic_product_tools_v2 branch from c559d9f to e5278a6 Compare March 4, 2025 20:56

cmsbuild added tests-pending code-checks-pending and removed tests-started code-checks-rejected labels Mar 4, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 4, 2025

fwyzard commented Mar 4, 2025

View reviewed changes

cmsbuild added the code-checks-approved label Mar 10, 2025

fwyzard force-pushed the implement_generic_product_tools_v2 branch from 4b85303 to d32b15b Compare March 11, 2025 10:47

cmsbuild added code-checks-pending and removed code-checks-approved labels Mar 11, 2025

cmsbuild added code-checks-rejected and removed code-checks-pending labels Mar 11, 2025

fwyzard added 3 commits March 11, 2025 12:06

Introduce the TrivialCopyTraits

9e2a878

Specialise the TrivialCopyTraits for PortableHostObject and PortableH…

a6452be

…ostCollection

Optimize GenericCloner using the TrivialCopyTraits

4988fc2

fwyzard force-pushed the implement_generic_product_tools_v2 branch from d32b15b to 4988fc2 Compare March 11, 2025 11:06

cmsbuild added code-checks-pending tests-started and removed code-checks-rejected tests-pending labels Mar 11, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 11, 2025

cmsbuild added tests-approved and removed tests-started labels Mar 11, 2025

This was referenced Mar 11, 2025

Remove configuration support for SubProcess and its tests #47565

Merged

Implement the CheckClassInfo module #47567

Draft

Create new main ProductRegistry if input products change #47584

Merged

This was referenced Mar 27, 2025

Move RandomNumberGenerator service base class to FWCore/AbstractServices #47719

Merged

New constructor for the SoAView and aggregate method for Collections #47306

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a `GenericCloner` test module #47504

Implement a `GenericCloner` test module #47504

fwyzard commented Mar 4, 2025

cmsbuild commented Mar 4, 2025 •

edited

Loading

fwyzard commented Mar 4, 2025

fwyzard commented Mar 4, 2025

cmsbuild commented Mar 4, 2025

cmsbuild commented Mar 4, 2025

cmsbuild commented Mar 4, 2025

fwyzard Mar 4, 2025 •

edited

Loading

fwyzard Mar 4, 2025

fwyzard Mar 4, 2025

cmsbuild commented Mar 10, 2025

cmsbuild commented Mar 11, 2025

fwyzard commented Mar 11, 2025

fwyzard commented Mar 11, 2025

cmsbuild commented Mar 11, 2025

cmsbuild commented Mar 11, 2025

cmsbuild commented Mar 11, 2025

Implement a GenericCloner test module #47504

Are you sure you want to change the base?

Implement a GenericCloner test module #47504

Conversation

fwyzard commented Mar 4, 2025

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Mar 4, 2025 • edited Loading

fwyzard commented Mar 4, 2025

fwyzard commented Mar 4, 2025

cmsbuild commented Mar 4, 2025

cmsbuild commented Mar 4, 2025

cmsbuild commented Mar 4, 2025

fwyzard Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

fwyzard Mar 4, 2025

Choose a reason for hiding this comment

fwyzard Mar 4, 2025

Choose a reason for hiding this comment

cmsbuild commented Mar 10, 2025

cmsbuild commented Mar 11, 2025

fwyzard commented Mar 11, 2025

fwyzard commented Mar 11, 2025

cmsbuild commented Mar 11, 2025

cmsbuild commented Mar 11, 2025

cmsbuild commented Mar 11, 2025

Comparison Summary

Implement a `GenericCloner` test module #47504

Implement a `GenericCloner` test module #47504

cmsbuild commented Mar 4, 2025 •

edited

Loading

fwyzard Mar 4, 2025 •

edited

Loading