Skip to content

SONIC configuration updates #47748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 10, 2025
Merged

Conversation

kpedro88
Copy link
Contributor

@kpedro88 kpedro88 commented Apr 1, 2025

PR description:

This PR makes some improvements to configuration details for SONIC:

  1. Extend SONIC workflows to all of Run 3 (previously restricted to 2022)
  2. Migrate from VarParsing to argparse in SonicCore unit test scripts
  3. Introduce customization functions for SonicTriton tests. These can be used for relval submission or private testing setups.
  4. Print more information for a specific error message

PR validation:

Ran unit tests and matrix workflows successfully with expected output.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, not intended to be backported.

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 1, 2025

cms-bot internal usage

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 1, 2025

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47748/44330

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 1, 2025

A new Pull Request was created by @kpedro88 for master.

It involves the following packages:

  • Configuration/PyReleaseValidation (pdmv, upgrade)
  • HeterogeneousCore/SonicCore (heterogeneous)
  • HeterogeneousCore/SonicTriton (heterogeneous)

@AdrianoDee, @Moanwar, @cmsbuild, @DickyChant, @fwyzard, @makortel, @miquork, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @riga, @rovere, @slomeo this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@kpedro88
Copy link
Contributor Author

kpedro88 commented Apr 1, 2025

test parameters:
workflow = 11634.9001,12434.9001
relvals_opt = --what cleanedupgrade,standard,highstats,pileup,generator,extendedgen,production,identity,ged,machine,premix,nano,gpu,2017,2026

@kpedro88
Copy link
Contributor Author

kpedro88 commented Apr 1, 2025

please test

Comment on lines 147 to 148
if not hasattr(process,'MessageLogger'):
process.load('FWCore/MessageService/MessageLogger_cfi')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to note that this piece is needed only if the user has explicitly deleted process.MessageLogger. MessageLogger has been part of cms.Process for quite some time already.

If kept, while this works, I'd prefer

Suggested change
if not hasattr(process,'MessageLogger'):
process.load('FWCore/MessageService/MessageLogger_cfi')
if not hasattr(process,'MessageLogger'):
process.load('FWCore.MessageService.MessageLogger_cfi')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I actually did not know it was included in the initialization of cms.Process. Then I think this condition is not necessary.

Comment on lines 151 to 155
setattr(process.MessageLogger.cerr, msg,
cms.untracked.PSet(
limit = cms.untracked.int32(10000000),
)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be

Suggested change
setattr(process.MessageLogger.cerr, msg,
cms.untracked.PSet(
limit = cms.untracked.int32(10000000),
)
)
setattr(process.MessageLogger.cerr, msg,
dict(
limit = 10000000,
)
)

Comment on lines 79 to 87
cms.PSet(
name = cms.untracked.string(options.serverName),
address = cms.untracked.string(options.address),
port = cms.untracked.uint32(options.port),
useSsl = cms.untracked.bool(options.ssl),
rootCertificates = cms.untracked.string(""),
privateKey = cms.untracked.string(""),
certificateChain = cms.untracked.string(""),
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be

Suggested change
cms.PSet(
name = cms.untracked.string(options.serverName),
address = cms.untracked.string(options.address),
port = cms.untracked.uint32(options.port),
useSsl = cms.untracked.bool(options.ssl),
rootCertificates = cms.untracked.string(""),
privateKey = cms.untracked.string(""),
certificateChain = cms.untracked.string(""),
)
dict(
name = options.serverName,
address = options.address,
port = options.port,
useSsl = options.ssl,
)

Comment on lines +96 to +102
return dict(
compression = cms.untracked.string(options.compression),
useSharedMemory = cms.untracked.bool(not options.noShm),
timeout = cms.untracked.uint32(options.timeout),
timeoutUnit = cms.untracked.string(options.timeoutUnit),
allowedTries = cms.untracked.uint32(options.tries),
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could probably be

Suggested change
return dict(
compression = cms.untracked.string(options.compression),
useSharedMemory = cms.untracked.bool(not options.noShm),
timeout = cms.untracked.uint32(options.timeout),
timeoutUnit = cms.untracked.string(options.timeoutUnit),
allowedTries = cms.untracked.uint32(options.tries),
)
return dict(
compression = options.compression,
useSharedMemory = not options.noShm,
timeout = options.timeout,
timeoutUnit = options.timeoutUnit,
allowedTries = options.tries,
)

if the EDModule object being modified is expected to originate from fillDescriptions().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, that is not always the case. I had originally written it this way and then hit the "parameter does not already exist" error. I do not know another way around that besides always specifying the types.

Comment on lines 110 to 112
modules.update(process._Process__producers)
modules.update(process._Process__filters)
modules.update(process._Process__analyzers)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use e.g.

Suggested change
modules.update(process._Process__producers)
modules.update(process._Process__filters)
modules.update(process._Process__analyzers)
modules.update(process.producers_())
modules.update(process.filters_())
modules.update(process.analyzers_())

instead of accessing "private" members of the Process.

def applyClientOptions(client, options):
return configureClient(client, **getClientOptions(options))

def configureModules(process, modules=None, returnConfigured=False, **kwargs):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to make sure, if user specifies the modules argument, they are expected to provide it as a dictionary of those along

labels = # list of strings
modules = {label: getattr(process, label) for label in labels}

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is mainly there to allow users to apply a configuration to a selected subset of modules (based on whatever criteria they might have). e.g. they could populate the dictionary by filtering out unwanted entries from process.producers_().

@kpedro88
Copy link
Contributor Author

kpedro88 commented Apr 1, 2025

@makortel implemented all but #47748 (comment) (as explained in comment there)

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 1, 2025

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47748/44332

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 1, 2025

Pull request #47748 was updated. @AdrianoDee, @Moanwar, @cmsbuild, @DickyChant, @fwyzard, @makortel, @miquork, @srimanob, @subirsarkar can you please check and sign again.

@kpedro88
Copy link
Contributor Author

kpedro88 commented Apr 1, 2025

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 1, 2025

+1

Size: This PR adds an extra 52KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1be31a/45323/summary.html
COMMIT: 8ed3de1
CMSSW: CMSSW_15_1_X_2025-04-01-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47748/45323/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 8 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 3913985
  • DQMHistoTests: Total failures: 111
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3913854
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 215 log files, 184 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@makortel
Copy link
Contributor

makortel commented Apr 2, 2025

Comparison differences are related to #47071

@makortel
Copy link
Contributor

makortel commented Apr 2, 2025

+heterogeneous

@kpedro88
Copy link
Contributor Author

kpedro88 commented Apr 7, 2025

@cms-sw/upgrade-l2 @cms-sw/pdmv-l2 please check and sign (just adding some Run 3 special workflows)

@Moanwar
Copy link
Contributor

Moanwar commented Apr 7, 2025

+Upgrade

@AdrianoDee
Copy link
Contributor

+pdmv

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @mandrenguyen, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@mandrenguyen
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 41fb4ba into cms-sw:master Apr 10, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants