Add support for setting process.options from the command line

[fwyzard]

PR description:

Add support for setting process.options from the command line:

cmsRun -o numberOfThreads=32 -o numberOfStreams=24 -o 'accelerators="gpu-*"' config.py

is equivalent to setting

process.options.numberOfThreads = 32
process.options.numberOfStreams = 24
process.options.accelerators = [ "gpu-*" ]

in config.py.

PR validation:

New and existing unit tests pass.

[cmsbuild]

cms-bot internal usage

[fwyzard]

@cms-sw/core-l2 here is a first implementation, including some new tests.

Let me know if you have comments -- for example, if I should use a different approach to the parsing, or put code in different place, or add more tests, etc.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50585/48766

• There are other open Pull requests which might conflict with changes you have proposed:
  • File FWCore/ParameterSet/src/Entry.cc modified in PR(s): Introduce vFloat and float types in ParameterSets #45216

[cmsbuild]

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

• FWCore/Framework (core)
• FWCore/ParameterSet (core)
• FWCore/Utilities (core)

@Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@felicepantaleo, @makortel, @wddgit this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[fwyzard]

I'm probably not going to implement the generic -p option, but I may add a special option for the number of events.

[fwyzard]

please test

[cmsbuild]

+1

Size: This PR adds an extra 72KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9299ca/52325/summary.html
COMMIT: 0a1decf
CMSSW: CMSSW_16_1_X_2026-03-28-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50585/52325/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4186890
• DQMHistoTests: Total failures: 4
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4186866
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 198 edm output root files, 53 DQM output files
• TriggerResults: no differences found

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 8 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

• Error: Workflow 4.53_RunPhoton2012B step3 max memory diff 104.7 exceeds +/- 90.0 MiB
• Error: Workflow 136.874_RunEGamma2018C step3 max memory diff 109.6 exceeds +/- 90.0 MiB
• Error: Workflow 2024.0050001_RunBTagMu2024G_10k step3 max memory diff 119.2 exceeds +/- 90.0 MiB
• Error: Workflow 2024.0060001_RunMuon02024H_10k step3 max memory diff 105.0 exceeds +/- 90.0 MiB
• Error: Workflow 2025.0010001_RunJetMET02025C_10k step2 max memory diff 93.0 exceeds +/- 90.0 MiB
• Error: Workflow 10224.0_TTbar_13+2017PU step3 max memory diff 108.3 exceeds +/- 90.0 MiB
• Error: Workflow 25202.0_TTbar_13 step3 max memory diff 117.7 exceeds +/- 90.0 MiB
• Error: Workflow 34634.999_TTbar_14TeV+Run4D121PU_PMXS1S2PR step4 max memory diff 109.8 exceeds +/- 90.0 MiB

[Dr15Jones]

As a simple test, I found that the following script does exactly the same job

from argparse import ArgumentParser
import importlib

parser = ArgumentParser(description='add options')
parser.add_argument("-o","--options", help="add to process.options", action = "append", type = str)
parser.add_argument("cfg", help="name of config file")
args = parser.parse_args()

cfg = importlib.__import__(args.cfg[:-3])

process = cfg.process

if args.options:
    for opt in args.options:
        exec("process.options."+opt, globals(), locals())

This can be run by doing

cmsRun command_line_cfg.py -o 'numberOfThreads=4' test_cfg.py

[makortel]

On a first, cursory look the code is somewhat complicated (with additional parsers etc). How about just appending the process.options.<name>=<value> to the string representation of the python configuration before the configuration is passed to the python interpreter?

[fwyzard]

On a first, cursory look the code is somewhat complicated (with additional parsers etc).

It is :-)

How about just appending the process.options.<name>=<value> to the string representation of the python configuration before the configuration is passed to the python interpreter?

I can try that.

From a first look, it will require some changes to PyBind11ProcessDesc and on the way there, but shouldn't be too hard.

One possible concern is that with this approach it may be harder to diagnose invalid options 🤔