upgrade CMSSW data reading via xroot protocol for packet labeling

[stlammel]

During February O&C Week Marian Babik presented the network packet labeling project.

Could we please upgrade data reading in CMSSW to automatically set flow labels

196656 in case of Production Input
196664 in case of Analysis Input
196700 Secondary Input
196704 Simulation Pileup Input

in case of reading via xroot protocol? For xrootd the flow label can be set via CGI
parameter, i.e. scitag.flow=<scitag_id> or just adding "?scitag.flow=<scitag_id>" to
the URL path. (I believe Tier-0 already adds a "?eos.app=cmst0" to the path right
now.)

I would make 196664 and 196700 the default for primary/secondary input and
provide a mechanism for WM to override the default (always for primary input
and in case of pileup library as secondary input).

Thanks,

• Stephan

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @stlammel.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

How urgent is this, i.e. when should these tags be used in production?

What exactly does "Secondary Input" mean? Given the distinction from "Simulation Pileup Input", I'd assume the "two file solution", but then I'm confused of

I would make 196664 and 196700 the default for primary/secondary input and
provide a mechanism for WM to override the default (always for primary input
and in case of pileup library as secondary input).

because the "secondary input" of two-file solution is orthogonal to "pileup input", and theoretically both could be used in the same job.

[makortel]

I suspect we don't have any easy/straightforward way to communicate the nature of "Production Input" / "Secondary Input" / "Simulation Pileup Input" from the PoolSource / EmbeddedRootSource to XrdAdaptor (so that XrdAdaptor would do the modification of URL).

If that is the case, I guess the PoolSource / EmbeddedRootSource would have to add the CGI parameter to the PFN in case of root:// protocol or something. So far we have been able to keep the Sources in general agnostic on the protocols.

[stlammel]

Hallo Matti,
not urgent, packet labeling is an HL-LHC project but of course we would like to have it well before then.
I thought we had the option to read/access missing information from the parent dataset. That's what i thought of secondary input until the pileup library came. I assume the two use a similar "second input". If the two a separate, then both can be set directly and only an override for the primary input is needed.
Yes, i think it could be done after the LFN to PFN translation before/outside of XrdAdaptor via a PFN manipulation, thus me mentioning the Tier-0 query add.
Thanks,
cheers, Stephan

[makortel]

but of course we would like to have it well before then.

What could that mean? End of this year? End of 2026? End of 2027?

I thought we had the option to read/access missing information from the parent dataset.

This corresponds to the "two-file solution" I mentioned above.

I assume the two use a similar "second input". If the two a separate, then both can be set directly and only an override for the primary input is needed.

The technical mechanism is indeed separate. Which is also why "secondary input" is in many occasions rather confusing name :) Note that we envision being able to have even more kinds of inputs.

What if an analysis job uses "Secondary Input"?

[stlammel]

Well, there is no hard need-by date. The earlier the better, the sooner we can see what goes over the network. It would be great if this were available next year.
We could distinguish production and analysis secondary input further but i assume it's small compared to primary input (except pileup) thus thought one activity for both production and analysis secondary input, and separating pileup was more important. If you feel it's important, we could add an activity.

• Stephan

[makortel]

Well, there is no hard need-by date. The earlier the better, the sooner we can see what goes over the network. It would be great if this were available next year.

Ok, thanks. I interpret this as "would be great to have in CMSSW_16_0_0" (the data taking release of next year).

We could distinguish production and analysis secondary input further but i assume it's small compared to primary input (except pileup) thus thought one activity for both production and analysis secondary input, and separating pileup was more important. If you feel it's important, we could add an activity.

I can't tell if that would be important (I feel this classification should primarily come from elsewhere).

Few questions more

Does "Simulation Pileup Input" cover both the MinBias files read by classical mixing (and premixing stage1 that creates the premix library), and the Premixed files read by premixing overlay?

What about other users of EmbeddedRootSource? Beyond MixingModules it is presently used in SiStripSpyEventMatcherModule, but I guess it is run in grid, and I don't know if it would read over xrootd.

Should we classify all present and future "secondary files" (via PoolSource's secondaryFileNames, via EmbeddedRootSource, via something we add in the future) that are not about pileup as "Secondary Input" ?

[stlammel]

Hallo Matti,
so, i would make all "Secondary Input"/?use of EmbeddedRootSource? report as activity 196700 except for the Premixed Library files (activity 196704). So, classical mixing would be 196700 in this arrangement. (If the default activity 196700 can be overidden (via config or env variable) then we could separate the min bias dataset input for classical mixing later, if needed. With ongoing HL-LHC discussions of maybe generating the pileup on the fly, i cannot exclude the need to adjust/correct our current strategy before/during HL-LHC.)
Thanks,
cheers, Stephan