on behaviour of `OutputModules` in unscheduled execution, and its interplay with HLT

[missirol]

As suggested by @silviodonato, I'm opening an issue related to a recent discussion between Core-Sw and HLT concerning the behaviour of OutputModules in unscheduled execution, to keep better track of this problem and its possible solutions (the original exchange was had in the Core-Sw Mattermost channel on Nov, 8-9).

The use case is HLT's goal to fully support workflows with Tasks and SwitchProducers where needed in the HLT menu for Run-3 (e.g. triggers with GPU offloading).

I'm copying below part of the original question [*].

FYI: @fwyzard @Sam-Harper @Martin-Grunewald @makortel @Dr15Jones

[*]

Context : in Run-2 HLT did not make use of Tasks; in Run-3 it will, mainly b/c of the use of SwitchProducers in triggers with GPU offloading. We are currently testing if it's possible to move to Tasks in other parts of the HLT menu, beyond what would be strictly necessary. Note that "migrating HLT to Tasks" for us currently means that EDProducers of a given Path are moved to a Task, but EDFilters are not (they remain directly on the Path, or on a Sequence used in the Path), because the filter decision is needed (as known, in a HLT path one wants to reject the event as soon as possible, without running producers unnecessarily).

After migrating all paths of an HLT config to Tasks as a test, we encountered problems related to OutputModules.

I tried to exemplify the issue in this dummy config. There is 1 OutputModule writing events for the OR of 2 triggers. The OutputModule expects to write the products of 1 producer which runs unscheduled. This producer (theVerticesAfterStep2) is associated to 1 Path (thePathWithProducts) and, in turn, it needs the products of an EDFilter in the same Path, but said EDFilter (theVerticesAfterStep1) runs scheduled. The other trigger (thePathAlwaysTrue) always fires, while thePathWithProducts fails before the filter theVerticesAfterStep1 is executed; the OutputModule tries to run theVerticesAfterStep2 (unscheduled), and this leads to a crash b/c the products of theVerticesAfterStep1 are missing.

The issue boils down to the fact that, as explained in the documentation, OutputModules can trigger the unscheduled execution of a module if its products are expected to be saved in the output file.

Another way this could complicate things for HLT is shown in a 2nd example. Here again, we have an OutputModule writing events for the OR of 2 triggers. A common situation for HLT is that one path gets disabled (or, 'prescaled to zero') to avoid running certain modules (here, theVerySlowModule). But, since theVerySlowModule is unscheduled and expected by the OutputModule, it is executed despite thePathWithSlowModules being "disabled". Clearly, the Fwk is working as intended, just not in the most convenient/intuitive way for HLT.

Question is: would it make sense to have the option to disable the ability of OutputModules to require the unscheduled execution of modules (in other words, simply having outputCommands as a list of directives for saving products only if they were produced)? If yes, how difficult would that be to implement?

[cmsbuild]

A new Issue was created by @missirol Marino Missiroli.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Martin-Grunewald]

assign hlt

[Martin-Grunewald]

assign core

[cmsbuild]

New categories assigned: core,hlt

@missirol,@Dr15Jones,@smuzaffar,@makortel,@Martin-Grunewald you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

We're planning to introduce a new Path type, cms.FinalPath. Only OutputModules would be allowed to be inserted in those, and those OutputModules would be run for an Event only after all other EDModules have been run for that Event, and those OutputModules would not trigger prefetching of the consumed data products (i.e. those OutputModules alone would not cause any other EDModule to run).

[Martin-Grunewald]

Hmm, I do not know whether this restriction (only OMs on FinalPaths) would work: within the HLT menu, we currently have OMs on EndPaths which also have Filters (eg, HLTPrescaler and/or TriggerResultsFilter) and Producers (eg, Scouting) on these EndPaths (in front of their OM).

[makortel]

Thanks @Martin-Grunewald. Naively the current EDFilters in an EndPath could be moved to another regular Path, and configure the OutputModule to store only those Events passing that Path (via SelectEvents). But could you remind us, again, on the use cases for (specific) EDFilters in EndPaths?

[fwyzard]

Hi Matti, the most common use case is to apply prescales and "smart prescales" (i.e. per-trigger-path prescales and other logical expressions) in addition to the event selection applied by the OutputModule itself. For example, the Express and DQM streams apply both an event selection in the OutputModule and a more dynamic preselaction via an EDFilter in the EndPath. An other use case (that may be implemented using dedicated prescales, or a different filtrt altogether) is to enable or disable a DAQ stream (i.e. an output file, so an OutputModule) dynamically during the run. I agree that it should be possible to implement these use cases with an additional dedicated Path; in fact, we may do it anyway to simplify the definition of the DAQ streams and primary datasets. There may be an additional use case: splitting a DAQ stream and its datasets into N copies; I'm not sure if we can cover this use case without EDFilters on the FinalPaths. .A

[makortel]

Thanks Andrea.

In principle a sequence of EDFilters is equivalent to logical AND between the EDFilter return values. We could enhance the SelectEvents ability of OutputModule to evaluate an AND of given Paths, such that this AND would be evaluated before the OR of current SelectEvents (although strictly speaking the same effect can be achieved already now with specific structure of Paths and PathStatusFilter).

[Martin-Grunewald]

Well, the HLTPrescaler and TriggerResultsFilter can not be simulated by an AND of the paths, as individual path-dependent or global prescales are applied. Do you suggest that for each stream/OM, we first define a path which does that prescaling, and then the endpath with the OM then refers only to that path? That seems cumbersome and error prone.