Dirac CWL Executor + testing with LHCb Workflows (simulation and analysis productions) by ryuwd · Pull Request #57 · DIRACGrid/dirac-cwl

ryuwd · 2025-11-19T17:03:52Z

Adds a new dirac-cwl-run utility for running CWL workflows with input data resolution according to the concepts introduced in #69 with Dirac LFNs, the "replica catalog", and a proof-of-concept for using input data (in LHCb this is the "Bookkeeping") queries to pre-fill empty input-data workflow parameters.

Also adds converters for LHCb production YAMLs --> CWL workflow files.

Tested with LHCb simulation and analysis productions workflows.

src/dirac_cwl_proto/lhcb/run_lbprodrun.py

…sider not using explicit name in CWL workflow which is unreadable

…with artifacts that go to LogSE

aldbr · 2025-12-08T07:20:05Z

test/workflows/lhcb/production_requests/anaprod.yaml

+      - 0
+      events: -1
+      priority: 2
+      multicore: true


FYI, with the latest changes made to dirac-production-request-launch, multicore is going to be false for analysis productions.
It's going to be merged within the week if everything works as expected 🤞

test/workflows/lhcb/production_requests/simulation.yaml

aldbr · 2025-12-08T07:30:09Z

src/dirac_cwl_proto/lhcb/prod_request_to_cwl/converter.py

+    for step_index, step in enumerate(steps):
+        step_name = _sanitizeStepName(step.get("name", f"step_{step_index}"))
+        step_names.append(step_name)
+        cwl_step = _buildCWLStep(production, step, step_index, workflow_inputs, step_names)


I guess you need the changes I am making to lb-mc to integrate the submission-info section here to build subworkflows (that would become transformations).

workflows

aldbr

Thanks! That looks very promising 🙂

aldbr · 2026-01-08T08:45:46Z

test/workflows/lhcb/production_requests/anaprod_pi0_OO_2025.yam

Oops: .yam -> .yaml

aldbr · 2026-01-08T09:32:45Z

test/workflows/lhcb/generated/AnaProd#lc2pkpidcs#2018_MagDown_collision_LbToLcpkpi.cwl

+- class: dirac:inputDataset
+  event_type: '90000000'
+  conditions_dict:
+    inTCKs: ALL
+    inProPass: Real Data/Reco17/Stripping29r2p2
+    configName: LHCb
+    inFileType: CHARM.MDST
+    configVersion: Collision17
+    inProductionID: ALL
+    inDataQualityFlag: OK
+  launch_parameters:
+    end_run:
+    start_run:
+    run_numbers:
+  conditions_description: Beam6500GeV-VeloClosed-MagUp


A bit of context about the dirac:execution-hooks hint because I feel like the concepts are very similar here, but I might be wrong.

The current dirac:execution-hooks hint is used at 2 different locations:

transformation system: use specific parameters to query a FileCatalog/Bookkeeping to get LFNs and attach them to a job to submit

job: use the hooks to pre/post process the jobs and the specific parameters to register the outputs in a specific part of the FileCatalog/Bookkeeping

Now you are right, the current dirac:execution-hooks hint is not global to a CWL production, but actually defined in every transformation.
I think the reason is that I initially wrote CWL workflows where the input parameters were used to build the query (based on an example provided by CTAO).

Example with mandelbrot:

transformation1: no parameter defined because the input parameters of the workflow are used to define the query: https://github.com/aldbr/dirac-cwl-proto/blob/52cb47ad1406e108732aa4b3f03332f32ce95f6b/test/workflows/mandelbrot/type_dependencies/transformation/metadata-mandelbrot_imageprod.yaml

transformation2: no information about the query parameter because the input are different, so we have to provide the query parameters (corresponding to the input of transformation1): https://github.com/aldbr/dirac-cwl-proto/blob/52cb47ad1406e108732aa4b3f03332f32ce95f6b/test/workflows/mandelbrot/type_dependencies/transformation/metadata-mandelbrot_imagemerge.yaml
https://github.com/aldbr/dirac-cwl-proto/blob/52cb47ad1406e108732aa4b3f03332f32ce95f6b/test/workflows/mandelbrot/image-prod.cwl

Then CTAO wrote an example where they were using parameters not defined as inputs of the workflow: https://github.com/aldbr/dirac-cwl-proto/blob/dc2cee46c1b804a6ec5add6da5dd7123c990630e/src/dirac_cwl_proto/execution_hooks/plugins/core.py#L32-L35

Is that something you actually need @arrabito?

In this PR, you also define query parameters not defined as inputs of the workflow @ryuwd .

If that's the case for everyone, then may be we should have a discussion about:

are the query parameters always different from the input parameters of the workflows? Can we use both as query parameters?

are the query parameters the same for the whole workflow? (e.g. production with 3 transformations that would have the same query parameters)

Then we should think whether and how we should separate:

execution hooks (pre/post processing of the jobs),

query parameters (used by the transformation system to get LFNs, as well as the jobs to store LFNs at the end),

other input/output parameters (e.g. output data and sandbox).

In the meantime, just for your context, you could have potentially created an execution hooks plugin such as (but you would have to define it for every transformation, which I agree, is not convenient in this context):

class LHCbBasedPlugin(ExecutionHooksBasePlugin): # LFN parameters in_tcks: str = Field(...) in_pro_pass: Optional[str] = Field(...) config_name: Optional[str] = Field(...) in_file_type: Optional[str] = Field(...) ...

aldbr · 2026-01-08T09:39:44Z

test/workflows/lhcb/generated/AnaProd#lc2pkpidcs#2018_MagDown_collision_LbToLcpkpi.cwl

+
+  Generated by ProductionRequestToCWL converter
+inputs:
+- id: production-id


I am just curious, why do you need the production-id here?
Same question for the prod-job-id.

I would have been tempted, if possible, to remove anything DIRAC-related from the CWL workflow itself, and move DIRAC-specific attributes to a hint (may be this is not possible).

This was addressed in newer versions of the LHCb CWL generator in LbAPCommon

aldbr · 2026-01-08T09:47:52Z

test/workflows/lhcb/generated/AnaProd#lc2pkpidcs#2018_MagDown_collision_LbToLcpkpi.cwl

+  doc: Production Job ID
+  default: 6789
+  type: int
+- id: input-data


Out of curiosity, what do you expect here? LFNs you got from you bk query? Is this going to work if you have a very large number of files?

I had issues with some workflows with my converter. You can see how I dealt with it if you check the archive I gave you a few weeks/months ago but basically, I did:

input-data: class: File contents: '["/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023715_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023714_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023753_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023754_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023685_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023697_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023710_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023711_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023693_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023686_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023708_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023682_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023674_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023688_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023673_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023680_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023675_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023706_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023681_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023691_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023699_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023712_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023695_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023713_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023704_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023696_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023676_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023732_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023719_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023747_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023716_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023724_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023728_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023723_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023740_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023749_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023743_1.charm_d02hh_dvntuple.root", "/lhcb/LHCb/Collision25/CHARM_D02HH_DVNTUPLE.ROOT/00296922/0002/00296929_00023748_1.charm_d02hh_dvntuple.root"]'

aldbr · 2026-01-08T09:54:00Z

test/workflows/lhcb/generated/AnaProd#lc2pkpidcs#2018_MagDown_collision_LbToLcpkpi.cwl

+      configuration:
+        cpu: '1000000'
+        priority: 2
+        multicore: true


priority would likely go in the scheduling hint (though I think it's going to disappear in diracx with the work on the new matcher)

cpu would likely be defined in the CWL itself as a requirements no?

multicore would also be part of the CWL requirements

I think output_se, remove_inputs_flags, output_file_mask, output_mode, input_data_policy would be defined outside of configuration (because configuration contains your query parameters for the FC/Bookkeeping)

do you need to use ancestore_depth and events as query parameters? Do you know why it's here?

src/dirac_cwl_proto/lhcb/run_lbprodrun.py

src/dirac_cwl_proto/executor/fs_access.py

aldbr · 2026-01-08T15:04:48Z

src/dirac_cwl_proto/executor/pathmapper.py

+logger = logging.getLogger("dirac-cwl-run")
+
+
+class DiracPathMapper(PathMapper):


I fail to understand why we need this DiracPathMapper, can you explain please?

Because, I would have naively thought that adding the replica catalog to the active CLT directory would be enough: the application needing the replica catalog would read it and use whatever PFNs it needs.

My naive understanding is CWL needs to be told various ways to deal with paths and this class is the way. The StdFsAccess deals with filesystem access. I'm not sure yet if there's a cleaner way to do all this. The nice feature of having this is we can directly map LFNs to PFNs using the replica catalog before the paths reach the command line. I am not sure how much of a good idea this is for LHCb jobs, but it seems like a nice default to have to hide this detail from end users writing their workflows.

The better answer is the the CWL PathMapper can't deal with LFNs so we need to write our own to pass them through or resolve them to PFNs using the replica map

aldbr · 2026-01-08T15:25:19Z

src/dirac_cwl_proto/executor/executor.py

+
+
+# Apply the monkey-patch
+CommandLineTool.make_path_mapper = staticmethod(_custom_make_path_mapper)


I guess you would need to override the CommandLineTool to avoid the monkey patch.
Overall, I think that would be a good idea to ask for guidance (just to make sure we are going in the right directions here) in the CWL forum. What do you think?

I think that is ideal

#115 has dealt with this but I ran into some bizarre issues with mypyc distributed cwltool which might need some further discussion

src/dirac_cwl_proto/executor/executor.py

ryuwd · 2026-01-24T17:15:54Z

New plan:

Remove replica catalog stuff and import from diracx
Refactoring / cleanup of the executor
Remove converters (now lives in analysis productions, eventually may move simulation part to lbmcsubmit)
Regenerate some example CWL files that can be used in tests.
Write tests of the cwl executor.

ryuwd · 2026-02-02T17:11:44Z

Moving the executor to #94

ryuwd · 2026-02-17T14:03:45Z

Another PR to be opened focused purely on tests with the dirac CWL executor and with LHCb workflows.

aldbr reviewed Nov 20, 2025

View reviewed changes

src/dirac_cwl_proto/lhcb/run_lbprodrun.py Show resolved Hide resolved

ryuwd added 4 commits November 27, 2025 16:09

feat: pydantic model for json catalog

dcee5a9

fix: types

f3e7f39

fix: remove redundant Path

28574f3

fix: update pixi.lock

14886b3

ryuwd changed the title ~~LHCb Workflows~~ LHCb Workflows (simulation) Dec 1, 2025

ryuwd added 13 commits December 2, 2025 22:29

feat: more validation and better name

3bd7914

feat: it's a start ...

2f27c79

fix: ensure prodconf.json created with the correct filename TODO: con…

d344608

…sider not using explicit name in CWL workflow which is unreadable

fix: improved wrapper

8f054d1

fix: output globbing

c9cfddd

feat: getting closer

58bcdf6

fix: warnings

84b67ba

fix: deal with pool XML catalog and passing between steps. Also deal …

ca66d02

…with artifacts that go to LogSE

fix: collect output files, create log files

2127eb9

refactor into package, properly pass input between steps

e4d0989

fix: hacking things together

1528fc3

fix: removed redundant test files

5716526

fix: added analysis productions YAML example

957d8a0

ryuwd force-pushed the roneil-lhcb-wf branch from a391398 to 957d8a0 Compare December 7, 2025 12:32

aldbr reviewed Dec 8, 2025

View reviewed changes

ryuwd added 9 commits December 9, 2025 16:55

feat: TODO transformations as sub

7147d43

workflows

fix: remove unused checksums

cd7e4ae

feat: replica catalog generator using DIRAC

c28a552

Merge branch 'roneil-json-catalog' into roneil-lhcb-wf

951d584

developments TODO deal with subworkflow steps

a27b738

fix: workflow

c0022d6

working executor

58fac54

refactor: support anaprod

3e12649

feat: a working analysis production!

be32b6f

ryuwd added 2 commits December 24, 2025 07:09

chore: start fixing pre-commit stuff

255784d

AnaProds and Simulation working

2553e63

ryuwd changed the title ~~LHCb Workflows (simulation)~~ LHCb Workflows (simulation and analysis productions) Dec 24, 2025

fixes for env

9f3a7f8

ryuwd changed the title ~~LHCb Workflows (simulation and analysis productions)~~ Dirac CWL Executor + testing with LHCb Workflows (simulation and analysis productions) Dec 29, 2025

ryuwd mentioned this pull request Jan 5, 2026

feat: pydantic model for json catalog and a generator w/ input data resolution #63

Closed

aldbr reviewed Jan 8, 2026

View reviewed changes

This was linked to issues Jan 20, 2026

LHCb Workflow: AnalyseXMLSummary command #68

Open

feat: Implement a "JSON catalog" input data resolution format #62

Closed

How to deal with input data resolution in different execution contexts and with a replica catalog #69

Open

ryuwd mentioned this pull request Jan 22, 2026

How to deal with input data resolution in different execution contexts and with a replica catalog #69

Open

aldbr removed a link to an issue Jan 22, 2026

feat: Implement a "JSON catalog" input data resolution format #62

Closed

1 task

aldbr removed a link to an issue Feb 9, 2026

LHCb Workflow: AnalyseXMLSummary command #68

Open

4 tasks

ryuwd closed this Feb 17, 2026

		logger = logging.getLogger("dirac-cwl-run")


		class DiracPathMapper(PathMapper):



		# Apply the monkey-patch
		CommandLineTool.make_path_mapper = staticmethod(_custom_make_path_mapper)

Conversation

ryuwd commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aldbr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryuwd Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryuwd commented Jan 24, 2026

Uh oh!

ryuwd commented Feb 2, 2026

Uh oh!

ryuwd commented Feb 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ryuwd commented Nov 19, 2025 •

edited

Loading

ryuwd Jan 21, 2026 •

edited

Loading