Skip to content

Parallel workflow execution#48

Merged
aldbr merged 6 commits intoDIRACGrid:mainfrom
AcquaDiGiorgio:issue-43-parallel-worflows
Nov 6, 2025
Merged

Parallel workflow execution#48
aldbr merged 6 commits intoDIRACGrid:mainfrom
AcquaDiGiorgio:issue-43-parallel-worflows

Conversation

@AcquaDiGiorgio
Copy link
Copy Markdown
Contributor

Add parallel execution at job submission by default

Tests check the following:

  • If the workflow got actually executed parallely (against a sequential workflow)
  • Having only a singular available core, if it acts like a sequential workflow

See #43

@AcquaDiGiorgio AcquaDiGiorgio self-assigned this Nov 4, 2025
@AcquaDiGiorgio AcquaDiGiorgio changed the title Parallel worflow execution Parallel workflow execution Nov 4, 2025
@aldbr aldbr linked an issue Nov 5, 2025 that may be closed by this pull request
3 tasks
Copy link
Copy Markdown
Contributor

@aldbr aldbr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thank you @AcquaDiGiorgio 🙂

I just have a few comments:

  • We are streamlining the tests in parallel and the mandelbrot workflow will likely go away. Do you think you could get the same results using the crypto workflow? I think it contains 4 independent steps.
  • Do you think we need to have a "sequential" version of the tested workflow? Naively, I think it is not necessary because it's already tested everywhere else: if dependent steps would run in parallel, other tests couldn't work.
  • I think I would test the crypto workflow with taskset -c 0 and then taskset -c 0-1 for instance to demonstrate the difference in duration.

Do you have any opinion? Please let me know if you think I 'm wrong

@AcquaDiGiorgio
Copy link
Copy Markdown
Contributor Author

Due to the nature of the crypto algortithms, it might be difficult to differentiate between a parallel and a sequential execution, in both cases they get executed extremely quick. If pi-simulate is going to stay in the future, I might be able to use it, because the size of inputs will vary the execution time.

I'm using a forced sequential workflow to be able to compare it with a parallel one that executes in one core.
If you find this test innecesary, I can completely remove it and the sequential cwl, bacause for first test, as you said, the sequential workflow can be a taskset -c 0 of the parallel one.

@aldbr aldbr merged commit 68cf5fd into DIRACGrid:main Nov 6, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use the --parallel option

2 participants