Support concurrent execution of the same DAG

[yottahmd]

[ghansham]

Out of curiosity, what is the reason? Do we have different params?

[yottahmd]

There are two reasons:

1. It allows to reuse common functionalities that are called from multiple DAGs. For now it can only run single instance per a DAG at a time.
2. It will unlock more interesting future functionalities such as for loop or map reduce.

[ghansham]

One more query, how do you plan to save history for such parallel dag runs.

[yottahmd]

I don't think we need to make changes to the persistence layer or history data. Currently, it uses a Request ID (UUID) to identify individual execution history. Even with parallel execution, I believe we can continue using the same mechanism.

The biggest changes required are in the agent process and the Web UI. Currently, all functionalities rely on the fact that only one agent process is running for a particular DAG at a time (e.g., interacting a running process to get the realtime update). I've been pondering how to manage multiple agent processes to enable parallel execution of a DAG. At the moment, I plan to create a new parent process (a service) that manages a group of agent processes.

[ghansham]

You may need to append some uuid kind of thing with dagname to uniquely identify it and that may become your new ID on agent process side. On client side, you may need to identify it using params because I don't think we will be running same dag with exactly same params.

[yottahmd]

Currently, the DAG execution history is managed using composite keys consisting of the DAG name and Request ID. The status files are stored in the path data/{DAG name}_{Hash}/{DAG name}-{Timestamp}-{Request ID}.dat. At the moment, I don't currently see a need to append an ID to the DAG name. Please let me know if I'm overlooking something :)

And yes, parameters can be a way to differentiate executions for users on both the Web UI and CLI.

[ghansham]

One of the point may be to disallow user to run same dag with same set of params.

[yottahmd]

I'm not sure if it's necessary, as some common utility DAGs might get the same parameters, for example target_directory for backup task. What is the reasoning behind your view that identical parameters should be disallowed?

[ghansham]

Logically a program or script should generate same output for same input. Backup scripts that you mentioned run in cron but not concurrently. And if they are running concurrently then source and destination given as cmdline args (aka params in dagu) should be different otherwise they may corrupt the output files being backed up. For map reduce task and "for" loops, it should have data divided into chunks and same dag should be running on different parts of same data.

[yottahmd]

Ah, I see. You're absolutely right. Yes, normally, users should not run the same DAG with the same set of parameters. That's an excellent point. I'll try to implement to prevent it. Thanks for bringing it up!

[ghansham]

What we are trying to implement actually already exists if we invoke the common dag as a Subdag with different params.

[yottahmd]

I might be misunderstanding something, but here’s the scenario I’m thinking of:

Let's say we have a common DAG called backup, and we define two separate DAGs like this:

DAG: backup_1:

steps:
  - name: call_backup
    run: backup
    params: DIR=/data1

DAG: backup_2:

steps:
  - name: call_backup
    run: backup
    params: DIR=/data2

If you try to run these two DAGs simultaneously, one of them will fail with the current implementation even though the params are different.