Built in support for partitioning of multi-step flows instead of single steps

[abbhhiisseek]

Expected Behavior

Built in support or tags for full flows to be partitioned.

Example: a flow F consists of 5 steps s1-> s2 -> s3 -> s4 -> s5
Support for flow F to be portioned i.e. multiple instances of flow F ( s1-> s2 -> s3 -> s4 -> s5 ) could run in parallel through portioning.

Current Behavior

No specified tags to execute entire flows in parallel, although single steps partitioning can be done, there is no such support for portioning a flow of multiple steps.

Context

How has this issue affected you? My application required a flow consisting of multiple steps to be portioned, which could not be accomplished directly
What are you trying to accomplish? A batch job where each flow, consisting of series of steps, to be partitioned using Manager-Worker pattern
What other alternatives have you considered? I tried to use create custom portioning on a flow instead of step, but that did not work, and also caused several data anomalies due to sharing of resources.

Are you aware of any workarounds? Not that i am aware of

[fmbenhassine]

Thank you for opening this feature request. What do you mean by partitioning a flow? I can understand how to partition a data set, but I am not sure I understand what you mean by partitioning the flow itself.

You can run parallel flows using a split flow, where each sub flow is a sequence of steps like s1-> s2 -> s3 -> s4 -> s5. Is that what you are looking for?