Description
Search before asking
- I had searched in the issues and found no similar feature requirement.
Description
I used dolphinscheduler for a few months now, I would like to offer a few suggestions for discussion:
Question 1: Parameter passing
In dolphinshceduler, task level parameters and workflow level parameters are supported, but I found that some parameter sets could not be reused throughout the project or even across workflows;
In view of the above requirements, I hope to add the following optimization in the future:
- Correlation between workflows [sub_process] supports parameter passing. If I define the pre - and post-position relationships of multiple workflow in a sub_process task, the parameter list I defined in the first place can be reused in the current task execution cycle.
- Reuse of the whole project-level parameter set (note the set), similar to shell or sql task support parameter definition;
In general, our scheduling tools in the real world use more reusable parameters, so that the parameters are not limited to a single job flow or job;
We first need to define a general parameter set script, through each run batch automatically get the latest parameters only, I only need to use the parameters where the introduction of my dependency script can achieve parameter reuse, this is really very traversal.
Question 2: Event monitoring
The event monitoring function is introduced in the scheduling policy to support multiple job triggering modes.
In the actual scenario, we need to interact with other systems to achieve the automatic triggering and flow of job batches, and usually the more common interaction is the tag file (such as.OK file),
The presence of a tag file indicates that the upstream job or task we depend on has been executed and can trigger the post task defined in our system.
Question 3: Daily switching mechanism
At present, there is only timing policy in the system to realize date switching.
Often when the system job runs batch, there may be some delays, errors and other problems that may lead to the overall delay of the day's job execution batch;
If the batch is cross-dated, some of the previous day's jobs may have missed execution, which can lead to data errors.
Use case
No response
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct