Ability to schedule pipelines #5282
Description
We have been discussing how it would be useful to sometimes be able to schedule pipelines so they only run at certain times of the days e.g 10pm to 2am.
Priority kind of does by running low priority jobs when there is little resource contention, however if you are doing some back-processing, especially with some more lumpy data sizes, we think it could be useful to be able to more directly control when the pipeline is able to process.
Feedback from devs:
Yes this could be done. I think internally this would translate to scheduling when the processing is performed whereas the other scheduling that ensures ref data is available first might be more about when we actually create tasks ready for processing if that makes sense, i.e. it is a 2 part process of task creation and then selecting tasks for processing.
When we have a pipeline that needs to reprocess very large streams (which could impact the scheduling of other tasks), we sometimes use the Max Concurrent task limiter to cap the number of jobs that are actively processing at once, for that processor filter only. This may partially meet your needs, though this figure won't ramp up overnight, when you may have additional capacity to process more jobs concurrently.
image.png