Improve Large Backfill Submission UX

[akan72]

What's the use case?

Submitting large backfills to QueuedRunCoordinator can be very slow and painful. How can we speed up the submission of large backfills to the Queue?

Internal context:

Self hosting Dagster and on 1.3.11. Using the K8sRunLauncher + Multiprocess Executor with the QueuedRunCoordinator
We’re submitting multiple large backfills (in the 1000s of runs) and have noticed it takes quite awhile for jobs to be enqueued. It appears that all backfills use the same Queue, and if you submit sequential backfills all jobs in earlier backfills must be enqueued before later backfills can be enqueued. This is somewhat problematic if we want to start different backfills in parallel.
Here’s our config for QueuedRunCoordinator:

  class: QueuedRunCoordinator
  config:
    dequeue_interval_seconds: 5
    dequeue_num_workers: 4
    dequeue_use_threads: true
    max_concurrent_runs: 100

Ideas of implementation

Enqueue runs in parallel, similar to #8642 which was for run submission

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[sryza]

Hi @akan72 – can't promise we'll be able to get to this in the near term, but a clarifying question for when we are able to get to it: is this a backfill that you're launching from the job page, or from the asset graph / asset details page?

[akan72]

Hi @akan72 – can't promise we'll be able to get to this in the near term, but a clarifying question for when we are able to get to it: is this a backfill that you're launching from the job page, or from the asset graph / asset details page?

The job page!

[scottincrypto]

Confirming I'm impacted by this too.

My workaround is to batch backfills into smaller chunks which is extremely time consuming.

[psebek-sf]

Hello,
we're using dagster to run all our data pipelines. We run 10k-20k runs every day. Currently backfill queueing speed is around 1 backfill / s in our production environment. We often need to start a backfill for thousands of partitions at a time (e.g. several months of hourly partitioned asset), but since the queueing is so slow we are doing it by smaller changes (<1000 partitions) to not block everybody from creating new backfills which is manual, time consuming and error-prone. We'd be very happy if we could increase the speed ten or even hundred times. We're using Postgres as a storage.
I've tried to profile current run with py-spy and am attaching the resulting flamechart (it gets interactive if you save the image and open it in your browser)
. I ran a backfill of 66 partitions. Locally I can see the speed is about 5 partitions / s.
From the profile we can see (it helps to search for "backfill" in top right corner and click on it to see it in greater detail) that practically all the time is spent in communication with postgres. For example a lot of time (962 / 2268 samples) is spent in function get_run_by_id which could be prevented if Dagster wouldn't use reserved_run_ids. I tried to remove and saw a minor improvement, but I'm not sure why is reserved_run_ids used so it is just an experiment.
Nonetheless, I think that to make backfilling much faster the architecture of communication with database would have to change to enable bulk insert. The linked function _submit_runs_and_update_backfill_in_chunks already works with chunks but still calls get_run_by_id for each run separately and also calls submit_asset_run one by one.
Are there any plans to make backfilling much faster? I see there was some activity in September. We're willing to help, but since it is a crucial part of Dagster it needs to be properly coordinated.

[psebek-sf]

Hi, just an update from my side. We placed a PgBouncer instance in front of the Postgres database (as recommended here) and could see improvement in backfill speed by 125 %, and drop of CPU from 35 % to 15 %. I'd recommend PgBouncer to all users using postgres.
But I still think there is much more to be done on the code side like properly using connection pool with bigger size than 1 or even better submitting backfill at once in one bulk insert to db (if that's possible, if not I'd be curious about the reasons).
As always, thank you for your work! We rely on Dagster with our pipelines and are overall happy with it.