Partitioned asset backfill hangs with QueuedRunCoordinator

[zaneselvans]

Unresponsive QueuedRunCoordinator with partitioned job

• I'm using Dagster core v1.12.7 on MacOS 26.2 and trying to materialize a set of partitioned assets locally in development.
• There are 50 partitions, and up to 4 of them can be running simultaneously without overwhelming my local compute resources.
• I'm using the QueuedRunCoordinator and have set max_concurrent_runs to 4.
• When I start up the Dagster UI (using dagster dev) and tell it to run a backfill that encompasses all or even a significant subset of the partitions, they appear to start -- in the UI they are listed as started, and in the console, I see a run for each of them get kicked off.
• However, no computation happens. Instead it seems to sit in an endless loop with these logging messages displaying over and over again.
• The data is being read from zipped CSVs on disk, and processed into Parquet files on disk using DuckDB. There are hundreds of thousands of input files (100GB zipped, 600GB uncompressed) and one file per table per quarter in the outputs (about 70GB of Parquet across 4 tables x 50 quarters).

2025-12-20 00:10:11 -0600 - dagster.daemon.BackfillDaemon - INFO - Resuming job backfill for wcujvnbp from 2022q2
2025-12-20 00:10:11 -0600 - dagster.daemon.BackfillDaemon - INFO - Backfill wcujvnbp has unfinished runs. Status will be updated when all runs are finished.
2025-12-20 00:10:15 -0600 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - 4 runs are currently in progress. Maximum is 4, won't launch more.
2025-12-20 00:10:21 -0600 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - 4 runs are currently in progress. Maximum is 4, won't launch more.
2025-12-20 00:10:26 -0600 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - 4 runs are currently in progress. Maximum is 4, won't launch more.
2025-12-20 00:10:31 -0600 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - 4 runs are currently in progress. Maximum is 4, won't launch more.
2025-12-20 00:10:36 -0600 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - 4 runs are currently in progress. Maximum is 4, won't launch more.

• Then, mysteriously, after doing that for about 40 minutes (!) it suddenly decided to start processing data and the 4 runs actually seemed to get going.
• When it completes a partition, it seems to take a long time for it to kick of a new one, and sometimes it seems to lapse back into the situation where it thinks 4 runs are in progress, but it's not using any CPU, or it's only using 1 CPU, for long periods, despite not having finished the run.
• What might cause this kind of behavior? How can I make it start processing immediately? How can I make it continue processing until it is actually complete? This does not seem functional!

Default Run Coordinator?

I have also tried using the DefaultRunCoordinator with max_concurrent_runs of 4, but it did not seem to limit the number of concurrent runs. Instead all of the partitions show up as active simultaneously. However, despite all the partitions apparently starting simultaneously, it still did not start doing a bunch of data processing.

Dagster UI Hanging?

Another thing that has started happening frequently, especially when working on the partitioned assets, is that the web UI will hang, and no longer be able to update the status, even though I can see that the backfill is proceeding in the console.

Zombie Processes

I'm also experiencing lots of instances of zombie processes that linger after I quit out of Dagster at the console. They are unwilling to die from a polite kill or kill -15 and I usually end up having to kill -9 them.