[dagster_k8s] hard-coded WAIT_TIMEOUT_FOR_READY in consume_pod_logs is causing a problem

[takb]

Hi guys, thank you for all your hard work.

I've run into some trouble with my pipeline, and as far as I can see the only solution requires the WAIT_TIMEOUT_FOR_READY parameter to be made configurable.

The trouble is with following situation: I work for the openrouteservice project, and I'm trying to set up a processing pipeline for routing graphs which take openstreetmap data and calculates a graph representation of the road network and some derived data for use with routing algorithms. I've set up an asset with partitions for each input dataset (we want to build a planet graph and some smaller subgraphs for use by project partners for their specific needs) and each of our different profile setups (representing different modes of transport, resulting in different graphs and prepared data for each - currently we build 9 profiles for our public API and several specialized setups for project partners), that uses dagster_k8s to run our docker image with different configurations. I've set up sensors for different input files and all is fine, until I try to start the larger production setups with the OSM planet file.

Each profile (and therefore each pod I start) requires several hunded GBs of memory and storage and runs for 12-36 hours. The resources in our k8s cluster allow for (depending on the load situation from other projects within our institution) 1-3 of these pods to run in parallel. The problem is now that my sensor triggers 9 asset materializations, each creates a pod, but multiple pods need to wait for the first pods to finish until they can be provisioned to actually run. By the time the first pod finishes, however, every waiting pod raises a dagster_k8s.client.DagsterK8sError: Timed out while waiting for pod to get to status READY with pod info ... and fails.

I've looked in your latest code, and it looks to me that

dagster/python_modules/libraries/dagster-k8s/dagster_k8s/pipes.py

Line 576 in 3f9e402

wait_timeout=WAIT_TIMEOUT_FOR_READY,

is causing this behavior. My questions, then, would be:

• Is this the cause of the problem, or am I overlooking something?
• Is there any workaround to allow dagster to wait for to-be-provisioned pods longer, that I overlooked?
• Is there a good reason why this timeout should remain hard-coded, or is this something that you would consider changing?
• Should I create a feature-request issue, how long would that take to get addressed, or should I try and contribute a fix?

Thanks for your time.