Locations can end up being unloaded in spite of the watch thread

[freewilll]

Dagster version

1.4.11

What's the issue?

When running in kubernetes, using the default RollingUpdate deployment strategy, during the replacement of the pods, it can happen that one pod is serving traffic and another has shut down. The service layer takes a while to update and round robins across both pods. The grpc watch thread in dagster-webserver detects a location is up by hitting the healthy pod and via on_updated, sends an update message to the WorkspaceProcessContext. It then tries to load the location, hits, the dead pod and fails. In the meantime, the service has been updated and the watch thread keeps hitting the healthy pod. The result is the location ends up being permanently unloaded.

What did you expect to happen?

I expect healthy locations to be (re)loaded.

How to reproduce?

• Add a 10 second sleep at the top of _start_watch_thread in context.py to simulate a grpc location flipping from unavailable to available
• Start dagit-webserver with a workspace configured to connect to the grpc location, it will log a failure to load the location
• Start the location grpc server
• The watcher starts when the sleep completes and thinks the location is alive

The UI indicates the location isn't loaded and this doesn't change.

Deployment type

Other Docker-based deployment

Deployment details

We use a kubernetes deployment with the default RollingUpdate strategy.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[smackesey]

cc @johannkm

[kwmlodozeniec]

This is really plaguing us in production. Any news on this?

[johannkm]

Just to clarify: you upgrade the location, it shows failed in the UI, and will succeed if you manually reload it from the UI right?

[freewilll]

Correct. We typically click reload a few times since we are running multiple dagit webservers. It can also be fixed by killing the location pod: the watcher picks up the recycled pod.

[johannkm]

It can also be fixed by killing the location pod: the watcher picks up the recycled pod.

Which pod are you killing, the new one?

[freewilll]

When dagster is in a state where the UI claims a location is unloaded while it is responding, deleting the pod, so that it gets recreated automatically by kubernetes, fixes the problem. The watcher first picks up that the pod has gone down and goes into reconnect_loop. Once the pod has come up, an update is logged and the location gets loaded.

[freewilll]

This is also happening on startup. If dagit starts up at the same time as a location, the main thread can see the location while it is down. The watcher thread later on sees it as being up and doesn't signal the main thread the repository is alive.

[mfogel]

@johannkm looks like a fix for this was filled (finally!) in #33642. Any chance you or somebody at Dagster could review that? Would be awesome to have this addressed.

[panandr]

We would really appreciate if #33642 could be merged. We plan to use Dagster in production with frequent redeploys and multiple code locations, thus, this heavily affects us!