Kubernetes hybrid agent redeploys every code location on restart

[williamho123]

What's the use case?

We are currently running a hybrid Dagster deployment in EKS. When the cloud agent is restarted, whether it be through a configuration update or it's rescheduled onto another node, the agent redeploys every code location, even though they haven't changed at all.

When the agent is restarted, we would like it to resume watching / monitoring the already existing code location deployments, rather than deploying new ones.

Ideas of implementation

I'm not sure how the hybrid agent is tracking the deployments it's managing now, but there is some mechanism where it knows to spin up new code location deployments on restart, and then delete the old ones it was tracking before.

Since this is the case, the agent should be able to resume tracking the existing code locations.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[gibsondan]

Hi @williamho123 - to help us prioritize this, can you say more about what problems the current behavior is causing? For example, are the extra pods while it's redeploying the code locations causing excess resource usage?

One thing we would have to figure out for this to work is how to make things work when there are multiple agents in the mix - if you have agent replicas, they each maintain their own set of code servers so that the operation of each agent is fully isolated from each other, so we'd have to figure out what do do when one of those agents restarts in that case (maybe by attributing some unique identifier to each agent)

[williamho123]

Hey @gibsondan! Excess resource usage is one component of it.

But I think the bigger issue we're seeing is thrashing. We use Karpenter to scale our nodes, and here's a scenario we've been running into:

1. Agent gets restarted for some reason
2. New agent creates new K8s deployments
3. Karpenter scales up new node(s) to accommodate the redeployed code locations. This can take a few minutes, between waiting for a new node to come up, pulling the image, and starting the code location server.
4. Once the new code location deployments are running, the old ones are deleted and resources are freed.
5. Karpenter determines consolidation is possible. It deletes the new node(s) that just came up, forcing a rescheduling of the new code location pods that were just created onto the existing node(s)
6. Repeated process of pulling the image and starting the code location server.

Let me know if I can provide more detail around this.

We've also been seeing some intermittent GRPC errors during this time (unsure if related). My guess is that the code location pod being created -> deleted -> created with a new IP might be causing something funky with the service IP resolution?

I noticed that agents have a unique ID in the Dagster+ UI that are also attached as K8s labels to the code location deployments. Are these ID's re-generated every time the agent restarts? Could this be the unique identifier that each agent needs to track the code locations it manages?

Edit to add: if the node that got consolidated by Karpenter in step 5 happened to be the one where the new agent was scheduled to, then the situation is compounded (since we restart in step 1 all over again).