Add out-of-cluster support for KubernetesJob

[thedavekwon]

We currently only support in-cluster KubernetesJob, but there is great usefulness in having a client external to the cluster driving the job. This allows workflows such as:

• A jupyter notebook that can send messages to meshes on the cluster. Reloading single cells enables fast iteration speed while providing access to remote hardware at a larger scale
• User scripts running on local clients outside the cluster have greater flexibility in what code they can write
• Starting jobs from a separate cluster, for example from a web server, and being able to send commands into the other cluster

There is one major blocker to get this working: monarch clients rely on a way to directly communicate with all hosts in the mesh, either by IP address + port or a hostname + port. Kubernetes does not expose public static IP addresses or hostnames by default outside of the cluster. This task is about finding a way to bridge the communication gap.

A second subtask is how to efficiently get the code from the local client onto the remote cluster. We would also like if we can get the user's installed PyPI packages synced as well, so users can quickly try things that require new packages. For an initial implementation the existing HostMesh.sync_workspace will be fine, but we'll need to enhance that for scale, reliability, and connectivity issues.

[dulinriley]

@dongreenberg has a PR for this that uses kubetorch, but we'd like for there to be a way to do this without requiring a third-party library too, so we'll iterate on this.
#2476

[aojea]

I already implemented this functionality in https://github.com/aojea/krun , that is just a POC code because some people needed it for a similar project to monarch (pathways).
My point is that using the existing kubernetes streaming apis for attach, exec and portforward is perfectly possible and "relatively" simple ... I'm happy to help here if you are interested

[aojea]

@dulinriley is this always http traffic?

[dulinriley]

Our current implementation plan for the main part of monarch is to add a "monarch-gateway" service into the Kubernetes cluster that exposes a public IP + port. This service will use the monarch internal MailboxServer primitives to multiplex connections from outside the cluster to meshes inside the cluster. This service will only be enabled if your cluster wants to allow external access. The client can use TLS to authenticate themselves and ensure they are the one that started the job.
I think we only need one of these gateways per cluster, but our existing MonarchMesh CRD is per-mesh, so we may also make this gateway per-mesh to avoid making extra changes. I don't think there's any obvious downside to that (two actors on different meshes but in the same cluster can still communicate with each other).

Then the HostMesh (+ProcMesh and ActorMesh) connection just becomes a change from sending to some IP address + port to sending a message to the external IP + the cluster-level IP to forward to. So this should be minimally invasive and allow the same code to work in-cluster and out-of-cluster.

Any changes to how the environment is synced (code, pip packages, conda environment, etc.) is not in scope for this issue and will be handled separately. This issue is purely about allowing the connection to occur.

[aojea]

"monarch-gateway" service into the Kubernetes cluster that exposes a public IP + port.

I think we only need one of these gateways per cluster, but our existing MonarchMesh CRD is per-mesh, so we may also make this gateway per-mesh to avoid making extra changes. I don't think there's any obvious downside to that

If you have a public IP/Port per gateway mesh, the friction will be the cost since these are typically charged by cloud providers

So this should be minimally invasive and allow the same code to work in-cluster and out-of-cluster.

Yeah, and you have a central point to enforce authN/Z

[dulinriley]

If you have a public IP/Port per gateway mesh, the friction will be the cost since these are typically charged by cloud providers

Thanks for that information. We changed the design slightly, now we are going to have the client proc "attach" to a remote host. We will only attach to a single host in the cluster, and that will act as the client's address, while forwarding messages back to it.
This will use the new duplex channel feature, which allows Procs to initiate connections to hosts, rather than the other way around