Skip to content

tpu-inference/k8s: attach to a dispatched workload without naming its cluster - #489

Draft
theminghuang wants to merge 1 commit into
kube-launcher-finalfrom
tpu-inference/k8s-mkexec
Draft

tpu-inference/k8s: attach to a dispatched workload without naming its cluster#489
theminghuang wants to merge 1 commit into
kube-launcher-finalfrom
tpu-inference/k8s-mkexec

Conversation

@theminghuang

Copy link
Copy Markdown
Collaborator

MultiKueue decides which worker a job lands on, and a worker's pods are
invisible from the manager because MultiKueue skips pod creation there. So the
person who submitted a job has no way to reach it without first finding out
where it went and configuring credentials for that cluster.

mkexec.sh asks the manager instead, which is the component that made the
decision: Job -> Workload by ownership, Workload -> worker by
.status.clusterName, worker -> ClusterProfile, ClusterProfile -> the Connect
Gateway URL the fleet publishes. A throwaway kubeconfig is synthesized for that
endpoint and authenticated as the caller, so no shared worker kubeconfig is
distributed and the user's own kubeconfig is untouched. It is an ordinary CLI -
a workstation or a plain VM with gcloud works the same as anything in-cluster.

The winning cluster is read from the typed .status.clusterName, falling back to
parsing the admission check message only if that field is absent; the message
is human-readable prose Kueue may reword between releases.

Access is documented, not provisioned. exec is a create on pods/exec, so a
human needs gatewayEditor rather than the launcher's gatewayReader, plus an
RBAC binding on each worker. Who may exec into a shared CI cluster is an
access-control decision rather than a topology one, so it does not belong on
the generate-and-apply path that builds the queues.

The demo pair exercises the whole resolver on a CPU-only queue riding the same
MultiKueue admission check, so the path can be shown without spending TPU.

… cluster

MultiKueue decides which worker a job lands on, and a worker's pods are
invisible from the manager because MultiKueue skips pod creation there. So the
person who submitted a job has no way to reach it without first finding out
where it went and configuring credentials for that cluster.

mkexec.sh asks the manager instead, which is the component that made the
decision: Job -> Workload by ownership, Workload -> worker by
.status.clusterName, worker -> ClusterProfile, ClusterProfile -> the Connect
Gateway URL the fleet publishes. A throwaway kubeconfig is synthesized for that
endpoint and authenticated as the caller, so no shared worker kubeconfig is
distributed and the user's own kubeconfig is untouched. It is an ordinary CLI -
a workstation or a plain VM with gcloud works the same as anything in-cluster.

The winning cluster is read from the typed .status.clusterName, falling back to
parsing the admission check message only if that field is absent; the message
is human-readable prose Kueue may reword between releases.

Access is documented, not provisioned. exec is a create on pods/exec, so a
human needs gatewayEditor rather than the launcher's gatewayReader, plus an
RBAC binding on each worker. Who may exec into a shared CI cluster is an
access-control decision rather than a topology one, so it does not belong on
the generate-and-apply path that builds the queues.

The demo pair exercises the whole resolver on a CPU-only queue riding the same
MultiKueue admission check, so the path can be shown without spending TPU.
@theminghuang

Copy link
Copy Markdown
Collaborator Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant