tpu-inference/k8s: attach to a dispatched workload without naming its cluster - #489
Draft
theminghuang wants to merge 1 commit into
Draft
tpu-inference/k8s: attach to a dispatched workload without naming its cluster#489theminghuang wants to merge 1 commit into
theminghuang wants to merge 1 commit into
Conversation
… cluster MultiKueue decides which worker a job lands on, and a worker's pods are invisible from the manager because MultiKueue skips pod creation there. So the person who submitted a job has no way to reach it without first finding out where it went and configuring credentials for that cluster. mkexec.sh asks the manager instead, which is the component that made the decision: Job -> Workload by ownership, Workload -> worker by .status.clusterName, worker -> ClusterProfile, ClusterProfile -> the Connect Gateway URL the fleet publishes. A throwaway kubeconfig is synthesized for that endpoint and authenticated as the caller, so no shared worker kubeconfig is distributed and the user's own kubeconfig is untouched. It is an ordinary CLI - a workstation or a plain VM with gcloud works the same as anything in-cluster. The winning cluster is read from the typed .status.clusterName, falling back to parsing the admission check message only if that field is absent; the message is human-readable prose Kueue may reword between releases. Access is documented, not provisioned. exec is a create on pods/exec, so a human needs gatewayEditor rather than the launcher's gatewayReader, plus an RBAC binding on each worker. Who may exec into a shared CI cluster is an access-control decision rather than a topology one, so it does not belong on the generate-and-apply path that builds the queues. The demo pair exercises the whole resolver on a CPU-only queue riding the same MultiKueue admission check, so the path can be shown without spending TPU.
Collaborator
Author
|
This change is part of the following stack: Change managed by git-spice. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MultiKueue decides which worker a job lands on, and a worker's pods are
invisible from the manager because MultiKueue skips pod creation there. So the
person who submitted a job has no way to reach it without first finding out
where it went and configuring credentials for that cluster.
mkexec.sh asks the manager instead, which is the component that made the
decision: Job -> Workload by ownership, Workload -> worker by
.status.clusterName, worker -> ClusterProfile, ClusterProfile -> the Connect
Gateway URL the fleet publishes. A throwaway kubeconfig is synthesized for that
endpoint and authenticated as the caller, so no shared worker kubeconfig is
distributed and the user's own kubeconfig is untouched. It is an ordinary CLI -
a workstation or a plain VM with gcloud works the same as anything in-cluster.
The winning cluster is read from the typed .status.clusterName, falling back to
parsing the admission check message only if that field is absent; the message
is human-readable prose Kueue may reword between releases.
Access is documented, not provisioned. exec is a create on pods/exec, so a
human needs gatewayEditor rather than the launcher's gatewayReader, plus an
RBAC binding on each worker. Who may exec into a shared CI cluster is an
access-control decision rather than a topology one, so it does not belong on
the generate-and-apply path that builds the queues.
The demo pair exercises the whole resolver on a CPU-only queue riding the same
MultiKueue admission check, so the path can be shown without spending TPU.