Part 2 - LauncherPopulationPolicy & Dual-pods controller logic refinement

[osswangxining]

Need one sample to explain what the total number defined within policy object means.

Does the following example help? Suppose that there is one LauncherPopulationPolicy object, containing one CountForLauncher. We restrict our attention to the LauncherConfig and Node mentioned there. Suppose that the CountForLauncher's LauncherCount is 2. Suppose that before any server-requesting Pods are created, the population policy controller creates the 2 launcher Pods. Next, suppose that some user creates a server-requesting Pod that refers to the same LauncherConfig and gets scheduled on the same Node. This does not call for the creation of another launcher Pod; the semantics in

llm-d-fast-model-actuation/api/fma/v1alpha1/launcherpopulationpolicy_types.go

Lines 38 to 40 in 886b603

	// the number of launchers that should exist is the larger of
	// (a) what `PopulationPolicy` says for that pair, and
	// (b) the number needed to satisfy the server-requesting Pods.

uses a max of two numbers for the count of launchers, not two disjoint pools of launchers. The dual-pods controller will connect that server-requesting Pod with one of the already-existing launcher Pods. After that association is made and the inference server is launched, suppose that some time later the LauncherCount is reduced to 0. That does not call for the deletion of the launcher Pod --- because the server-requesting Pod is still using it; that's part (b) of

llm-d-fast-model-actuation/api/fma/v1alpha1/launcherpopulationpolicy_types.go

Lines 38 to 40 in 886b603

	// the number of launchers that should exist is the larger of
	// (a) what `PopulationPolicy` says for that pair, and
	// (b) the number needed to satisfy the server-requesting Pods.

OK, lets continue the scenario. Suppose all the 2 launcher pods are associated with existing 2 server-requesting pods, and the new server-requesting pod is created, what it should do? not call or call for the creation of another launcher pod?

Originally posted by @osswangxining in #195 (comment)

[MikeSpreitzer]

Suppose all the 2 launcher pods are associated with existing 2 server-requesting pods, and the new server-requesting pod is created, what it should do?

This calls for creation of another launcher Pod, due to part (b) of the cited semantics. Again, a launcher Pod exists for either or both of two reasons.

[MikeSpreitzer]

We merged #195 because the review was getting too long, not because it was really "done". It was admittedly just a start on the logic anyway. There are several open discussions in that PR, which should get worked in follow-on PRs.

The main issue that needs further work is correctly implementing the semantics for the population of launcher Pods.

[MikeSpreitzer]

We also need to extend the structure of a launch request to include a list of GPU UUIDs.

[MikeSpreitzer]

We also need to define what we want to happen when various things get changed by (well, anybody).

• What happens to existing uses when a LauncherConfig changes?
• What happens to existing uses when an InferenceServerConfigSpec changes?
• What happens when a server-requesting Pod changes what it is requesting?
• What happens if somebody changes one of the annotations that our implementation uses to link things together?

[MikeSpreitzer]

Regarding how to cope with mutations of relevant API objects, following are some questions and possible answers.

Q1: What happens if somebody changes one of the annotations that our implementation uses to link things together?

We will use Kubernetes mechanisms to prevent it. E.g., a validating webhook (or, bettter yet, a CEL based validation)

Q2: What happens when a server-requesting Pod changes what it is requesting?

We will use Kubernetes mechanisms to prevent that while the Pod is bound to a server-providing Pod. Changes before the binding are not problematic, those are allowed.

Q3: What happens to existing launcher Pods when a LauncherConfig changes?

We have discussed three possible answers.

A1: No change to any existing launcher Pod.

A2: Each corresponding launcher Pod with zero children is deleted and replaced.

A3: Each corresponding launcher Pod that is not bound to a server-requesting Pod is deleted and replaced.

Q4: What happens to existing uses when an InferenceServerConfigSpec changes?

We have discussed two possible answers.

A1: No changes to existing inference servers.

A2: Each corresponding inference server that is sleeping is deleted.

A3: Each corresponding inference server that is sleeping is deleted and replaced.

[MikeSpreitzer]

We should agree on how to use https://github.com/go-logr/logr/blob/v1.4.2/logr.go#L308 --- what level to use for what. So far we have been following an outline like the following. These are designed to roughly align with the usage of V in the Kubernetes libraries.

• V(0): Messages that appear O(1) times per run.
• V(1): Messages that appear O(1) times per run, higher verbosity (more messages, more details).
• V(2): Messages that appear O(1) times per lifecycle of a major kind of object.
• V(3): Messages that appear O(1) times per lifecycle of a major kind of object, higher verbosity.
• V(4): Messages that appear O(1) times per minor object, conjunction of major objects, or update to a major object.
• V(5): Messages that appear O(1) times per minor object, conjunction of major objects, or update to a major object --- higher verbosity

[MikeSpreitzer]

My guess is that RapidLLMServe's behavior is closest to A3 for Q3, A2 for Q4. @osswangxining is that right?

[osswangxining]

My guess is that RapidLLMServe's behavior is closest to A3 for Q3, A2 for Q4. @osswangxining is that right?

Yes, this looks simpler and clearer. We can further enrich it for futher requirements.

[aavarghese]

For Q3, in the case of what's specified in A3 (Each corresponding launcher Pod that is not bound to a server-requesting Pod is deleted and replaced), replacement would mean that we also launch the same number of sleeping children as before (trying on the same GPUs)? Could there be a case where the new LauncherConfig is incompatible with the sleeping vLLM instances?