Outline of changes from milestone 2 to 3

[MikeSpreitzer]

Software engineering

We have a choice to make about how to develop m3 while maintaining a working m2. I can see three obvious possible approaches.

1. Use different branches. E.g., copy current m2 to branch milestone-2 and start developing m3 on branch main.
2. Use different Go packages. E.g., leave pkg/controller/dual-pods/ as it is, holding m2, and copy it into new package pkg/controller/dual-pods-m3 and develop from there. Remember, m3 changes not only the controller but all the recipes and scripts that use it.
3. Introduce an option into the existing controller. E.g., a command line flag --use-launcher that takes a Boolean value.

I currently favor approach 3, because I think it will be least disruptive. I expect there may be some on-going maintenance for m2, and I hate maintaining two copies of (more or less) the same code. Also, the change from m2 and m3 can be fairly smooth, in terms of the code in the controller, I think.

Changes to the dual-pods controller

Authoritative store of binding state

In m2 this is an annotation on the server-providing Pod. I think that this can remain.

Index into sleeping vLLM instances

In m2 this is an index maintained by the Pod informer. We can add an additional index into launched vLLM instances, in the controller's data structure. In nodeData, have a map from server-providing Pod name to *launcherData. Let launcherData have a map from nominal hash to last-used time. Use the nominal hash as the instance ID in the launcher. There is no need for a launcher to have multiple instances with the same nominal hash.

Other data/logic notes

Let a launcher-based server-providing Pod have an annotation that says it is such a thing. Let the controller's handler for Pod notifications queue a reference to the Pod when notified of such a thing.

Let launcherData have a boolean indicating whether the set of nominal hash is accurate.

Let the controller use the extension of generic controller that knows whether the initial load of objects has been processed.

When syncing a launcher-based server-providing Pod, create and populate the launcherData if it is not already present. If it is present but not accurate, update it to become accurate.

When it is time to create a new vLLM instance, use either the m2 code or the m3 code, as indicated by the option.

Before creating a launcher-based vLLM instance, check that every lancherData of the Pod has an accurate set of nominal hash. If not then enqueue a reference to the Pod and consider this a transient failure.

Each launcher has a personality. Create a new instance using only a launcher with the right personality.

Create a new launcher when needed. Delete an old launcher when its set of instances is empty and it has not been used for two minutes.

When the call to create or delete a launched instance fails with a networking problem, this controller considers this a transient failure and marks the launcherData as being uncertain about the correctness of the map of instances.

[waltforme]

Each launcher has a personality. Create a new instance using only a launcher with the right personality.

May I have a few more words about what is 'personality'?

[MikeSpreitzer]

Personality is the immutable stuff in a PodSpec. E.g., container image, volumes.

[MikeSpreitzer]

Readiness Relay

Instead of taking readiness from the server-providing Pod, the dual-pods controller has to explicitly poll the readiness query of vLLM.

When the readiness request fails with a networking error, the controller should follow up by asking the launcher whether the child is still running.

[MikeSpreitzer]

Port Management

(copied from #142 (comment))

Regarding port management, I envision two stages of development.

1. The model deployer, who also deploys the InferencePool objects, has different port numbers in different InferencePool objects. The nominal service-providing Pod listens for inference requests on the right port for the corresponding InferencePool. The dual-pods controller and the launcher are not involved in port management.

2. This later stage of development happens after GIE and llm-d have been generalized so that the port number can come from an annotation on the server-providing Pod instead of a literal in the InferencePool. At this time there is no longer a constraint of InferencePools having disjoint lists of port numbers. The dual-pods controller will take charge of the port number options on vLLM, ensuring that distinct children of a given launcher listen on distinct ports. The dual-pods controller will be configured with a range of port numbers to use for this purpose. The launcher remains uninvolved in port management.

[waltforme]

Some thoughts on port management.

We may introduce a precedence between our proposed annotation-based port management and the existing inferencepool-based port management, instead of replacing the inferencepool-based port management.

For example, if an annotation which expresses 'target port(s)' exists on a server-providing pod, use the expressed port(s). Otherwise, fall back to the inferencepool-specified TargetPorts.

This precedence could

1. keep the backward compatibility of the InferencePool API;
2. make our upstream changes less intrusive: We only change the EPP and keep the InferencePool API untouched.

[MikeSpreitzer]

... after GIE and llm-d have been generalized so that the port number can come from an annotation on the server-providing Pod ...

That "can" is deliberately only "can", not "must". In IETF words, it means "MAY" not "MUST". We definitely would not propose a change that required everyone doing existing use cases to change!

Note: this IS an API change. An API includes the contract between both sides (producers and consumers). We are proposing a change in the meaning of an InferencePool object.

Note: "backward compatible" has an implicit assumed direction: which side (author of InferencePool, or consumer?) is backward compatible with which side? When we are proposing a change to an interface, we have to think about compatibility in both directions. What happens when a new InferencePool is consumed by an old consumer? What happens when an old InferencePool is consumed by a new consumer?

The case of "old InferencePool is consumed by a new consumer" is easy. The case of "new InferencePool is consumed by an old consumer" is harder. I am going to leave it up to @lionelvillard to suggest an answer to that.

[MikeSpreitzer]

How Inference Servers Are Defined

In milestones 1 and 2 we took the approach of having the server-requesting Pod have a .metadata.annotation whose value is a "server patch" that derives the nominal server-providing Pod from the server-requesting Pod. This particular interface was chosen in order to minimize the disruption to model deployers. The server-requesting Pod can be nearly an ordinary server-providing Pod, it only needs to differ in having a requester container instead of an inference server container --- the rest of the Pod definition (e.g., volumes) can be the same.

In milestone 3 this interface gets a little more challenging to implement. The dual-pods controller has to factor the nominal server-providing Pod into (1) the actual server-providing Pod (i.e., a Pod that runs the launcher) and (2) the create-instance request to the launcher. This should be fairly do-able because the create-instance request to the launcher has a defined schema and it is small. The controller can dissect what is needed there from the nominal server-providing Pod, and what remains (plus injection of the launcher, whose details can be configuration data for the controller) becomes the actual server-providing Pod.

In contrast, the RapidLLMServe project took a different approach, in which the users of that system do the factoring and put the two factors into different custom resources. A ModelServerConfig defines the arguments to the launcher, while a ModelServerPool both (a) defines a set of launcher Pods and (b) carries the parameters of some policy about managing that set.

In FMA we could take a similar approach, defining two custom resources to hold the two factors: launcher's create-instance parameter values and definition of launcher Pod. Since we are locating pool management issues elsewhere, that would not be part of those two custom resources. As before in FMA: it is the model deployer's prerogative to manage the server-requesting Pods, and the dual-pods controller manages the server-providing Pods in a reactive and opportunistic way. The dual-pods controller has a policy for deleting sleeping vLLM instances when needed to free up GPU memory, and will have a policy for deleting launchers that are no longer of value. vLLM instances are created when needed, and launcher Pods are created when needed for that.

This leaves room for pro-active creation of vLLM instances and pro-active creation of launchers. The former is costly: creating a vLLM instance involves actually running it, including all the startup costs for the chosen LLM. Since it is so costly, responsibility for making these decisions should be located in something doing global planning/optimization for the cluster. Pro-active creation of launchers is much cheaper. I do not think we have yet agreed on what sort of policy to use for that nor how it will be expressed. It is possible that we might put the control knobs for that into the custom resource the defines a launcher Pod.

Software Engineering

If we pivot to using custom resources instead of a server patch annotation, this is a bigger change. It might behoove us to use different Git branches for m3 development and m2 maintenance.

[MikeSpreitzer]

A point in favor using custom resources rather than a server patch annotation is that the custom resource defining launcher pods has a lifecycle independent of any particular server-requesting Pod and therefore provides a more appropriate thing with which to associate pro-active creation of those launchers.

[osswangxining]

A point in favor using custom resources rather than a server patch annotation is that the custom resource defining launcher pods has a lifecycle independent of any particular server-requesting Pod and therefore provides a more appropriate thing with which to associate pro-active creation of those launchers.

Some thoughts (also put into the slack):

1. Introduce two new CRDs—InferenceServerTemplate and InferenceServerConfig— to manage the nominal server-providing pod definition and the configuration parameters required to launch the vLLM process inside the launcher pod.

2. The nominal server-providing pod supports multiple types. Currently, we prioritize two modes:

• Mode A (direct): Launch the vLLM instance directly in a container without using a launcher—i.e., the current standard vLLM engine deployment approach.
• Mode B (launcher): Use a launcher process to start vLLM engine subprocesses—this aligns with the approach used in the RapidLLMServe or FMA project. Here, a key point is that each server-providing pod may contain at most one running vLLM engine at any time. Zero or more sleeping vLLM engines may coexist, and the maximum number of sleeping instances should be defined in the InferenceServerTemplate custom resource.

3. A server-requesting pod, together with one InferenceServerTemplate CR and one InferenceServerConfig CR, jointly determine how a server-providing pod is created. The exact creation logic depends on the following cases:

3.1) If the InferenceServerTemplate CR explicitly specifies mode: direct, then a pod running the vLLM engine directly is created by combining the template defined in the CR with the server-requesting pod. In this case, the InferenceServerConfig CR is not required.

3.2) If the InferenceServerTemplate CR specifies mode: launcher, the following prioritized steps are applied:

• Reuse an existing Launcher Pod that has no running vLLM engine subprocess, but does have a sleeping vLLM engine subprocess whose parameters match those in the InferenceServerConfig CR → wake up that sleeping subprocess.
• Reuse an existing Launcher Pod that has no running vLLM engine subprocess, and launch a new vLLM engine subprocess using the parameters from the InferenceServerConfig CR.
• Create a new Launcher Pod based on the template in the InferenceServerTemplate CR (combined with the server-requesting pod), then launch a new vLLM engine subprocess using the parameters from the InferenceServerConfig CR.

[aavarghese]

Hi @osswangxining and @MikeSpreitzer. CC: @rubambiza @lionelvillard
Thank you for the very detailed write-up! In addition to the separate CRDs with the template and the config, we also discussed adding a policy to fill the gap in how idle launchers could proactively be provisioned across nodes before requests start to come in. This could be another (third?) lightweight CRD or maybe should be part of the InferenceServerTemplate CRD (with a different name).

RapidLLMServe has this:

IdleEnginePilots *int32 `json:"idleEnginePilots,omitempty"`

We propose something like:

spec:
  idlePoolPolicy:
    poolsPerNode:
      - selector:
            acceleratorType: "nvidia.com/a100"
            minMemoryGB: 80
            acceleratorCount: 8
        launcherCount: 2
      - selector:
            acceleratorType: "nvidia.com/l40"
            minMemoryGB: 40
            acceleratorCount: 4
        launcherCount: 1

Gives us a predictable supply of launcher pods when running mode: launcher sized according to GPU memory and based on an InferenceServerTemplate. The launcher policy/launcher provisioner controller could be another sub-component or process within our DPC controller.