"Pool manager" to create a set of pre-warmed/reusable inference server pods

[aavarghese]

...these inference server pods will run the launcher process that starts up a VLLM server and puts it to sleep.

What is the pool manager to be implemented in Go responsible for?

• warm inventory of inference server-running pods
• makes sure pool always has a min size of launcher pods ready to start vllm servers
• allocates a launcher pod to the dual pod controller when request comes in, and retrieves pod back when done (stop vllm server)

[MikeSpreitzer]

A Pod does not sleep. A launcher does not sleep. A vLLM engine sleeps.

What says what to pre-warm? That sounds like it is requiring guidance from a llm-d component that does not exist (predicting future needs for inference servers). I thought that we were pursuing a lazy approach (creating a new Pod when a need arises).

To be clear, since we are talking about vLLM engine sleep/wake here, we are talking about the "Swap sleeping vLLM" technique, in which one launcher can have several vLLM instances launched at once (most sleeping, maybe one awake).

I think that with model-swapping, we could maintain N launcher pods on a node with N accelerators. At a given time the number needed might be less than N, but it will never be more than N.

[jimcadden]

What says what to pre-warm? That sounds like it is requiring guidance from a llm-d component that does not exist (predicting future needs for inference servers). I thought that we were pursuing a lazy approach (creating a new Pod when a need arises).

We may want a way to populate a "fresh" cluster with sets of pre-warmed instances in anticipation of future hot starts. For example, "deploy a sleeping instance of model M to 4 GPUs on node N"

[MikeSpreitzer]

For example, "deploy a sleeping instance of model M to 4 GPUs on node N"

Who/what would say that?

There would need to be an API, between the user/client saying that and the implementation. As far as I know, this API has not been designed/reviewed.

[github-actions]

This issue is marked as stale after 90d of inactivity. After an additional 30d of inactivity (15d to become rotten, then 15d more), it will be closed. To prevent this issue from being closed, add a comment or remove the lifecycle/stale label.

[rubambiza]

This is satisfactorily implemented, but may be referenced in future issues for reference.