[Feature]: Multiple base model deployment

[asm582]

Problem Statement

To test WVA optimization strategies, we would like to add multiple base models in the system. This would include a single gateway, two epps, two inference pools, and a custom HTTPRoute with multiple backendRefs. We would like to know if such a scenario is supported by modelservice.

Proposed Solution

ModelService should be able to launch multiple base models.

Alternatives Considered

llm-d/llm-d-workload-variant-autoscaler#1014

Willingness to Contribute

Yes, with guidance

Additional Context

No response

[jgchn]

Hi @asm582!

Short answer is yes, this is supported, by two model service releases. Note that MS does not deploy inference pools, httpRoutes , and other GAIE-related artifacts. So you would have to configure them outside of MS. If you want to deploy alongside MS, there's an extraObjects field, but note that there is a sequencing issue with GAIE objects. I think HTTPRoute needs to be deployed prior to InferencePools. I'm not sure if this has been fixed.

For a multi-model scenario, it could look like this:

1. Deploy one HttpRoute with multiple backendRefs
2. Deploy two inferencePools + EPP (inferencePool chart)
3. Deploy two model service releases, make sure that the labels match to previous step

helm install model-pool-a llm-d-modelservice -f values.yaml \
  --set "modelArtifacts.labels.pool=pool-a"

helm install model-pool-b llm-d-modelservice -f values.yaml \
  --set "modelArtifacts.labels.pool=pool-b"