Migrate from model-service as a component to dependency chart

[Gregory-Pereira]

Component

I don't know

Desired use case or feature

For composability and with the goal in mind of taking a non-opinionated stance, we would like to remove the modelservice as a component in the deployer helm charts, and rather levarage the new model service helm charts

Proposed solution

Use the modelservice helm chart as a dependency in the deployer chart

Alternatives

No response

Additional context or screenshots

cc @sriumcp

[sriumcp]

@Gregory-Pereira Why would we want the modelservice helm chart as a dependency in the deployer? I don't it should be...

Essentially, the deployer will deploy llm-d infra components, and when it is time to deploy a model, the model-owner will do a helm install <name-of-the-model-release> model-service-chart-name ...... once per model deployment.

[jgchn]

Yes, we've come to the consensus that the deployer should not depend on model service. Instead, Quickstart will guide users to install one model service release.

So perhaps the following steps

Quickstart @nerdalert

1. In QuickStart/llmd-installer.sh, replace modelservice CRD installation with a pinned version of helm repo add llm-d-modelservice
2. After llmd-installer.sh, apply a helm release of model service. Perhaps the only thing to maintain in llm-d-deployer is a modelservice.values.yaml file. Applying this will create the routing + vllm deployments
3. In "Customizing your deployment" section, link to model service repo where examples can be found. We can provide a concise explanation of how to add another model release, but users can go to the model service repo so we don't have to maintain examples in the deployer repo simultaneously.

After the above, we can update the llm-d-deployer helm chart

1. Remove modelservice object --> this means llm-d contains only the infra components and nothing specific to models.
2. Maybe we can also remove sampleApplication?

WDYT?

[sriumcp]

3 looks super important. Quickstart probably should have the simplest example using modelservice. The rest of the samples can be maintained in the modelservice repo.

[sriumcp]

Remove modelservice object --> this means llm-d contains only the infra components and nothing specific to models. Maybe we can also remove sampleApplication?

The above looks right to me.

[jgchn]

Removing model service as a dependency from llm-d also means that serving 1+ base models is supported. We've tested this out which works with Istio. So the idea is that we can have many model service releases on top of llm-d.

[Gregory-Pereira]

I believe we should deal with the modelservice chart migration at the same time as the migration to the upstream GAIE charts because those manifests (EPP, EPP SA, EPP RBAC, inferencepool, etc.) were created by the modelservice controller reconciliation on CR, but post migration will simply be removable from the list of templates in modelservice.
As for how we do this I believe:
- Existing examples within deployer will be converted to helmfiles with many values files for the different installations of the charts (deployer, modelservice, and upstream GAIE). This will support multiple installations of the modelservice and GAIE charts but will be capped at 1 of each per namespace.
- Most of the content of the deployer charts will be moved to the modelservice charts with the exception of the Gateway, GatewayParameters, and ingress
- A significant portion of the install scripts will need to move to modelservice charts as well. This will end up with 2 sets of quickstart directories (1 per project). The deployer Quickstart will still take care of things like deploying grafana infrastructure for llm-d obersability if desired and owning the full deployment examples, while components like validating the HF token and creating a secret for it will be moved to modelservice charts as these will need to live in every namespace for which there will be an instance of the inferencing stack (IE combo of GAIE chart installation + modelservice chart installation)
- Based on this discussion I agree that we wont need modelservice as a dependency of deployer. If anything, we should have the deployer charts be a dependency of Modelservice