Feature request: Be able to pick your runtimeClassName.

[cdrage]

Is your feature request related to a problem? Please describe.

Related to: llm-d/llm-d-deployer#273

Unfortunatley, nvidia is not my default runtime class, and I'm unable to set it / override the deployments to use runtimeClassName nvidia.

This is because the controller (llm-d.ai/v1alpha1) keeps overriding it / patching it and not allowing me to use the runtime class :(

Describe the solution approach you'd like

I'm requesting that the CRD is updated to include being able to supply your own runtimeclass name so we can more easily specify which one (nvidia, nvidia-experimental, etc.).

Describe alternatives you've considered

I tried patching the deployment, but it's overridden by the controller.

Additional context

N/A

[jgchn]

Hi @cdrage thanks for opening this issue! I read through llm-d/llm-d-deployer#273, and I think the runtimeClassName would probably be a configuration on the deployer side. RuntimeClassName can be part of the baseconfig labels for deployments, like here. If present, then those labels will get merged with the labels the controller sets up for routing. Thoughts? @tumido

But I think you raised a good point. Currently, the controller owns all fields of its child resources. There might be some fields where we'd want to allow the user to apply patches, like labels and annotations.

[sriumcp]

Could you please share the msvc and baseconfig you are using? AFAIK, controller doesn't hardcode runtimeclassname anywhere (not sure what it refers to). A search for runtimeclassname returns a single result in this repo, and that's this issue!

---

For now, a simple fix might be to create your own baseconfig to refer inside the msvc.

[tumido]

I also think this should come from the presets we compose on the deployer side, rather than this necessarily being a feature of the model service controller. It should already be possible to set that via a baseconfig.