Problem Statement
Based of #205
I want to be able to fulfill requests specifying one model with another model instead of having those requests error when sent to the backend/IGW.
Proposed Solution
model_name_override can be specified for a Pool object.
If model_name_override is not empty, every request processed will have the model name replaced in the inference request with the override name. The original name will be stored as internal metadata on the request and be replaced back into the response before placing into the result channel.
Alternatives Considered
Leaving model overriding to inference-gateway backend:
https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencemodelrewrite/
the problem with this solution is it's incompatible if async-processor is not using IGW. IGW is not a hard dependency for async-processor. Async-processor can be used with model servers directly.
Willingness to Contribute
Yes, I can submit a PR
Additional Context
No response
Problem Statement
Based of #205
I want to be able to fulfill requests specifying one model with another model instead of having those requests error when sent to the backend/IGW.
Proposed Solution
model_name_override can be specified for a Pool object.
If model_name_override is not empty, every request processed will have the model name replaced in the inference request with the override name. The original name will be stored as internal metadata on the request and be replaced back into the response before placing into the result channel.
Alternatives Considered
Leaving model overriding to inference-gateway backend:
https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencemodelrewrite/
the problem with this solution is it's incompatible if async-processor is not using IGW. IGW is not a hard dependency for async-processor. Async-processor can be used with model servers directly.
Willingness to Contribute
Yes, I can submit a PR
Additional Context
No response