Problem
The predictor topology supports runtimeVersion and runtime (by name), but does not allow:
- Custom runtime image: Override the image URI of the selected ServingRuntime per Instance.
- Custom protocol versions: Specify
protocolVersions (e.g., only v2 for gRPC, only v1 for REST).
- Image pull secrets:
imagePullSecrets for pulling from private registries.
- Resource limits on init containers: The storage initializer init container runs with default resource requests; users may want to tune it or the model server's resources independently.
This is especially relevant for teams who:
- Maintain a patched/custom build of Triton / vLLM / sklearn.
- Use private registries (e.g., Harbor, ECR with pull-through cache).
- Need specific protocol versions for their client SDKs.
Suggested approach
Add to ModelServerCustomSpec:
runtimeImage — string, overrides the runtime image for this instance.
protocolVersions — []string (e.g., ["v1"], ["v1", "v2"]).
imagePullSecrets — []corev1.LocalObjectReference.
storageInitializerResources — corev1.ResourceRequirements (optional, to tune the init container).
For LLM topology, add equivalent fields or document how to achieve this via the config escape hatch.
Acceptance criteria
Problem
The predictor topology supports
runtimeVersionandruntime(by name), but does not allow:protocolVersions(e.g., onlyv2for gRPC, onlyv1for REST).imagePullSecretsfor pulling from private registries.This is especially relevant for teams who:
Suggested approach
Add to
ModelServerCustomSpec:runtimeImage— string, overrides the runtime image for this instance.protocolVersions— []string (e.g.,["v1"],["v1", "v2"]).imagePullSecrets— []corev1.LocalObjectReference.storageInitializerResources— corev1.ResourceRequirements (optional, to tune the init container).For LLM topology, add equivalent fields or document how to achieve this via the
configescape hatch.Acceptance criteria
runtimeImageoverrides the KServe ServingRuntime image for the predictor.protocolVersionsrestricts the serving protocol.imagePullSecretsis propagated to the InferenceService spec.storageInitializerResourcesis propagated whendisableStorageInitializeris false.