Skip to content

Add custom ServingRuntime overrides: image, protocolVersions, imagePullSecrets #19

Description

@tartkri-bot

Problem

The predictor topology supports runtimeVersion and runtime (by name), but does not allow:

  1. Custom runtime image: Override the image URI of the selected ServingRuntime per Instance.
  2. Custom protocol versions: Specify protocolVersions (e.g., only v2 for gRPC, only v1 for REST).
  3. Image pull secrets: imagePullSecrets for pulling from private registries.
  4. Resource limits on init containers: The storage initializer init container runs with default resource requests; users may want to tune it or the model server's resources independently.

This is especially relevant for teams who:

  • Maintain a patched/custom build of Triton / vLLM / sklearn.
  • Use private registries (e.g., Harbor, ECR with pull-through cache).
  • Need specific protocol versions for their client SDKs.

Suggested approach

Add to ModelServerCustomSpec:

  • runtimeImage — string, overrides the runtime image for this instance.
  • protocolVersions — []string (e.g., ["v1"], ["v1", "v2"]).
  • imagePullSecrets — []corev1.LocalObjectReference.
  • storageInitializerResources — corev1.ResourceRequirements (optional, to tune the init container).

For LLM topology, add equivalent fields or document how to achieve this via the config escape hatch.

Acceptance criteria

  • runtimeImage overrides the KServe ServingRuntime image for the predictor.
  • protocolVersions restricts the serving protocol.
  • imagePullSecrets is propagated to the InferenceService spec.
  • storageInitializerResources is propagated when disableStorageInitializer is false.
  • Chainsaw tests cover custom image override and imagePullSecrets.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions