Skip to content

Add raw InferenceService config escape hatch for predictor topology #22

Description

@tartkri-bot

Problem

The LLM topology has a powerful escape hatch: the config field on VllmCustomSpec accepts raw YAML that the provider materializes into an LLMInferenceServiceConfig and attaches as a baseRef. This allows users to set any field in the KServe API without waiting for provider updates.

The predictor topology has no equivalent escape hatch. If a user needs a KServe InferenceService feature that the provider doesn't expose (e.g., custom liveness probes, batcher, logger, multi-container predictor, advanced networking), they are completely blocked. They must either:

  • Wait for a provider update.
  • Create the InferenceService manually via kubectl, bypassing OpenEverest entirely.

Suggested approach

Add a rawConfig or config field to ModelServerCustomSpec (or predictor topology parameters) that accepts a raw InferenceService spec fragment. The provider would:

  1. Parse the raw YAML as serving.kserve.io/v1beta1.InferenceServiceSpec.
  2. Deep-merge the user-provided spec with the provider-generated spec (user overrides win).
  3. Validate that the merged result is still a valid InferenceService.

Alternatively, support a pointer to an externally defined InferenceService or InferenceServiceConfig-like resource.

Why this matters

  • KServe's v1beta1 API is large and evolves frequently. The provider will always lag behind.
  • Teams with unique requirements need flexibility.
  • The LLM topology already proved this pattern works well.

Acceptance criteria

  • Predictor topology supports a config or rawConfig field for raw InferenceService YAML.
  • Provider deep-merges user config with generated config (structured fields still win).
  • Provider validates the merged result and reports parse/validation errors.
  • Documentation shows examples of advanced use cases (batcher, logger, custom probes).
  • Chainsaw tests verify that raw config overrides are applied correctly.
  • LLM topology config behavior is documented for parity reference.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions