Problem
The LLM topology has a powerful escape hatch: the config field on VllmCustomSpec accepts raw YAML that the provider materializes into an LLMInferenceServiceConfig and attaches as a baseRef. This allows users to set any field in the KServe API without waiting for provider updates.
The predictor topology has no equivalent escape hatch. If a user needs a KServe InferenceService feature that the provider doesn't expose (e.g., custom liveness probes, batcher, logger, multi-container predictor, advanced networking), they are completely blocked. They must either:
- Wait for a provider update.
- Create the
InferenceService manually via kubectl, bypassing OpenEverest entirely.
Suggested approach
Add a rawConfig or config field to ModelServerCustomSpec (or predictor topology parameters) that accepts a raw InferenceService spec fragment. The provider would:
- Parse the raw YAML as
serving.kserve.io/v1beta1.InferenceServiceSpec.
- Deep-merge the user-provided spec with the provider-generated spec (user overrides win).
- Validate that the merged result is still a valid
InferenceService.
Alternatively, support a pointer to an externally defined InferenceService or InferenceServiceConfig-like resource.
Why this matters
- KServe's v1beta1 API is large and evolves frequently. The provider will always lag behind.
- Teams with unique requirements need flexibility.
- The LLM topology already proved this pattern works well.
Acceptance criteria
Problem
The LLM topology has a powerful escape hatch: the
configfield onVllmCustomSpecaccepts raw YAML that the provider materializes into anLLMInferenceServiceConfigand attaches as a baseRef. This allows users to set any field in the KServe API without waiting for provider updates.The predictor topology has no equivalent escape hatch. If a user needs a KServe
InferenceServicefeature that the provider doesn't expose (e.g., custom liveness probes, batcher, logger, multi-container predictor, advanced networking), they are completely blocked. They must either:InferenceServicemanually via kubectl, bypassing OpenEverest entirely.Suggested approach
Add a
rawConfigorconfigfield toModelServerCustomSpec(or predictor topology parameters) that accepts a rawInferenceServicespec fragment. The provider would:serving.kserve.io/v1beta1.InferenceServiceSpec.InferenceService.Alternatively, support a pointer to an externally defined
InferenceServiceorInferenceServiceConfig-like resource.Why this matters
Acceptance criteria
configorrawConfigfield for raw InferenceService YAML.configbehavior is documented for parity reference.