Problem. Models that don't fit on a single node (beyond tensor parallelism) require KServe's multi-node head/worker layout, which the provider doesn't expose.
Impact. Largest models can't be served through the Instance API without hand-writing the full LLMInferenceService.
Current workaround. Inline config / baseRefs only.
Upstream. WorkloadSpec.Worker (presence triggers a distributed multi-node deployment; controller wires head/worker discovery).
Scope. A structured worker-group field (worker count + per-worker resources) on llmEngine.
Problem. Models that don't fit on a single node (beyond tensor parallelism) require KServe's multi-node head/worker layout, which the provider doesn't expose.
Impact. Largest models can't be served through the
InstanceAPI without hand-writing the fullLLMInferenceService.Current workaround. Inline
config/baseRefsonly.Upstream.
WorkloadSpec.Worker(presence triggers a distributed multi-node deployment; controller wires head/worker discovery).Scope. A structured worker-group field (worker count + per-worker resources) on
llmEngine.