[Feature]: Configurable OpenAI API exposure + inference load balancing

[atharvamhaske]

Summary

Think of this as persona: a Developer chooses how to expose the OpenAI-compatible API (ClusterIP, NodePort, LoadBalancer) and which inference load-balancing strategy to use (Least Load, Prefix Hash). The provider defines a proxy resource and follows KubeAI’s load-balancing model so apps get a stable /openai/v1/... endpoint without hand-editing Service/proxy YAML.

Why

Exposure and LB strategy are not first-class in the provider. Teams need a clear way to publish the API and to pick LB suited to vLLM (even load vs prefix-cache affinity for chat).

Scope

• Proxy / exposure setting in the provider (Service type: ClusterIP | NodePort | LoadBalancer).
• Wire Least Load and Prefix Hash per KubeAI load-balancing concepts.
• Surface choices in Instance/topology UI and/or Helm defaults.
• Sync applies exposure + LB without thrashing Model spec.replicas or breaking scale-to-zero.
• Docs: how to expose the API and when to use each LB strategy.

Checklist

• Developer can select API Service type: ClusterIP / NodePort / LoadBalancer.
• Provider configures proxy path per KubeAI load-balancing.
• Least Load and Prefix Hash are selectable (or documented defaults) for supported endpoints.
• Chat/completions works with chosen LB; Prefix Hash targets prefix-cache friendly routing.
• Docs cover exposure + LB behavior.

Additional notes

Prefix Hash is most useful for chat (full thread sent each request). Supported endpoints include /openai/v1/completions and /openai/v1/chat/completions.

Resources

• https://www.kubeai.org/concepts/load-balancing/
• https://www.kubeai.org/reference/kubernetes-api/