Skip to content

Fix predictor connection details when InferenceService Status.URL is nil #23

Description

@tartkri-bot

Problem

In statusPredictor, when an InferenceService becomes Ready but Status.URL is nil (which is the common case in RawDeployment mode, where KServe only creates an in-cluster ClusterIP Service), the provider returns controller.Ready() with no connection details at all. This means:

  • The user sees "Ready" but has no idea how to reach the model.
  • The OpenEverest UI/CLI cannot show an endpoint for the predictor.
  • External tools or other OpenEverest resources that depend on connection details (e.g., API Gateway routing, downstream consumers) get an empty host/port.

The LLM topology solves this elegantly by creating a provider-owned external Service (LoadBalancer or NodePort) when the user selects a non-ClusterIP type, and populating connection details from that. The predictor topology does no such thing.

Suggested approach

Align predictor with LLM:

  1. Read comp.Service.ServiceType for predictor topology (ClusterIP, LoadBalancer, NodePort).
  2. When LoadBalancer or NodePort is selected, create a provider-owned external Service fronting the predictor pods (similar to externalServiceSuffix logic in llminferenceservice.go).
  3. Populate connection details from the external Service's endpoint (or from Status.URL if KServe eventually populates it).
  4. When ClusterIP is selected, continue to return Ready() with no connection details (document this as "use port-forward or enable external exposure").

KServe predictor pods use the label selector kserve.io/inferenceservice: <name> — use that for the external Service's selector.

Acceptance criteria

  • Predictor topology supports ClusterIP, LoadBalancer, and NodePort via service.serviceType.
  • For LoadBalancer: provider creates an external Service of type LoadBalancer, waits for an external IP, and publishes it in connection details.
  • For NodePort: provider creates an external Service of type NodePort, publishes <node-ip>:<nodePort> in connection details.
  • For ClusterIP: returns Ready with no connection details (documented limitation).
  • Provider validates serviceType matches the LLM allowed set (no unsupported types).
  • Chainsaw tests verify connection details for each service type.
  • Existing LLM external Service behavior is unchanged.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions