Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 44 additions & 10 deletions pkg/epp/framework/plugins/datalayer/discovery/file/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ datastore on each change.
## How It Works

- **Initial load.** On `Start`, the file is read once. Each entry is
validated (address must be a parseable IP, port must be in `[1, 65535]`)
and applied via `notifier.Upsert`. Per-entry validation errors are logged
and the entry is skipped; file-level problems (open, parse, size > 1 MiB)
abort startup.
validated (address must be a valid IPv4 or RFC 1123 hostname, port must be
in `[1, 65535]`) and applied via `notifier.Upsert`. Per-entry validation
errors are logged and the entry is skipped; file-level problems (open,
parse, size > 1 MiB) abort startup.
- **Reload (optional).** When `watchFile: true`, fsnotify Write / Create /
Remove events trigger a reload. After an atomic rename or ConfigMap-style
symlink swap (which destroys the inode being watched), the watcher is
Expand All @@ -42,12 +42,27 @@ plugin's `path` parameter.
endpoints:
- name: <string> # required -- unique within the file
namespace: <string> # optional -- defaults to "default"
address: <IPv4> # required -- must be a valid IPv4 address
address: <string> # required -- IPv4 address or RFC 1123 hostname
port: <string> # required -- integer 1-65535 as a string
metricsPort: <string> # optional -- metrics scrape port (defaults to port)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this value reflected into the datalayer's metrics collection?
An alternative path might be to define a new data source to scrape remote endpoints by providing a label/tag with the hostname:port.
I think that the current endpoints assume an inference server and you are scraping remote clusters (EPPs, I presume?), so the use of the current metrics data source and extractors seem at odds with your design/goal.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, metricsPort feeds into MetricsHost which the metrics scraper uses to connect. You're right that the current core-metrics-extractor assumes vLLM-format metrics - that's why a spoke-epp engine type is being added in PR #1928 to handle the different metric format from spoke EPPs. This PR only adds discovery support; it intentionally does not modify the metrics extraction logic.

@elevran elevran Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming.
I would prefer we don't overload the engine type for this but instead define a new Extractor for EPP metrics (which are not the same as inference engine metrics). You are essentially extracting different metrics from different objects for slightly different use cases and the overload can be confusing (e.g., introduce logic into existing components to separate server and cluster instead of just creating a different extractor for cluster).
Left comments on #1928 with more detail.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @elevran , les put the extra effort in here to make this feel more native and idiomatic.

labels: # optional -- arbitrary key/value labels
<key>: <value>
```

When `address` is an IPv4, the endpoint is treated as a pod (`PodName` is set
to the entry name). When `address` is a hostname, the endpoint is treated as a
cluster (`PodName` is empty).

### Endpoint Type Label

Each endpoint is tagged with the label `llm-d.ai/endpoint-type` so downstream
plugins can distinguish pod endpoints from cluster endpoints explicitly. The
label can be set in the endpoints file; when omitted, the plugin auto-detects
it from the address format:

- IPv4 address → `llm-d.ai/endpoint-type: pod`
- Hostname → `llm-d.ai/endpoint-type: cluster`

## Configuration

**Location:** `dataLayer.discovery.pluginRef` referencing a plugin entry of
Expand Down Expand Up @@ -75,7 +90,7 @@ dataLayer:
pluginRef: file-discovery
```

A two-endpoint file referenced by the config above:
A two-endpoint file with pod IPs:

```yaml
endpoints:
Expand All @@ -87,13 +102,32 @@ endpoints:
port: "8000"
```

A cluster endpoints file with host names instead of IP addresses:

```yaml
endpoints:
- name: cluster-us-east
address: spoke-us-east.example.com
port: "443"
metricsPort: "9090"
labels:
region: us-east
- name: cluster-eu-west
address: spoke-eu-west.example.com
port: "443"
metricsPort: "9090"
labels:
region: eu-west
```

## Limitations

- The endpoints file is capped at 1 MiB.
- `address` must be a literal IPv4 address. Hostnames are not resolved;
IPv6 is not supported.
- Metrics are scraped from `address:port` (same host and port that serves
inference); separate metrics endpoints are not supported.
- `address` must be a valid IPv4 address or an RFC 1123 hostname. IPv6 is
not supported. Hostnames are not resolved by the plugin; DNS resolution
happens at scrape/connect time.
- Metrics are scraped from `address:metricsPort` (or `address:port` when
`metricsPort` is not set).
- File-discovery mode runs the EPP without a Kubernetes controller manager,
so several K8s-only features are inactive: the `InferenceModelRewrite`
and `InferenceObjective` reconcilers do not run, and any
Expand Down
56 changes: 44 additions & 12 deletions pkg/epp/framework/plugins/datalayer/discovery/file/plugin.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,26 +28,36 @@ import (
"os"
"sort"
"strconv"
"strings"
"sync"

"github.com/fsnotify/fsnotify"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/validation"
"sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/yaml"

fwkdl "github.com/llm-d/llm-d-router/pkg/epp/framework/interface/datalayer"
fwkplugin "github.com/llm-d/llm-d-router/pkg/epp/framework/interface/plugin"
)

const PluginType = "file-discovery"
const (
PluginType = "file-discovery"

// EndpointTypeLabel distinguishes pod from cluster endpoints.
EndpointTypeLabel = "llm-d.ai/endpoint-type"
EndpointTypePod = "pod"
EndpointTypeCluster = "cluster"
)

// EndpointEntry is the YAML/JSON representation of a single endpoint.
type EndpointEntry struct {
Name string `json:"name" yaml:"name"`
Namespace string `json:"namespace,omitempty" yaml:"namespace,omitempty"`
Address string `json:"address" yaml:"address"`
Port string `json:"port" yaml:"port"`
Labels map[string]string `json:"labels,omitempty" yaml:"labels,omitempty"`
Name string `json:"name" yaml:"name"`
Namespace string `json:"namespace,omitempty" yaml:"namespace,omitempty"`
Address string `json:"address" yaml:"address"`
Port string `json:"port" yaml:"port"`
MetricsPort string `json:"metricsPort,omitempty" yaml:"metricsPort,omitempty"`
Labels map[string]string `json:"labels,omitempty" yaml:"labels,omitempty"`
}

// EndpointsFile is the top-level structure of the endpoints YAML/JSON file.
Expand Down Expand Up @@ -238,9 +248,12 @@ func (f *FileDiscovery) load(notifier fwkdl.DiscoveryNotifier) error {
incoming := make(map[types.NamespacedName]struct{}, len(ef.Endpoints))
var errs []error
for _, e := range ef.Endpoints {
if ip := net.ParseIP(e.Address); ip == nil || ip.To4() == nil {
errs = append(errs, fmt.Errorf("endpoint %q: invalid IPv4 address %q", e.Name, e.Address))
continue
ip := net.ParseIP(e.Address)
if ip == nil || ip.To4() == nil {
if len(validation.IsDNS1123Subdomain(strings.ToLower(e.Address))) > 0 {
errs = append(errs, fmt.Errorf("endpoint %q: address %q is not a valid IPv4 or hostname", e.Name, e.Address))
continue
}
}
if portNum, err := strconv.Atoi(e.Port); err != nil || portNum < 1 || portNum > 65535 {
errs = append(errs, fmt.Errorf("endpoint %q: invalid port %q", e.Name, e.Port))
Expand All @@ -250,13 +263,32 @@ func (f *FileDiscovery) load(notifier fwkdl.DiscoveryNotifier) error {
if ns == "" {
ns = "default"
}
podName := e.Name
if ip == nil {
podName = ""
}
mPort := e.Port
if e.MetricsPort != "" {
mPort = e.MetricsPort
}
labels := e.Labels
if labels == nil {
labels = make(map[string]string)
}
if v := labels[EndpointTypeLabel]; v != EndpointTypePod && v != EndpointTypeCluster {
if ip == nil {
labels[EndpointTypeLabel] = EndpointTypeCluster
} else {
labels[EndpointTypeLabel] = EndpointTypePod
}
}
meta := &fwkdl.EndpointMetadata{
NamespacedName: types.NamespacedName{Name: e.Name, Namespace: ns},
PodName: e.Name,
PodName: podName,
Address: e.Address,
Port: e.Port,
MetricsHost: net.JoinHostPort(e.Address, e.Port),
Labels: e.Labels,
MetricsHost: net.JoinHostPort(e.Address, mPort),
Labels: labels,
}
incoming[meta.NamespacedName] = struct{}{}
notifier.Upsert(meta)
Expand Down
Loading
Loading