feat(extension-server): Hook to modify EDS endpoints before sending to Envoy

[codefromthecrypt]

Description

Extension server hooks exist for Clusters, Routes, VirtualHosts, and Listeners, but not for EDS (ClusterLoadAssignment). For EDS-type clusters (K8s Services, Backend with static IPs), LoadAssignment is nil at translation time and endpoints arrive via the EDS stream, bypassing all hooks. Extension servers cannot stamp per-endpoint metadata such as per_route_rule_backend_name that ai-gateway's ext_proc needs to resolve backends for OpenTelemetry spans and metrics.

Use case

ai-gateway stamps per_route_rule_backend_name on each endpoint so the upstream ext_proc can resolve the backend and populate OpenTelemetry spans and metrics with gen_ai.system (provider name), model, and token usage per backend.

Without endpoint metadata, the ext_proc receives no backend identity, so OTel data is missing or wrong.

    ext_proc resolves backend from endpoint metadata
         |
         v
    SetBackend(backend)
         |
         v
    metrics.SetBackend(backend)
         |
         v
    OTel span/metric attributes:
      gen_ai.system = "openai"    <-- derived from backend schema
      gen_ai.request.model = ...
      gen_ai.response.model = ...
      gen_ai.usage.* = ...

    When endpoint metadata is missing (EDS clusters):
      gen_ai.system = "unknown"   <-- backend unresolved

A cluster-level fallback exists (envoyproxy/ai-gateway#1822), but it only stamps BackendRefs[0], so multi-backend route rules still can't distinguish backends (see envoyproxy/ai-gateway#902, envoyproxy/ai-gateway#950).

Proposal

Add a new extension server hook (e.g., PostEndpointModify) that is called when ClusterLoadAssignment resources are built or updated, allowing extension servers to modify LocalityLbEndpoints metadata before they are sent to Envoy via EDS.

EG xDS Translation
     |
     v
+--------------------+        +---------------------+
| PostClusterModify  |        | EDS Stream          |
| PostTranslateModify|        | (ClusterLoadAssign) |
+--------------------+        +---------------------+
     |                               |
     v                               v
Cluster sent to Envoy         +---------------------+
                               | PostEndpointModify  |
                               +---------------------+
                                     |
                                     v
                               Endpoints sent to Envoy
                               (extension server CAN
                                stamp per-endpoint
                                metadata here)

Open Issues:

• Bring back Kubernetes Service support in AIServiceBackend's ref ai-gateway#902 — K8s Service BackendRef disabled; no endpoint metadata path
• Backend with static ip is not working ai-gateway#950 — Static IP Backend unresolvable; EDS clusters lack metadata

Related PRs:

• feature(translator): cluster and endpoint xds metadata #5875 — Propagated K8s metadata to Cluster and endpoint xDS
• fix: resolve backend name from cluster metadata when LoadAssignment is nil ai-gateway#1822 — Cluster-level metadata fallback; only resolves first backend per rule

[arkodg]

cc @guydc

[guydc]

If we're creating a dedicated hook, I would really like to see a way for users to mark the relevant endpoints that require mutation. For the general case, I imagine that users won't always need all endpoints to be mutated.

Some options:

• a custom backend resource that points to the actual service/backend.
• an extension server policy that attaches to backends?
• specific EG annotation on such resources?

[codefromthecrypt]

In the case of AI gateway, I think it would be all dataplane (LLM and MCP) backends (so not the incidental otel ones). @nacx you are more familiar with the arch and EG model, do you have advice on @guydc comment? by "users" I suspect he meant consuming projects like ai-gateway.

[zhaohuabing]

For the AI gateway use case, we can send endpoints for all backends. A few may not be needed by the AI gateway ext proc, but it can filter them server-side. If other use cases come up that require endpoints for only a subset, we can revisit and consider the options @guydc suggested. Would it be okay to defer this decision until we have clearer requirements for the filter?

[zhaohuabing]

Hi @codefromthecrypt This has been discussed in today’s community meeting.

Could we solve this via Service/Backend annotations instead of adding a new hook? Annotations prefixed with gateway.envoyproxy.io are already propagated into xDS metadata.

  apiVersion: v1
  kind: Service
  metadata:
    name: test-service
    annotations:
      gateway.envoyproxy.io/foo: bar
  spec:
    ports:
    - port: 8080
      name: http

    metadata:
      filterMetadata:
        envoy-gateway:
          resources:
          - annotations:
              foo: bar
            kind: ""
            name: ""
            namespace: ""

If that doesn’t cover the EDS endpoint mutation needs, then we can proceed with a dedicated PostEndpointModify hook.

[codefromthecrypt]

@zhaohuabing on using gateway.envoyproxy.io/ Service annotations to pass backend identity through xDS metadata, avoiding a new hook. AFACIT, there are a couple challenges to this:

Metadata level mismatch: Annotations land on LocalityLbEndpoints.Metadata, but ext_proc reads xds.upstream_host_metadata which returns LbEndpoint.metadata — a different proto field. To overcome this, EG would need to copy annotation metadata from LocalityLbEndpoints down to each individual LbEndpoint. That's essentially what PostEndpointModify lets extension servers do — but hardcoding it in EG's translator would be less flexible and would couple EG to ai-gateway's metadata format.

Inaccessible data structure: Even using xds.upstream_host_locality_metadata, the envoy-gateway metadata puts annotations inside a resources ListValue. Envoy's structValue() navigates map keys only, not list indices — no attribute path can extract resources[0].name. To overcome this, either Envoy would need list index support in attribute path resolution (C++ change), or EG would need to flatten resources into direct map keys (metadata format breaking change affecting all consumers of envoy-gateway metadata).

Suggestion: Add a PostEndpointModify hook — one new RPC added to the existing five, called at translator.go:1107 after buildXdsClusterLoadAssignment(). Indeed, the hook sends all endpoints and the extension server filters server-side. Note that ai-gateway already does this by cluster name prefix. UUIC if we did this, it would be no impact to the 8 known extension server consumers.

TL;DR; We should continue your Draft PR on PostEndpointModify hook. If that's not acceptable, probably we need to chat with Tencent, Bloomberg etc as IIUC, the annotation approach has a substantial impact to ai-gateway.

[guydc]

Hi,

Metadata level mismatch: Annotations land on LocalityLbEndpoints.Metadata, but ext_proc reads xds.upstream_host_metadata which returns LbEndpoint.metadata — a different proto field.

Can't we expose that metadata field as an attribute in envoy as well? I think it should be a minor change in Envoy.

Inaccessible data structure: Even using xds.upstream_host_locality_metadata, the envoy-gateway metadata puts annotations inside a resources ListValue. Envoy's structValue() navigates map keys only, not list indices — no attribute path can extract resources[0].name

1. The entire data structure can be passed on and then parsed in ext-proc.
2. The current list structure is a pain for many consumers of metadata (e.g. also access logging, tracing, etc.) . I think it's about time to change it, with a runtime guard.

I'm not against an endpoint hook, I just think that this can be solved in way that better serves other users that do not leverage extension servers.