inference-gateway: ext_proc-based A2LLM inference gateway support#23700
Open
kswap wants to merge 14 commits into
Open
inference-gateway: ext_proc-based A2LLM inference gateway support#23700kswap wants to merge 14 commits into
kswap wants to merge 14 commits into
Conversation
…, CLI)
Add the inference gateway groundwork:
- New ServiceKindInferenceGateway (api + structs), wired into IsProxy/IsGateway.
- Structured `ai { role = "ai-model" }` block on service definitions: AIConfig on
NodeService/ServiceNode/ServiceDefinition with full conversions, IsSame, proto
round-trip (pbservice node/service regen via buf+mog), deep-copy, api mirror,
and agent endpoint response. Marks LLM model upstreams for gateway discovery.
- ext-proc extension: Unix-socket (loopback) Target.Path -> Envoy pipe cluster,
plus inference-gateway in the ext_proc ProxyType allow-list.
- consul connect envoy -gateway inference[-gateway]: register + bootstrap the kind.
New kind = "ai-gateway" routing policy (structs + api), bound to inference gateways via ApplyTo. Carries the ext_proc Processor binding (UDS path, failure mode) and the Routing schema (match rules, compliance map, fallback chain, retry/timeout, optional scoring). Validate enforces an absolute UDS path, failure-mode/config-validation enums, duration parsing, non-empty reserved blocks (Budget/Cache/Mirror), and a first-match-wins shadow check that errors in strict mode and is permitted in warn mode.
Add handlerInferenceGateway and configSnapshotInferenceGateway. The handler watches CA roots and the gateway's own leaf (inbound mesh mTLS identity), the mesh config entry, and the bound ai-gateway config entry (by gateway service name). From the routing policy it derives the candidate model service set and starts/cancels per-candidate health watches; each candidate whose instances carry ai.role == "ai-model" is recorded with its catalog labels and endpoints for routing and listener-metadata injection. Wire the kind into ConfigSnapshot Valid/Clone/Leaf, the xDS auth check, logging, and deep-copy.
Generate the inference gateway's Envoy config from the snapshot: - Inbound listener: mesh mTLS + SPIFFE downstream (calling-agent identity enforced), an HTTP connection manager with the ext_proc filter ahead of the router (streaming to the policy processor: buffered request, streamed response), and listener FilterMetadata "consul.ai" carrying the discovered model catalog (names, roles, labels) for the processor. - Clusters: a STATIC HTTP/2 cluster over the processor's UDS (loopback), plus an EDS cluster per discovered model. - Routes: the processor sets x-ai-cluster and the gateway routes on it to the model cluster, with the routing policy's fallback as the catch-all. - Endpoints: EDS load assignments for the model upstreams.
Golden coverage: TestConfigSnapshotInferenceGateway fixture + an inference-gateway case in TestAllResourcesFromSnapshot, with golden listener (ext_proc + mesh mTLS + consul.ai model metadata), clusters (UDS + model EDS), routes (x-ai-cluster), and endpoints. Integration fixes found via a live dev-agent e2e (register ai-model service + ai-gateway policy + gateway, generate bootstrap): - state graph validation: handle the ai-gateway kind. - secrets xDS: no SDS secrets for the inference gateway. - downstream connect TLS: allow the inference-gateway kind so the inbound listener gets mesh mTLS + the SPIFFE validator. - config-entry streaming: new pbsubscribe Topic_AIGateway wired through the event publisher (memdb topic, kind->topic map, snapshot handler, fsm registration) and the proxycfg-glue topic map, so the gateway can watch its policy. - pbconfigentry: model the ai-gateway config entry in proto + mog conversions so config-entry events serialize (fixes a publish-time panic).
Render the inference-gateway HCM with ForwardClientCertDetails=APPEND_FORWARD
and SetCurrentClientCertDetails{Uri:true}, matching the connect-proxy public
listener. Without this Envoy did not inject the x-forwarded-client-cert header,
so the policy processor saw no caller SPIFFE identity and failed closed.
Regenerate the inference-gateway listener golden file.
Treat an inference-gateway service as connect-capable under its own service name in the connect index. It terminates inbound mesh mTLS on its own listener bearing its own SPIFFE leaf (svc/<name>), so mesh services dial it as a connect upstream by name, like a connect-native service. Without this it was absent from the connect index, so /v1/health/connect/<name> returned nothing and downstream EDS got 0 endpoints.
Go Test Coverage: 71.5%See the workflow run for the full per-package breakdown and downloadable HTML report. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## ext-proc-api-gateway #23700 +/- ##
========================================================
+ Coverage 58.46% 66.81% +8.34%
========================================================
Files 836 694 -142
Lines 106239 83259 -22980
========================================================
- Hits 62117 55627 -6490
+ Misses 38217 22022 -16195
+ Partials 5905 5610 -295 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kswap
changed the title
kswap
changed the base branch from
ft_ext_proc_support_lb
to
ext-proc-api-gateway
June 29, 2026 07:01
Render the ext_proc filter with response body mode BUFFERED and allow_mode_override=true (was response body STREAMED). Buffered keeps non-streamed provider responses normalized in one pass; allow_mode_override lets the processor flip the response body to STREAMED per-response for server-sent-event responses, so streamed completions pass through chunk by chunk while the processor meters the stream. Matches the gateway's static integration config.
Attach the selected model's {model, adapter} (from model_family/model_api
catalog meta) as consul.ai route metadata on each capability route, and
set RequestAttributes: ["xds.route_metadata"] on the ext_proc filter so
the processor receives it. This lets the processor run its full pipeline
pinned to the natively-selected model without re-deriving the capability
or picking a cluster. Route header mutations would not reach ext_proc
(it runs before the router); route metadata via a request attribute does.
Add a Policy block (PII detectors/mask/scope + audit level) to AIGatewayConfigEntry, mirroring the co-located processor's config shape by Go field name. Consul does not interpret it — it stores and returns it verbatim so the processor reads the same entry Consul renders Envoy from (GET /v1/config/ai-gateway/<name>), keeping one source of truth. Covered by a write/read + JSON round-trip test.
Split the inference gateway's ext_proc into the two-phase data path: - Downstream (HCM, pre-route): the existing filter now only prepares the request — identity, request PII, and promoting the body model to x-ai-model. It no longer processes the response or requests route metadata (response transform/metering move upstream). - Routes: match x-ai-model (the promoted body model) against each model's model_family meta (trailing "*" = prefix), replacing the x-ai-cluster steering routes. Capability routes stay but drop their route metadata. - Clusters: each backend model cluster carries its consul.ai metadata (adapter + model) and an upstream HTTP filter chain (ext_proc → upstream_codec). The upstream ext_proc filter requests xds.cluster_metadata + xds.cluster_name so the processor transforms to the selected backend's native schema and attributes metered usage. Regenerate the inference-gateway golden clusters/listeners/routes.
For an inference gateway, `consul connect envoy` now exec's the co-located policy processor with the rendered bootstrap instead of exec'ing Envoy directly. The processor launches Envoy from that bootstrap (over the same pipe) and runs the ext_proc server, supervising both — so the gateway is one process tree and one failure unit, started by the familiar command. Add -ext-proc-bin (default consul-inference-gateway) and -ext-proc-config-entry (defaults to -service) flags; the processor loads that ai-gateway entry for its PII policy and to derive the ext_proc socket, reaching the agent via the forwarded http-addr/token.
`consul config write` decodes into the api client types before sending, so the Policy block added to the structs type was rejected as an unknown key without the matching api field. Mirror Policy/PII/Mask/Detector here so the PII/audit block round-trips through `consul config write`/`read` to the co-located processor.
The inference-gateway co-launch had no way to set the processor's log level, so `consul connect envoy` could not surface the processor's debug output (identity/cluster-metadata). Add -ext-proc-log-level, forwarded as --log-level in inferenceProcArgs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds first-class inference-gateway support to Consul: a new gateway kind that fronts Agent-to-LLM (A2LLM) traffic, terminates mesh mTLS, and runs an Envoy
ext_procfilter pointing at a co-located policy processor which classifies, routes, and translates LLM requests to model upstreams. Targets theext-proc-api-gatewayintegration branch.What's included
ServiceKindInferenceGateway, the serviceai { role = "ai-model" }block (AIConfig) that marks a service as a routable model upstream, the ext_proc-over-UDS plumbing, andconsul connect envoy -gateway inferenceCLI support.ai-gatewayconfig entry (AIGatewayConfigEntry) — the routing policy bound to a gateway:Processor(UDS path, failure mode),Routing.MatchRules(first-match candidate selection),FallbackChain,Retry,Timeout,ComplianceMap,ApplyTo, with shadowed-rule validation.configSnapshotInferenceGateway: the gateway's own leaf, mesh config, the bound config entry, and discovered model upstreams (services whose instances carryai.role == "ai-model").ext_procfilter →local_ext_procUDS cluster; an EDS cluster per discovered model; RDS routes matchingx-ai-cluster: <service>; the discovered-model catalog surfaced to the processor viaconsul.ailistener metadata.ForwardClientCertDetails: APPEND_FORWARD+SetCurrentClientCertDetails{Uri: true}so the caller's SVID reaches the processor asx-forwarded-client-cert(without it the processor's identity stage fails closed). Golden regenerated.connectNameFromServiceNodeso a downstream sidecar's upstream to the gateway resolves its endpoints (it terminates mTLS under its ownsvc/<name>SAN, like a connect-native service).Validation
GOTOOLCHAIN=auto go build ./...clean;api/untouched.TestAllResourcesFromSnapshot) and connect-index/health state tests pass (TestConnectNameFromServiceNode,TestStateStore_CheckConnectServiceNodes_InferenceGateway).