Context
Evaluation monitors need an LLM proxy to route LLM-judge calls through the AI gateway. Currently in #700, the monitored environment's gateway is used for proxy deployment. This is semantically wrong.
Problem
Monitors and agents use "environment" for fundamentally different purposes:
|
Agent |
Monitor |
| Environment meaning |
Where the agent runs (deployed pod, ReleaseBinding) |
Which environment's traces to observe |
| Pod locality |
Long-running pod inside that environment |
Ephemeral WorkflowRun in the workflow plane |
The eval job pod runs in the workflow plane, not in the monitored environment. Deploying the monitor LLM proxy to the monitored environment's gateway is architecturally wrong — the eval job may not be able to reach it, and it ties proxy lifecycle to a gateway the user never intentionally selected for this purpose.
Proposed Solution
Introduce an org-level AI gateway concept — a gateway not scoped to a specific environment, intended for org-wide workloads like eval jobs and scheduled tasks.
- Monitor LLM proxies deploy to this org-level gateway
- Only LLM providers registered against the org-level gateway are shown in the monitor LLM provider picker
Immediate Fallback (until org-level gateways exist)
Allow the user to explicitly pick which gateway + LLM provider to use when creating/editing a monitor:
- Monitor form shows a gateway picker (from the org's available gateways)
- LLM provider list is filtered to providers available on the selected gateway
- This is intentional — the user knows which gateway their eval jobs can reach
This avoids the silent, incorrect automatic selection from the monitored environment.
Impact
- Gateway data model: add
scope field (org vs environment)
- Monitor request model: add
gatewayId field (for the fallback picker)
- Monitor manager:
resolveGatewayForMonitor() prefers org-scoped gateway, falls back to user-selected gateway
- Frontend: gateway picker + filtered LLM provider list in monitor form
Context
Evaluation monitors need an LLM proxy to route LLM-judge calls through the AI gateway. Currently in
#700, the monitored environment's gateway is used for proxy deployment. This is semantically wrong.Problem
Monitors and agents use "environment" for fundamentally different purposes:
The eval job pod runs in the workflow plane, not in the monitored environment. Deploying the monitor LLM proxy to the monitored environment's gateway is architecturally wrong — the eval job may not be able to reach it, and it ties proxy lifecycle to a gateway the user never intentionally selected for this purpose.
Proposed Solution
Introduce an org-level AI gateway concept — a gateway not scoped to a specific environment, intended for org-wide workloads like eval jobs and scheduled tasks.
Immediate Fallback (until org-level gateways exist)
Allow the user to explicitly pick which gateway + LLM provider to use when creating/editing a monitor:
This avoids the silent, incorrect automatic selection from the monitored environment.
Impact
scopefield (orgvsenvironment)gatewayIdfield (for the fallback picker)resolveGatewayForMonitor()prefers org-scoped gateway, falls back to user-selected gateway