Conversation
…ent-level overrides (#5455) ## Summary Replaces the binary on/off switches for deployment-level boolean overrides (Replicate's "use deployments endpoint" and the "use Anthropic endpoints" toggle for SGLang, Deepseek, Fireworks, and vLLM) with a three-way select control. Previously, a plain switch could not distinguish between "explicitly off" and "inherit from the key-level setting," meaning turning the switch off was indistinguishable from leaving it unset. The new `TriStateOverrideRow` component expresses three states: `undefined` (inherit the key's setting), `true` (explicitly on), and `false` (explicitly off). ## Changes - Added a `TriStateOverrideRow` component that renders a select with "Use key setting", "On", and "Off" options, mapping to `undefined`, `true`, and `false` respectively. - Replaced the `Switch`-based inline rows in `ReplicateSection` and `UseAnthropicEndpointsToggleSection` with `TriStateOverrideRow`, preserving the `onChange` contract but now passing the value through directly rather than coercing `false` to `undefined`. - Fixed inconsistent indentation (spaces vs. tabs) in `apiKeysFormFragment.tsx` and `deploymentsTable.tsx`. - Reformatted a few long JSX attribute lists and inline strings for readability. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test 1. Open a provider that supports deployment-level overrides (e.g., Replicate, SGLang, Deepseek, Fireworks, vLLM). 2. Add or edit a deployment and locate the relevant override row. 3. Verify the control renders as a three-option select ("Use key setting", "On", "Off") rather than a toggle switch. 4. Set the value to "Off" and save. Confirm the deployment stores an explicit `false` rather than `undefined`. 5. Set the value to "Use key setting" and save. Confirm the field is stored as `undefined`/absent. ```sh cd ui pnpm i || npm i pnpm build || npm run build ``` ## Screenshots/Recordings Before: A binary switch that could not express "explicitly off" — toggling it off was equivalent to leaving it unset. After: A three-way select with "Use key setting" / "On" / "Off", allowing deployments to explicitly disable a toggle that is enabled at the key level. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Briefly explain the purpose of this PR and the problem it solves. ## Changes - What was changed and why - Any notable design decisions or trade-offs ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [x] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test Describe the steps to validate this change. Include commands and expected outcomes. ```sh # Core/Transports go version go test ./... # UI cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` If adding new configs or environment variables, document them here. ## Screenshots/Recordings If UI changes, add before/after screenshots or short clips. ## Breaking changes - [ ] Yes - [ ] No If yes, describe impact and migration instructions. ## Related issues Link related issues and discussions. Example: Closes #123 ## Security considerations Note any security implications (auth, secrets, PII, sandboxing, etc.). ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…er background colors (#5451) ## Summary Fixes two issues with the trial expiry banner: the background colours were using Tailwind opacity-modifier classes that didn't render correctly, and the trial expiry date parser rejected RFC3339 timestamps (e.g. `2024-06-01T00:00:00Z`) injected by the Docker build, causing the banner to silently disappear. ## Changes - Replaced `bg-red-500/10` and `bg-amber-500/10` with explicit hex values (`#ffebea` and `#fff4e4`) to ensure the banner background renders as intended in both expired/critical and warning states. - Updated `parseTrialExpiry` to accept both bare `YYYY-MM-DD` dates and full RFC3339 timestamps. Only the calendar date portion is used (interpreted at local midnight), so the time component is stripped before parsing. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test 1. Set `TRIAL_EXPIRY_DATE` to an RFC3339 value such as `2024-06-01T00:00:00Z` and confirm the banner appears with the correct background colour. 2. Set it to a bare date such as `2024-06-01` and confirm the banner still renders correctly. 3. Set it to a date within the warning window and confirm the amber (`#fff4e4`) background is shown. 4. Set it to an expired or critical date and confirm the red (`#ffebea`) background is shown. ```sh cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` ## Screenshots/Recordings Before: banner background was invisible due to unresolved Tailwind opacity-modifier classes. After: banner displays the correct solid tinted background in both warning and expired/critical states. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary `ToReplicateImageGenerationInput` previously ignored `InputImages` entirely and could not surface validation errors to callers. This PR adds input image support to the image generation path (mirroring what already existed for image edits) and propagates URL sanitization errors instead of silently dropping them. ## Changes - Changed `ToReplicateImageGenerationInput` to return `(*ReplicatePredictionRequest, error)` so URL validation errors can be surfaced to callers. - Added `InputImages` handling in the generation path: each image URL is sanitized via `schemas.SanitizeImageURL`, and the resulting slice is routed to the correct model-specific field using the new shared helper. - Extracted the model-to-field dispatch logic (`image_prompt`, `input_image`, `image`, `input_images`) into a `setInputImageField` helper, eliminating the duplicated switch block that previously existed only in the edit path. - Updated both `ImageGeneration` and `ImageGenerationStream` call sites to handle the new error return. - Removed stale line-number references from the Replicate provider docs. ## Type of change - [ ] Bug fix - [x] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [x] Docs ## How to test ```sh go test ./core/providers/replicate/... ``` New test cases cover: - `InputImages_SingleImageField` — verifies that a kontext-pro model receives the first image in `input_image` and no other image fields are set. - `InputImages_ArrayFieldWithBase64Normalization` — verifies that a generic model receives all images in `input_images` and that bare base64 strings are prefixed with the `data:image/png;base64,` URI scheme. - `InputImages_InvalidURL` — verifies that a `file://` URI causes an error return and a `nil` result. ## Breaking changes - [x] Yes - [ ] No `ToReplicateImageGenerationInput` now returns `(*ReplicatePredictionRequest, error)` instead of `*ReplicatePredictionRequest`. Any external callers must be updated to handle the additional return value. ## Related issues ## Security considerations Input images are now validated through `schemas.SanitizeImageURL` before being forwarded to Replicate. This prevents schemes such as `file://` from being passed through to the provider. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Integer constraint fields in the Gemini `Schema` type were tagged with `,string` in their JSON struct tags, causing them to serialize as quoted strings (e.g., `"1"`) rather than JSON numbers (e.g., `1`). This broke strict JSON Schema validation upstream when these constraints were passed through `parametersJsonSchema` (issue #5433). ## Changes - Removed the `,string` option from the JSON struct tags for `MinItems`, `MaxItems`, `MinLength`, `MaxLength`, `MinProperties`, and `MaxProperties` on the `Schema` type, so these fields now marshal as JSON numbers instead of quoted strings. - Added a round-trip test (`TestGenAIToolSchemaConstraintsRoundTripAsNumbers`) that verifies integer constraints survive the genai → Bifrost → Gemini conversion as proper JSON numbers, covering both numeric and quoted input forms. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/providers/gemini/... -run TestGenAIToolSchemaConstraintsRoundTripAsNumbers -v go test ./core/providers/gemini/... ``` The new test sends a Gemini generation request with integer constraints specified both as raw numbers and as quoted strings, then asserts that after the full round-trip the output constraints are `float64` JSON numbers (not strings). ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues Closes #5433 ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
… header forwarding (#5476) ## Summary Documents how to inject dynamic, server-side headers into outgoing MCP requests from a `PreMCPHook` plugin, covering use cases that static configuration cannot address (e.g., per-user identity headers, short-lived service tokens, per-request correlation IDs). ## Changes - Added a tip to the header forwarding section in `connecting-to-servers.mdx` clarifying that forwarded headers come from the caller and are untrusted, and pointing readers to the new plugin recipe for server-side injection. - Added a new "Recipe: injecting dynamic headers server-side" section to `writing-go-plugin.mdx` (v1.5.x+) with a full `PreMCPHook` code example that merges plugin-injected headers into `BifrostContextKeyMCPExtraHeaders`, along with notes on the per-client allowlist, transport compatibility (HTTP/SSE only), and why Connect hooks are the wrong place for per-request identity. ## Type of change - [ ] Bug fix - [ ] Feature - [ ] Refactor - [x] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (React) - [x] Docs ## How to test Review the rendered documentation to confirm: - The tip in `connecting-to-servers.mdx` links correctly to the new recipe anchor in `writing-go-plugin.mdx`. - The code example in the recipe compiles without errors when dropped into a plugin project. - The `<Note>` and key-points list render correctly in the docs site. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations The added documentation explicitly calls out that caller-forwarded headers must be treated as untrusted input by upstream servers, and that plugin-injected identity headers (e.g., signed-in user email) are the appropriate mechanism when the caller must not control the value. The per-client `allowed_extra_headers` allowlist is noted as the enforcement boundary for both forwarded and plugin-injected headers. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [x] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Introduces a per-request upstream latency accumulator that tracks cumulative time Bifrost spends blocked on provider sockets across every attempt, retry, fallback, MCP tool call, and media fetch. Subtracting this from total wall time gives Bifrost's own processing overhead — a number that was previously impossible to derive accurately. ## Changes - **New `upstreamlatency.go` schema**: Installs an `*atomic.Int64` accumulator on the `BifrostContext` once per request. Uses an atomic pointer so streaming goroutines can keep writing after the request handler returns without touching the context's value map. - **`ResetUpstreamLatency` / `AddUpstreamLatency` / `GetUpstreamLatency`**: Core API for the accumulator. `Reset` is mandatory at request entry because Bifrost reuses a single process-global context for nil-ctx SDK callers; without it the counter would grow unboundedly. - **`DoStreamingRequest` / `DoHTTPRequest` helpers**: Thin wrappers around `fasthttp.Client.Do` and `net/http.Client.Do` that record the call duration as upstream latency. All provider call sites are migrated to these helpers. - **`idleTimeoutReader.Read` instrumentation**: Each blocking read in a streaming response is counted as upstream time, covering the token-generation window that `DoStreamingRequest` (which returns at first byte) cannot see. - **MCP tool call instrumentation**: `executeToolInternal` wraps `CallTool` with the same accumulator, since waiting on an MCP server is upstream time, not Bifrost overhead. - **`FetchAndEncodeURL` instrumentation**: Remote media fetches are counted as upstream, preventing multi-second fetches from appearing as Bifrost overhead. - **`StampUpstreamLatency` / `PopulateUpstreamLatency`**: Write the accumulated total onto the root trace span (`bifrost.upstream.duration_ms`) and onto `BifrostResponseExtraFields.UpstreamLatency` respectively. Both are called via a named-return `defer` in `handleRequest` so they fire even on error paths. - **`Trace.StampOverheadDuration`**: Computes `bifrost.overhead.duration_ms = root_span_duration - upstream_total` on the export snapshot, after the root span has ended. Clamped at zero to absorb clock skew. - **OTel plugin**: Reads `AttrBifrostOverheadDurationMs` from the root span and records it as a new `bifrost_overhead_latency_seconds` histogram with fine-grained sub-millisecond buckets appropriate for processing overhead rather than network latency. - **Prometheus plugin**: Records the same overhead histogram via the `HTTPTransportPreHook`/`HTTPTransportPostHook` window (widest available, matching the OTel root span). Falls back to the `PostLLMHook` window for SDK callers that bypass the transport layer. - **HTTP transport**: Emits `x-bifrost-upstream-latency-ms` response header so proxy callers can derive overhead from their own elapsed time without parsing the response body. - **New trace attributes**: `bifrost.upstream.duration_ms` and `bifrost.overhead.duration_ms` added to the attribute constant set. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [x] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./... ``` - Make a request through the HTTP transport and verify the `x-bifrost-upstream-latency-ms` response header is present and less than the total elapsed time. - Make a streaming request and confirm the header value grows to reflect the full generation window, not just time-to-first-byte. - Make a request that triggers a fallback and confirm the upstream latency reflects the sum of both attempts. - In OTel/Prometheus dashboards, verify `bifrost_overhead_latency_seconds` appears and that its values are in the sub-millisecond to low-tens-of-milliseconds range for healthy requests. - Confirm `bifrost.upstream.duration_ms` and `bifrost.overhead.duration_ms` appear on root spans in exported traces. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. The upstream latency value is derived from internal timing and contains no secrets or PII. The new response header exposes only a duration in milliseconds. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary
Fixes a bug where plain-text `SecretVar` objects (e.g. `{"value": "..."}` with no `ref`/`type` fields) were not being recognised as `SecretVar`-shaped during redaction restoration. This caused the UI to persist masked values instead of restoring the real stored secrets when saving Kafka SASL credentials or similar connectors that store secrets as plain strings but return them as value-only objects after a redacted GET.
## Changes
- `isSecretVarObject` previously required either `ref`+`type` or `env_var`+`from_env` alongside `value`, which excluded plain-text `SecretVar`s that marshal as `{"value": "..."}` alone (since `ref`/`type` are `omitempty`). The function now accepts any map whose keys are exclusively drawn from the known `SecretVar` field set (`value`, `ref`, `type`, `env_var`, `from_env`), with `value` required to be a string.
- This ensures that value-only objects round-tripped by the UI after a redacted GET are correctly identified and restored from the existing stored value, rather than being passed through with the masked content.
- Objects with a non-redacted value (e.g. username shown in clear) pass through unchanged, and intentional updates (new password, env reference) are not clobbered.
- Tests added for the Kafka SASL credential shape, the `FullyRedacted()` sentinel (`<REDACTED>`), and intentional secret rotation/env-ref switching.
## Type of change
- [x] Bug fix
- [ ] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI
## Affected areas
- [ ] Core (Go)
- [x] Transports (HTTP)
- [ ] Providers/Integrations
- [x] Plugins
- [ ] UI (React)
- [ ] Docs
## How to test
```sh
go test ./transports/bifrost-http/handlers/...
```
The new tests cover:
- Kafka SASL `password` and `ca_cert` restored from stored plain strings when the UI sends back value-only masked objects.
- `FullyRedacted()` sentinel (`<REDACTED>`) correctly triggers restoration.
- Rotated passwords and env-ref switches pass through without being overwritten by the stored value.
## Breaking changes
- [ ] Yes
- [x] No
## Security considerations
This change affects how redacted secret values are handled during plugin configuration updates. The fix ensures masked values are never persisted in place of real secrets, and that intentional secret rotations or env-ref changes are not silently discarded. No new secret exposure surface is introduced.
## Checklist
- [x] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [x] I verified the CI pipeline passes locally if applicable
## Summary Adds a server-configured `batch_role_arn` field to the Bedrock key configuration, allowing operators to pin the IAM service role used for Bedrock batch jobs at the server level rather than relying on clients to supply it via `role_arn` in request extra params. When set, the server-side value takes priority over any client-provided `role_arn`. ## Changes - Added `BatchRoleARN *SecretVar` to `BedrockKeyConfig` in `schemas/account.go`, stored under the JSON key `batch_role_arn` and kept separate from the STS AssumeRole identity (`bedrock_role_arn`). - Updated `BatchCreate` in `bedrock.go` so that `key.BedrockKeyConfig.BatchRoleARN` is resolved first; the client-supplied `role_arn` in `ExtraParams` is only used as a fallback when the server value is absent. - Added a database migration (`add_bedrock_batch_role_arn_column`) that adds the `bedrock_batch_role_arn` column to `config_keys`, with rollback support. - Wired `BedrockBatchRoleARN` through all RDB read/write paths (`tableKeyFromSchemaKey`, `UpdateProvidersConfig`, `UpdateProvider`, `AddProvider`) and through `BeforeSave`/`AfterFind` hooks including encryption and decryption. - Updated the `AfterFind` Bedrock config reconstruction condition to include `BedrockBatchRoleARN`. - Added `BatchRoleARN` to the `mergeUpdatedKey` preserve logic in the HTTP handler so partial updates do not accidentally clear the field. - Added `batch_role_arn` to the config JSON schema with a description noting its priority semantics and `env.` prefix support. - Added `batch_role_arn` to the Zod schemas (`providerForm.ts`, `schemas.ts`) and rendered a **Batch Role ARN** input field in the UI form, visible only when the provider supports the batch API. - Added redaction support for `BatchRoleARN` in `clientconfig.go`. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go version go test ./... # UI cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` **Manual validation:** 1. Configure a Bedrock provider key with `batch_role_arn` set (either as a literal ARN or via `env.AWS_BATCH_ROLE_ARN`). 2. Submit a batch create request that also includes `role_arn` in `extra_params`. 3. Confirm that the server-configured `batch_role_arn` is used and the client-supplied value is ignored. 4. Remove `batch_role_arn` from the key config and resubmit; confirm the client-supplied `role_arn` is now used. 5. Verify the value is stored encrypted in the database and appears redacted in API responses. **New config field:** | Field | JSON key | Description | |---|---|---| | `BatchRoleARN` | `batch_role_arn` | Service role ARN Bedrock assumes for batch S3 access. Supports `env.` prefix. Takes priority over client-supplied `role_arn`. | ## Breaking changes - [ ] Yes - [x] No ## Security considerations `BatchRoleARN` is treated as a secret: it is encrypted at rest via the existing `encryptSecretVarPtr`/`decryptSecretVarPtr` pipeline and redacted in API responses, consistent with other credential fields such as `RoleARN` and `ExternalID`. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Adds additive override state to governance budgets, allowing a temporary or permanent spending limit increase to be layered on top of a budget's base `MaxLimit`. Overrides can be scoped to a finite number of reset cycles (`cycles` mode) or kept active indefinitely (`forever` mode). ## Changes - Introduced `BudgetOverrideMode` type with `cycles` and `forever` constants in `tables/budget.go`. - Added three new columns to `TableBudget`: `override_amount`, `override_mode`, and `override_cycles_remaining`. - Added `validateOverride()` to enforce unambiguous override state (e.g., `cycles` mode requires both a positive amount and a positive cycle count; `forever` mode requires a positive amount and zero cycle count; non-finite amounts are rejected). - Wired `validateOverride()` into the existing `BeforeSave` GORM hook so invalid override combinations are rejected at persistence time. - Added the `add_budget_override_columns` database migration to introduce the three columns with safe defaults, including rollback support. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./framework/configstore/... go test ./framework/configstore/tables/... ``` - `TestMigrationAddBudgetOverrideColumns` verifies that legacy budget rows receive zero-value defaults after migration and that re-running the migration is idempotent. - `TestCreateBudgetWithOverride` verifies that both `cycles` and `forever` override states round-trip correctly through the config store. - `TestTableBudgetValidateOverride` covers all valid and invalid override field combinations, including NaN/infinite amounts, missing modes, and conflicting cycle counts. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. Override amounts are validated to be finite and positive before persistence, preventing malformed data from reaching the budget enforcement layer. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
* fix domains * docs update * [fix]: preserve tool_search_call argument wire shape * [fix]: preserve tool_search execution flag and tool_search_output tools Two further Responses-API wire-shape losses broke the codex tool_search round-trip through bifrost: - `execution: "client"` was dropped from tool_search_call items, so codex never recognized the call as client-dispatchable and silently ended the turn. Add Execution to ResponsesToolMessage. - `tool_search_output` discovered tools are ResponsesTool-shaped (namespace/function, discriminated by `type`), not mcp_list_tools-shaped. The embedded ResponsesMCPListTools decode dropped `type` and nesting, causing OpenAI to 400 with "Missing required parameter input[].tools[].type". Round-trip the tools array verbatim as raw JSON. Adds regression tests for both. Verified end-to-end with codex-acp against a local bifrost build: the full tool_search -> discover -> call -> result loop now completes instead of silently ending the turn. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: trim verbose comments on tool_search wire-shape handling Co-authored-by: Cursor <cursoragent@cursor.com> * fix: preserve tool_search metadata in response copies Keep tool_search_output raw tools, namespace, and execution metadata intact across the response deep-copy paths used by schema helpers and streaming accumulation. Also preserve object-shaped arguments on tool_search_output serialization and document the raw tools invariant for programmatic callers. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: explain response copy compatibility shim Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Akshay Deo <akshay@akshaydeo.com> Co-authored-by: akshaydeo <akshay@akshaydeo.com> Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Adds a dedicated `UpdateBudgetOverride` operation that atomically replaces or clears only the override fields on a budget, ensuring concurrent usage changes and base configuration are never clobbered. Exposes this via two new HTTP endpoints (`PUT` and `DELETE`) scoped to virtual-key budgets.
## Changes
- Added `UpdateBudgetOverride` to `RDBConfigStore` and the `ConfigStore` interface. The method locks the row, validates the new override state via `SetOverride`, and updates only `override_amount`, `override_mode`, `override_cycles_remaining`, and `updated_at`, leaving `current_usage`, `max_limit`, and `reset_duration` untouched.
- Added `HasActiveOverride`, `EffectiveMaxLimit`, `SetOverride`, and `ClearOverride` helpers to `TableBudget`. `SetOverride` rolls back to the previous state if validation fails, making it non-mutating on error.
- Registered `PUT /api/governance/virtual-keys/{vk_id}/budgets/{budget_id}/override` and `DELETE /api/governance/virtual-keys/{vk_id}/budgets/{budget_id}/override` in the governance HTTP handler. Both routes resolve the budget only through model configs scoped to the virtual key, so AP-managed direct mirror budgets are unreachable (404) through this endpoint.
- After a successful override mutation the handler reloads the virtual key in the governance manager so in-memory state stays consistent.
- Added `BudgetOverrideRequest` and `BudgetOverrideResponse` HTTP types; the response includes the refreshed budget and its computed `effective_max_limit`.
## Type of change
- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI
## Affected areas
- [x] Core (Go)
- [x] Transports (HTTP)
- [ ] Providers/Integrations
- [ ] Plugins
- [ ] UI (React)
- [ ] Docs
## How to test
```sh
go test ./framework/configstore/... ./transports/bifrost-http/handlers/...
```
Key scenarios covered by the new tests:
- `TestUpdateBudgetOverridePreservesBudgetState` — confirms `max_limit`, `reset_duration`, and `current_usage` are unchanged after a partial override update and after clearing.
- `TestTableBudgetOverrideLifecycle` — verifies finite-cycle, forever, and cleared override transitions on the model directly.
- `TestTableBudgetSetOverrideRestoresPreviousState` — confirms an invalid `SetOverride` call leaves the budget unchanged.
- `TestVirtualKeyBudgetOverrideLifecycle` — end-to-end HTTP test covering finite override, replacement, clear, and virtual key reload (expects 3 reload calls).
- `TestVirtualKeyBudgetOverrideRejectsDirectMirrorBudget` — confirms AP-managed budgets attached directly to a virtual key return 404 and are not mutated.
## Breaking changes
- [ ] Yes
- [x] No
## Related issues
## Security considerations
The override endpoints resolve budgets exclusively through model configs owned by the target virtual key. Budgets attached directly via `virtual_key_id` (AP mirror budgets) are not reachable, preventing unintended cross-scope mutations.
## Checklist
- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
## Summary Budget overrides with a finite cycle count were not being decremented or expired as budget reset cycles completed. This meant a temporary spending override would remain active indefinitely rather than expiring after the configured number of resets. This PR wires the override lifecycle into the reset path so finite overrides count down and clear themselves automatically. ## Changes - Added `ConsumeOverrideCycle()` to `TableBudget`, which decrements `OverrideCyclesRemaining` by one per reset cycle and calls `ClearOverride()` when the last cycle is consumed. Permanent (`BudgetOverrideModeForever`) overrides are left untouched. - Called `ConsumeOverrideCycle()` in both the normal `ResetBudgetAt` path and the inline reset fallback inside `BumpBudgetUsage`, ensuring the cycle counter advances regardless of which reset code path fires. - Updated `ResetExpiredBudgets` to persist the three override fields (`override_amount`, `override_mode`, `override_cycles_remaining`) alongside `current_usage` and `last_reset` so the decremented state is durably written to the database. - Replaced direct `budget.MaxLimit` reads in `CheckBudget` and `GetBudgetAndRateLimitStatus` with `budget.EffectiveMaxLimit()` so enforcement and status reporting both reflect the active additive override rather than the bare base limit. - The concurrent reset test was extended to assert that exactly one override cycle is consumed when multiple goroutines race to reset the same budget. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./framework/configstore/tables/... -run TestTableBudgetConsumeOverrideCycle -v go test ./plugins/governance/... -run TestGovernanceStoreCheckBudgetUsesOverride -v go test ./plugins/governance/... -run TestGovernanceStoreResetBudgetAdvancesOverride -v go test ./plugins/governance/... -run TestGovernanceStoreResetPersistsOverrideLifecycle -v go test ./plugins/governance/... -run TestResetBudgetAt_ConcurrentResettersCollapse -v go test ./plugins/governance/... -v ``` Expected: all tests pass; finite overrides expire after the configured number of reset cycles, permanent overrides survive resets, and the decremented cycle count is persisted to the database. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. Override amounts are operator-configured values already validated on write; no new inputs or privilege boundaries are introduced. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary
Adds support for additive budget overrides on virtual key budgets, allowing operators to temporarily increase a virtual key's spending capacity beyond its base limit without modifying the base budget itself.
## Changes
- Added `override_amount`, `override_mode`, and `override_cycles_remaining` fields to the `Budget` type, along with `BudgetOverrideRequest` and `BudgetOverrideResponse` types.
- Introduced three utility functions in `governance.ts`: `hasActiveBudgetOverride`, `getEffectiveBudgetLimit`, and `validateBudgetOverride`. These handle override state detection, effective limit calculation, and input validation respectively.
- Replaced all direct references to `b.max_limit` in exhaustion checks, progress bars, and CSV exports with `getEffectiveBudgetLimit(b)` so overrides are reflected everywhere budgets are displayed or evaluated.
- Added `setVirtualKeyBudgetOverride` (PUT) and `removeVirtualKeyBudgetOverride` (DELETE) RTK Query mutations to `governanceApi`, targeting `/governance/virtual-keys/:vkId/budgets/:budgetId/override`.
- Created `BudgetOverrideDialog` component that lets operators add, edit, or remove an override on a single budget. The dialog supports two modes: a fixed number of reset cycles or indefinite ("until removed"). It is surfaced in the virtual key detail sheet for non-managed keys.
- The `BudgetDisplay` tooltip and inline label now show an "override" badge and a breakdown of base + override amounts when an active override is present.
- Added unit tests covering effective limit calculation, expired/incomplete override detection, and input validation.
## Type of change
- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI
## Affected areas
- [ ] Core (Go)
- [ ] Transports (HTTP)
- [ ] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs
## How to test
```sh
cd ui
pnpm i || npm i
pnpm test || npm test
pnpm build || npm run build
```
1. Open a virtual key detail sheet for a key that is not managed by an access profile and has at least one budget with a persisted ID.
2. Click **Add override** next to a budget line.
3. Enter an additional amount, select a mode (cycles or forever), and save. Verify the usage bar and effective limit update immediately.
4. Re-open the dialog and click **Edit override** to confirm existing values are pre-populated.
5. Click **Remove override** and confirm the budget reverts to its base limit.
6. Verify that a key whose usage meets or exceeds the effective limit (base + override) is shown as exhausted.
7. Export virtual keys to CSV and confirm the Budget Limit column reflects the effective limit.
## Screenshots/Recordings
_Add before/after screenshots of the virtual key detail sheet showing the override badge, breakdown text, and dialog._
## Breaking changes
- [ ] Yes
- [x] No
## Related issues
## Security considerations
The override mutations respect the existing `RbacResource.VirtualKeys` / `RbacOperation.Update` permission check; the **Add/Edit override** button is disabled for users without that permission.
## Checklist
- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
## Summary Documents the virtual-key budget override feature, which lets users temporarily add spending capacity to an existing virtual-key budget without modifying its base limit, current usage, or reset schedule. Also corrects the behavior description for the `x-bf-async-webhook` header when an endpoint is not subscribed to the resulting event. ## Changes - Added a **Budget Overrides** entry to the virtual-key key features list. - Added a new **Budget Overrides** section under Usage in `virtual-keys.mdx`, covering: - How the effective limit is calculated (`Base budget + Override amount`). - Step-by-step UI instructions for adding, viewing, editing, and removing overrides. - A note clarifying that overrides require an existing budget and that budgets inherited from an enterprise access profile must be overridden at the access profile level. - Added three screenshots illustrating the add-override button, the configuration dialog, and the budget card with an active override. - Fixed the `x-bf-async-webhook` OpenAPI description: previously stated the job is rejected if the endpoint is not subscribed to the event; corrected to reflect that the job completes normally but no delivery is enqueued in that case. ## Type of change - [ ] Bug fix - [ ] Feature - [ ] Refactor - [x] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [x] Docs ## How to test Review the rendered documentation to confirm: - The Budget Overrides section appears correctly under the Usage heading in the Virtual Keys page. - All three screenshots render as expected. - The `x-bf-async-webhook` parameter description in the API reference reflects the corrected behavior. ## Breaking changes - [ ] Yes - [x] No ## Security considerations None. This is a documentation-only change. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [x] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…d model IDs for model-parameters lookup (#5482) ## Summary The `GET /api/models/parameters` endpoint was returning 404 for valid models when the query used a provider-qualified ID (e.g. `openai/gpt-5.5`, `openrouter/openai/gpt-5.5`) or a bare alias (e.g. `kimi-k2.5`) that didn't exactly match the key stored in the model-parameters datasheet. This PR introduces a fallback resolution chain so those IDs correctly resolve to their stored row. ## Changes - Added a `ModelReasoning` struct to `schemas.Model` that captures provider-advertised reasoning capabilities (mandatory flag, default enabled, supported effort levels, default effort), matching the shape returned by OpenRouter's list-models API. - Added `ResolveModelParameters` to the datasheet `Store`, which walks an ordered candidate list — exact key → progressively provider-prefix-stripped forms → canonical base model name → provider-qualified suffix matches — until a row is found or all candidates are exhausted. - Added `modelParameterCandidates` as the internal helper that builds that ordered list, with deterministic sorting for suffix-matched qualified keys. - Exposed `ResolveModelParameters` on `ModelCatalog` so the HTTP handler can use it. - Updated `getModelParameters` in the HTTP handler to call `ResolveModelParameters` when a model catalog is available, falling back to the direct DB lookup only when it isn't. A `nil` result with no error is now normalized to `ErrNotFound`. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/schemas/... ./framework/modelcatalog/datasheet/... ./transports/bifrost-http/handlers/... ``` The new `TestModelParameterCandidates` test covers the resolution candidate ordering for bare, provider-qualified, double-qualified, alias, and dated model IDs. `TestGetModelParameters_ResolvesQualifiedAndBareIDs` is an integration test that seeds a SQLite config store with `gpt-5.5` and `openrouter/moonshotai/kimi-k2.5` rows and verifies that all equivalent ID forms return HTTP 200 with the correct body, while an unknown model still returns 404. ## Breaking changes - [ ] Yes - [x] No ## Related issues #5105 ## Security considerations None. The change only affects read-path model-parameter lookups and introduces no new auth surface, secret handling, or PII exposure. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…ning` schema to `getModelParameters` (#5483) ## Summary The `GET /v1/models-parameters` endpoint previously returned parameters for all models without any filtering. This PR makes the endpoint accept a required `model` query parameter and resolves provider-qualified model IDs (e.g. `openai/gpt-4o`, `openrouter/openai/gpt-4o`) and bare aliases against the model-parameters catalog, returning the matching record or a 404 if none is found. Additionally, a new `ModelReasoning` schema is introduced to surface reasoning capabilities (mandatory, default_enabled, supported_efforts, default_effort) advertised by provider list-models APIs such as OpenRouter's per-model reasoning object. ## Changes - Updated the `getModelParameters` endpoint to require a `model` query parameter, with ID resolution that handles provider-qualified and bare alias forms - Added a `404` response to the endpoint for cases where no matching model-parameters record is found - Introduced a `ModelReasoning` schema and added it as an optional field on the `Model` schema to represent reasoning capabilities reported by providers ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [x] Docs ## How to test Call the endpoint with a provider-qualified model ID and verify the correct parameter record is returned: ```sh # Should return parameters for the model curl -H "Authorization: Bearer <token>" \ "http://localhost:PORT/v1/models-parameters?model=openai/gpt-4o" # Should return 404 curl -H "Authorization: Bearer <token>" \ "http://localhost:PORT/v1/models-parameters?model=nonexistent-model" # Provider-qualified and bare alias should resolve to the same record curl -H "Authorization: Bearer <token>" \ "http://localhost:PORT/v1/models-parameters?model=openrouter/openai/gpt-4o" ``` ## Breaking changes - [x] Yes - [ ] No The `model` query parameter is now required on `GET /v1/models-parameters`. Callers that previously called this endpoint without a `model` parameter will need to update their requests to include one. ## Related issues #5105 Security considerations No new auth mechanisms introduced. The endpoint remains protected by `ManagementBearerAuth`. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
) ## Summary `ApplyBifrostResponseHeaders` was a package-private function living in the `integrations` package, making it inaccessible to the native `/v1` handlers in `handlers/inference.go`. Both surfaces need to emit the same `x-bifrost-*` routing identity headers on every successful response, so the function has been promoted to a shared `lib` package and extended to also emit the full `x-bifrost-routing-info-*` header set derived from `ExtraFields.RoutingInfo`. ## Changes - Moved `applyBifrostResponseHeaders` (previously package-private in `integrations/utils.go`) to a new `transports/bifrost-http/lib/responseheaders.go` file as the exported `lib.ApplyBifrostResponseHeaders`, making it available to both the native handlers and the integration router. - Extended the function to emit a new `x-bifrost-routing-info-*` header family that mirrors `ExtraFields.RoutingInfo` fields 1:1 (provider, model, key, alias model ID/name/family, is-fallback, primary provider/model, server-side fallback model). Boolean and pointer fields follow the existing convention of being omitted when false/nil/empty. - Replaced all call sites in `handlers/inference.go` — previously calling the local `forwardProviderHeaders` with a nil-guard on `ProviderResponseHeaders` — with `lib.ApplyBifrostResponseHeaders`, which handles the nil case internally and also writes the routing identity headers. - Replaced all call sites in `integrations/router.go` — previously calling the package-private `applyBifrostResponseHeaders` — with `lib.ApplyBifrostResponseHeaders`. Added `bifrostExtraFields` capture in `handleBatchRequest`, `handleFileRequest`, `handleContainerRequest`, `handleContainerFileRequest`, and `handleCachedContentRequest` so the headers are applied before `sendSuccess` on every code path, including early-return binary-response branches. - Moved the `ApplyBifrostResponseHeaders` unit tests from `integrations/utils_test.go` into the new `lib/responseheaders_test.go` and added two new test cases covering the full `RoutingInfo` header set and the primary-route (no-fallback, no-alias) case. ## Type of change - [ ] Bug fix - [x] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./transports/bifrost-http/... ``` Validate that a successful chat completion response includes the following headers: - `x-bifrost-provider` - `x-bifrost-original-model` - `x-bifrost-resolved-model` - `x-bifrost-request-type` - `x-bifrost-routing-info-provider` - `x-bifrost-routing-info-model` - `x-bifrost-routing-info-key` - `x-bifrost-fallback-index` (only present when a fallback fired) - `x-bifrost-routing-info-is-fallback` (only present when `true`) Both native `/v1/chat/completions` and drop-in integration routes (e.g. `/anthropic/v1/messages`, `/openai/v1/chat/completions`) should emit the same header set. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations No new secrets, auth flows, or PII handling introduced. The forwarded provider response headers were already being passed through; this change does not alter the filtering logic for those headers. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Adds two utility helpers for trimming stray whitespace from form inputs before submission: `trimFields` for mutating named string or string-array fields on an object in place, and `trimSecretVar` for producing a trimmed copy of a `SecretVar` form value. ## Changes - Added `trimFields<T>` to `ui/lib/utils/strings.ts` — trims whitespace from named string or `string[]` fields on an object in place, leaving undefined fields untouched. Uses a `TrimmableKeys` conditional type to restrict callers to only string-compatible keys. - Added `trimSecretVar` to `ui/lib/utils/secretVarForm.ts` — returns a shallow copy of a `SecretVar`-shaped field with `value` and `ref` trimmed, preserving the original type parameter. - Added unit tests for `trimFields` covering string fields, string array fields, undefined fields, partial key selection, and return-value identity. ## Type of change - [ ] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` ## Breaking changes - [ ] Yes - [x] No ## Security considerations Trimming whitespace from secret variable references and literal values reduces the risk of accidentally storing or submitting credentials with leading/trailing whitespace that could cause silent authentication failures. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Tracing spans for failed streaming requests were silently reporting unknown status because error attributes were never stamped when a stream errored out. This fixes two code paths where a `BifrostError` exists but `PopulateLLMResponseAttributes` was never called: failed stream requests in the retry loop and streams that fail before the first chunk arrives during deferred span completion. ## Changes - In `executeRequestWithRetries`, added an `else if` branch to call `PopulateLLMResponseAttributes` with a nil response and the error when a stream request fails — previously the cast to a non-stream response type would miss this case entirely. - In `completeDeferredSpan`, added an `else if` branch to stamp error attributes when a stream errors before producing any chunks, ensuring the span is not left with unknown/empty attributes. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test Trigger a streaming request that is expected to fail (e.g., invalid model name, bad credentials, or a provider that returns an error before streaming begins) and verify that the resulting trace span contains the correct error attributes rather than reporting unknown status. ```sh go test ./... ``` ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations No security implications. This change only affects observability/tracing attribute population. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary
Adds end-to-end observability test coverage for provider error paths — both non-streaming and streaming requests that fail before the first chunk. This pins a regression where spans were exported without `gen_ai.error.*` attributes when a stream failed before delivering any data, and ensures the `bifrost_error_requests_total` Prometheus counter is correctly labeled with `status_code` for both request types.
## Changes
- Introduced an `ERROR_TRIGGER` marker in the mock provider that returns a provider-style 404 error body (`model_not_found`) for any request containing it, covering both streaming and non-streaming failure paths.
- Added `chatError(id, stream)` to fire requests expected to fail with a 404, validating both the non-stream and pre-first-chunk stream error paths.
- Added `assertOtelErrorTrace(id, label)` to verify that exported spans carry `gen_ai.error.type`, `gen_ai.error.code`, and `http.response.status_code` attributes for error requests.
- Added `assertPrometheusErrorScrape()` to confirm `bifrost_error_requests_total{status_code="404"}` is present for both `chat_completion` and `chat_completion_stream` methods.
- Enabled `chat_completion_stream` in the mock provider's `allowed_requests` config so streaming error requests are routed correctly.
- Generalized `assertMockProviderRequest` to accept an expected request count rather than always asserting exactly one.
## Type of change
- [x] Bug fix
- [ ] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI
## Affected areas
- [ ] Core (Go)
- [ ] Transports (HTTP)
- [ ] Providers/Integrations
- [ ] Plugins
- [ ] UI (React)
- [ ] Docs
## How to test
Run the local observability E2E test suite. The test will:
1. Fire a non-streaming request with the error trigger and assert the exported span contains `gen_ai.error.*` attributes.
2. Fire a streaming request with the error trigger and assert the same span attributes are present (pinning the deferred-span stamping regression).
3. Scrape `/metrics` and assert `bifrost_error_requests_total{status_code="404"}` is present for both `chat_completion` and `chat_completion_stream`.
```sh
node tests/e2e/api/runners/run-observability-local.mjs
```
## Breaking changes
- [ ] Yes
- [x] No
## Related issues
## Security considerations
None. The error trigger is scoped entirely to the mock provider used in tests and does not affect production routing logic.
## Checklist
- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [ ] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
## Summary Replaces em-dash (`—`) separators used as clause connectors in UI copy with grammatically appropriate alternatives: semicolons, colons, commas, or periods. This improves readability and consistency across tooltips, descriptions, labels, alerts, and inline help text throughout the UI. ## Changes - Replaced em-dashes used to join independent clauses with semicolons (e.g. "token is not revoked at the provider — it stays detached" → "...provider; it stays detached") - Replaced em-dashes used to introduce elaborations or examples with colons or commas (e.g. "Large payload request — input content..." → "Large payload request: input content...") - Replaced em-dashes used in parenthetical asides with parentheses or commas where appropriate - Changed one em-dash team name separator in a combobox label to a standard hyphen (`-`) since it appears in a data label context rather than prose ## Type of change - [ ] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [x] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh cd ui pnpm i || npm i pnpm build || npm run build ``` Visually inspect affected UI surfaces (logging config, MCP config, routing rules, virtual keys, provider keys, pprof page, log detail view, sessions table, observability fragments) to confirm copy reads correctly with no regressions. ## Screenshots/Recordings No visual layout changes expected; only text content is affected. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Streaming responses in Bifrost write HTTP response headers before the first SSE chunk arrives, but routing identity (provider, model, key, fallback status) was only available on per-chunk `ExtraFields`. This meant routed-identity `x-bifrost-*` headers were missing entirely from streaming responses. This PR fixes that by stashing a `RoutingInfo` snapshot into the `BifrostContext` at stream setup time, then reading it in the transport layer to emit the correct headers before the first chunk is written. ## Changes - Added `BifrostContextKeyRoutingInfo` as a new reserved context key. Core writes a `RoutingInfo` snapshot into the context at each stream attempt (overwritten on retry, so the winning attempt's snapshot survives). On successful fallback, the snapshot is updated to reflect `IsFallback`, `PrimaryProvider`, and `PrimaryModel`, mirroring the existing `SetFallbackRoutingInfo` logic. - Added `RoutingInfo.ToExtraFields(requestType)` to build a `BifrostResponseExtraFields` from a finalized `RoutingInfo`, using the same deprecated-triplet sync rules as the non-streaming response path. - Added `ApplyBifrostStreamResponseHeaders` in the transport lib, which reads the context snapshot and calls the existing `ApplyBifrostResponseHeaders` before any SSE write. When no snapshot is present (e.g. a plugin short-circuited the stream), only the request-type header is emitted. - All streaming handler entry points (`handleStreamingTextCompletion`, `handleStreamingChatCompletion`, `handleStreamingResponses`, `handleStreamingSpeech`, `handleStreamingTranscriptionRequest`, `handleStreamingImageGeneration`, `handleStreamingImageEditRequest`) now pass their `RequestType` through to `handleStreamingResponse`, which calls `ApplyBifrostStreamResponseHeaders` after the stream channel is obtained but before any SSE headers are flushed. - The generic router's `handleStreamingRequest` follows the same pattern, tracking `requestType` per branch and calling `ApplyBifrostStreamResponseHeaders` at the same point. - Tests cover the normal identity case, the fallback-layered case (including deprecated header derivation), and the missing-snapshot fallback. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/... ./transports/bifrost-http/... ``` To validate end-to-end: send a streaming chat completion request through the HTTP transport and inspect the response headers. Expect `x-bifrost-routing-info-provider`, `x-bifrost-routing-info-model`, `x-bifrost-routing-info-key`, and `x-bifrost-request-type` to be present on the response before any SSE data is received. For a fallback scenario, also expect `x-bifrost-routing-info-is-fallback: true`, `x-bifrost-routing-info-primary-provider`, and `x-bifrost-routing-info-primary-model`. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations The `RoutingInfo` snapshot stored in context contains provider key identifiers (alias names, not secret values). This is consistent with what was already emitted on non-streaming responses via `ExtraFields`. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary
Adds streaming retrieval of stored OpenAI Responses API objects via `GET /v1/responses/{id}?stream=true`. Previously, retrieving a stored response only supported a unary (non-streaming) path. This change introduces a parallel SSE streaming path that replays the stored response as a stream of typed events, matching the behaviour of the create-stream path.
## Changes
- Added `ResponsesRetrieveStreamRequest` as a new `RequestType` constant (`"responses_retrieve_stream"`) and registered it across all routing, accumulator, logging, and permission checks that handle stream request types.
- Added a `Stream *bool` field to `BifrostResponsesRetrieveRequest` and an `IsStreamingRequested()` helper method. `buildResponsesRetrieveQuery` now encodes the `stream` query parameter when set.
- Added `ResponsesRetrieveStream` to the `ResponsesLifecycleProvider` interface and implemented it on `OpenAIProvider`. The implementation issues a streaming-aware `GET` with `Accept: text/event-stream`, runs the same SSE loop used by the create-stream path (idle timeout, gzip decompression, cancellation, error event handling, `response.completed`/`response.incomplete` terminal events), and is intentionally kept separate from the create-stream path.
- Added `ResponsesRetrieveStreamRequest` on `Bifrost` with the same nil/empty-field guards as other lifecycle methods, delegating to `handleStreamRequest`.
- Wired `ResponsesRetrieveStreamRequest` into `handleProviderStreamRequest` via the `ResponsesLifecycleProvider` type assertion, returning an unsupported-operation error for providers that do not implement the interface.
- Updated the HTTP transport (`responsesRetrieve` handler and `extractResponsesLifecycleFromPath`) to parse the `stream` query parameter and branch into `handleStreamingResponsesRetrieve` when `stream=true`, keeping the unary path unchanged.
- Added a `StreamConfig` with a `ResponsesStreamResponseConverter` to the OpenAI route config for the retrieve endpoint so SSE chunks are serialised consistently with the create-stream path.
- Updated the generic router's `handleStreamingRequest` to dispatch `ResponsesRetrieveRequest` payloads through `ResponsesRetrieveStreamRequest`.
- Added `"responses_retrieve_stream"` to the UI log constants, labels, and colour map (violet, matching other stream types).
- Added unit tests covering `buildResponsesRetrieveQuery` with `stream=true`/unset and `IsStreamingRequested` across nil/unset/false/true cases.
## Type of change
- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI
## Affected areas
- [x] Core (Go)
- [x] Transports (HTTP)
- [x] Providers/Integrations
- [x] Plugins
- [x] UI (React)
- [ ] Docs
## How to test
```sh
# Core/Transports
go test ./core/... ./framework/... ./plugins/... ./transports/...
# Create a response, then retrieve it as a stream
curl -N "http://localhost:8080/v1/responses/{id}?stream=true&provider=openai" \
-H "x-bifrost-key: <key>" \
-H "Accept: text/event-stream"
# Expect: a sequence of SSE events ending with event: response.completed
# Retrieve without stream param — must still return a unary JSON response
curl "http://localhost:8080/v1/responses/{id}?provider=openai" \
-H "x-bifrost-key: <key>"
# UI
cd ui
pnpm i
pnpm build
```
## Breaking changes
- [x] No
`ResponsesLifecycleProvider` gains a new method (`ResponsesRetrieveStream`). Any external implementation of this interface must add the method. The built-in `OpenAIProvider` is the only implementation in this repository.
## Related issues
## Security considerations
No new auth mechanisms are introduced. The streaming retrieve path reuses the same bearer-token header injection and key-resolution logic as all other provider requests. No PII beyond what is already present in a stored response is exposed.
## Checklist
- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
## Summary Briefly explain the purpose of this PR and the problem it solves. ## Changes - What was changed and why - Any notable design decisions or trade-offs ## Type of change - [ ] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test Describe the steps to validate this change. Include commands and expected outcomes. ```sh # Core/Transports go version go test ./... # UI cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` If adding new configs or environment variables, document them here. ## Screenshots/Recordings If UI changes, add before/after screenshots or short clips. ## Breaking changes - [ ] Yes - [ ] No If yes, describe impact and migration instructions. ## Related issues Link related issues and discussions. Example: Closes #123 ## Security considerations Note any security implications (auth, secrets, PII, sandboxing, etc.). ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Fixes the `customResponseHandler` path in `HandleOpenAIResponsesStreaming` where `sendBackRawRequest` and `sendBackRawResponse` were hardcoded to `false`, and where subsequent error/completion handling was incorrectly scoped inside the `else` block, causing it to be skipped entirely when a custom response handler was provided. Fixes #5504 ## Changes - Replaced hardcoded `false, false` arguments in the `customResponseHandler` call with the actual `sendBackRawRequest` and `sendBackRawResponse` values. - Added `RawResponse` population for the custom handler path, mirroring the existing behavior in the non-custom handler path. - Moved error type handling (`ResponsesStreamResponseTypeError`, `ResponsesStreamResponseTypeFailed`), completion handling (`ResponsesStreamResponseTypeCompleted`, `ResponsesStreamResponseTypeIncomplete`), and chunk dispatch outside of the `else` block so they execute regardless of whether a custom response handler is used. - Removed the `// TODO fix this` comment that tracked this known issue. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./... ``` ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
… paginated log search (#5486) ## Summary Fixes a pagination `total_count` bug where matview-eligible windows (≥24h) returned an inflated count because `mv_logs_hourly` predicates on `hour` rounded both time boundaries out to full hour buckets, counting logs that the exact-range row list could never page to. Closes #5329. ## Changes - Replaced the single `mv_logs_hourly` sum in `getCountFromMatView` with a hybrid approach: only buckets fully contained within `[StartTime, EndTime]` are summed from the matview; the partial boundary buckets (at most one on each side) are counted directly from the raw `logs` table using a timestamp-indexed scan over ≤2 hours of rows. - Introduced `countRawTerminal` to count raw log rows restricted to terminal statuses when no status filter is present, ensuring both halves of the hybrid count cover the same population as `mv_logs_hourly`. - All bucket-grid arithmetic is performed in SQL via `date_trunc('hour', timestamptz)` rather than Go-side `time.Truncate`, so servers with fractional-hour timezone offsets (e.g. `Asia/Kolkata`, +05:30) produce correct bucket boundaries without misalignment. - Added `TestSearchLogsMatViewCountMatchesRawRange` to assert that `total_count` matches the number of terminal logs strictly within `[StartTime, EndTime]` across mid-hour, hour-aligned-start, and hour-aligned-end boundary cases. ## Type of change - [x] Bug fix ## Affected areas - [x] Core (Go) ## How to test ```sh go test ./framework/logstore/... -run TestSearchLogsMatViewCountMatchesRawRange -v ``` The test inserts logs at known timestamps straddling a 25h15m window, refreshes the matviews, forces the matview count path, and asserts that `total_count` equals exactly the number of terminal logs inside `[start, end]` for three boundary configurations (mid-hour boundaries, hour-aligned end, hour-aligned start). ## Breaking changes - [x] No ## Related issues Closes #5329 ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary This PR adds two improvements to the materialized view read path: cache-hit statistics are now served from the hybrid aggregate (interior buckets from `mv_logs_hourly`, boundary slivers classified raw) instead of a separate full-window raw scan, and a runtime self-heal mechanism automatically recovers from missing or stale-shaped materialized views without requiring a process restart. ## Changes - **Cache hits in the hybrid aggregate**: Three new columns (`direct_cache_hits`, `semantic_cache_hits`, `cache_debug_count`) are added to `mv_logs_hourly` using the same `cacheDebugJSONGuard` and `cacheDebugHitTypeExpr` expressions shared across the matview DDL, boundary sliver queries, and `aggregateCacheHits`. `cache_debug_count` preserves the nil contract: when no row in the window carried valid `cache_debug` JSON the fields are omitted from the response, and when cache rows exist but none were direct/semantic explicit zeros are returned. The previous approach issued a separate full-window raw scan for cache hits after the hybrid aggregate completed; that scan is removed. - **Runtime matview self-heal** (`matviewheal.go`): `isMatViewShapeError` classifies PostgreSQL error codes `42P01` (undefined table), `42703` (undefined column), and `55000` (object not in prerequisite state) as shape errors. `fallBackToRaw` is called at every matview dispatch site — on a shape error it disables the matview read path process-wide, logs a warning, and triggers a single-flight background repair via `triggerMatViewSelfHeal`. The repair runs `ensureMatViews` then `refreshMatViews` and re-enables the path on success. A 30-second cooldown (`matViewHealCooldown`) bounds repair frequency; while broken, every request continues succeeding via the raw fallback. Two new atomic fields (`matViewHealInFlight`, `matViewHealLastAttempt`) are added to `RDBLogStore`. - **Shared SQL constants**: The inline regex strings for the cache debug guard and hit-type extractor are replaced with named constants `cacheDebugJSONGuard` and `cacheDebugHitTypeExpr`, used consistently across the matview DDL, `applyFilters`, `rawTerminalStatsAgg`, and `aggregateCacheHits`. - **Tests**: `TestGetStatsMatViewCacheHitsHybrid` verifies the hybrid cache-hit path including nil and zero contracts and agreement with the raw path. `TestMatViewShapeErrorFallsBackAndSelfHeals`, `TestMatViewStaleShapeFallsBackAndSelfHeals`, and `TestFilterMatViewShapeErrorFallsBack` cover the 42P01, 42703, and filter-view drop scenarios end-to-end, including background self-heal convergence. ## Type of change - [x] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./framework/logstore/... -run TestGetStatsMatViewCacheHitsHybrid go test ./framework/logstore/... -run TestMatViewShapeErrorFallsBackAndSelfHeals go test ./framework/logstore/... -run TestMatViewStaleShapeFallsBackAndSelfHeals go test ./framework/logstore/... -run TestFilterMatViewShapeErrorFallsBack go test ./framework/logstore/... -run TestIsMatViewShapeError go test ./framework/logstore/... ``` The self-heal tests drop or replace `mv_logs_hourly` mid-run and assert that reads continue returning correct results from the raw table with no error, that `matViewsReady` is set to false immediately, and that the view is recreated with the correct shape within 90 seconds. ## Breaking changes - [x] No The new matview columns require a schema migration. On first deploy, `repairMatViewShapes` detects the missing columns, drops and recreates `mv_logs_hourly`, and the self-heal path handles any replica that reads before the rebuild completes. ## Related issues Closes #5384 ## Security considerations No auth, secrets, PII, or sandboxing changes. The new SQL expressions are constants composed only of built-in PostgreSQL operators and are not user-controlled. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…omerSelector` in virtual keys table and sheet (#5644) ## Summary The virtual keys page and sheet previously fetched the full teams and customers lists upfront and passed them down as props for filtering and entity assignment. This PR removes those bulk fetches in favour of server-side search selectors (`TeamSelector` / `CustomerSelector`) that resolve their own labels on demand, eliminating unnecessary data loading and prop drilling. ## Changes - Removed `useGetTeamsQuery` and `useGetCustomersQuery` calls from the governance virtual keys page; the page no longer fetches or holds teams/customers lists. - Dropped `teams` and `customers` props from `VirtualKeysTable` and `VirtualKeySheet`. - Replaced the static `ComboboxSelect` team/customer filters in the table toolbar with `TeamSelector` and `CustomerSelector` components, which search server-side. Added a `FilterClearButton` helper component to restore the "clear to all" affordance that `ComboboxSelect` previously provided for free. - In `VirtualKeySheet`, the entity assignment section is now always rendered (previously hidden when no teams/customers were loaded). The assignment type dropdown always shows both "Assign to Team" and "Assign to Customer" options. Switching type clears the id fields rather than defaulting to the first item in a list. - The locked-team banner in `VirtualKeySheet` now resolves the team name via a single `useGetTeamQuery(attachedTeamId)` call instead of searching through the full teams list. - The team/customer fallback labels in the selectors are now sourced from the `team` and `customer` objects embedded on the `VirtualKey` itself rather than from the previously fetched lists. - RBAC checks for `Teams` and `Customers` view permissions were removed from the governance page since no bulk fetches are gated on them anymore. - Error handling in the governance page was simplified to only track virtual key fetch errors. ## Type of change - [ ] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test 1. Navigate to the Governance → Virtual Keys page and confirm it loads without fetching `/teams` or `/customers` on mount. 2. Use the Team and Customer filter dropdowns — verify they search server-side and display results correctly. 3. Select a filter value and confirm the clear (`×`) button resets the filter back to "All". 4. Open the virtual key sheet for an existing key assigned to a team; confirm the team name resolves correctly in the locked banner and in the team selector fallback label. 5. Create a new virtual key, switch the assignment type between Team and Customer, and confirm the id fields are cleared on each switch. 6. Confirm error toasts still appear when the virtual keys fetch fails. ```sh cd ui pnpm i pnpm build ``` ## Screenshots/Recordings _Add before/after screenshots of the filter toolbar and entity assignment section if available._ ## Breaking changes - [x] No ## Related issues ## Security considerations No new auth surfaces introduced. Removed RBAC checks for teams/customers on this page are safe because the underlying selectors enforce their own access controls server-side. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Re-enables previously skipped pricing override tests and updates them to work with `InputCostPerToken` and `OutputCostPerToken` now being pointer types (`*float64`) rather than plain `float64` values. Also removes a stale skipped test for a removed Azure API version column. ## Changes - Removed `TestEncryptedColumns_AzureAPIVersion_FitsAfterWidening` which was already skipped with a note that the underlying column no longer exists. - Re-enabled multiple `t.Skip()`-guarded pricing override tests that were previously deferred. - Updated assertions in those tests to dereference pointer fields (`*pricing.InputCostPerToken`, `*pricing.OutputCostPerToken`) and added `require.NotNil` guards before each dereference. - Added `RequestTypes` fields to override fixtures in several tests to satisfy updated matching logic that requires request type scoping. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./framework/modelcatalog/datasheet/... go test ./framework/configstore/tables/... ``` All previously skipped tests should now run and pass. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary The vision model provider for HuggingFace tests has been switched from `groq` to `novita` for the Llama-4-Scout-17B-16E-Instruct model, likely due to availability or reliability issues with the `groq` provider for this model. ## Changes - Replaced `groq/meta-llama/Llama-4-Scout-17B-16E-Instruct` with `novita/meta-llama/Llama-4-Scout-17B-16E-Instruct` as the vision model in the HuggingFace test configuration. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/providers/huggingface/... ``` Ensure the HuggingFace vision model test passes using the `novita` provider endpoint. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations No security implications. This change only affects which provider backend is used for a test model route. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary A misconfigured or unreachable OTEL collector (one that completes a TCP handshake but never replies) previously blocked export goroutines indefinitely — one goroutine and one full trace snapshot per request. On the gRPC path there was no client-side deadline at all, so the leak was permanent. This PR adds layered defences: a per-export timeout, a per-plugin concurrency cap in the tracer, and a per-target circuit breaker in the OTEL plugin. ## Changes - **Per-export deadline (`export_timeout`)**: A new `export_timeout` field (default 5 s, max 60 s) is added to the OTEL profile. A `context.WithTimeout` wrapping every `Emit` call is now the only deadline on the gRPC path. The HTTP client timeout is wired to the same value instead of the previous hardcoded 30 s. - **Per-plugin concurrency cap in the tracer**: `obsPluginSlot` wraps each observability plugin with a buffered semaphore (`maxConcurrentInjectsPerPlugin = 1024`). A non-blocking acquire is attempted before spawning a flush goroutine; if the plugin is already saturated the trace is skipped and counted. Plugins are now fanned out in parallel rather than called sequentially, so a stalled connector can no longer delay plugins registered after it. - **Circuit breaker in the OTEL plugin**: After `breakerFailureThreshold` (5) consecutive export failures, a target suppresses further export attempts for `breakerCooldown` (30 s). A single probe is allowed through once the cooldown expires. Export failures and suppressed exports are tracked per target and exposed via `ExportStats()`. - **Bounded shutdown wait**: `Tracer.Stop()` previously called `flushWG.Wait()` with no timeout. It now calls `waitForFlushes(10 s)`, which logs a warning and continues if in-flight exports have not drained in time. - **Deduplication moved to `SetObservabilityPlugins`**: The per-flush `seen` map is replaced by deduplication at slot-build time, removing a map allocation from the hot path. - **`ObservabilityDropCounts()`**: New method on `Tracer` exposes per-plugin skip counts for observability and testing. - **Schema and UI**: `export_timeout` is added to `config.schema.json`, `values.schema.json`, `values.yaml`, the Zod schema, and the OTEL profile form with a numeric input (1–60 s) and description. ## Type of change - [x] Bug fix - [x] Feature ## Affected areas - [x] Core (Go) - [x] Plugins - [x] UI (React) ## How to test ```sh # Core/Transports go test ./framework/tracing/... -v -run TestCompleteAndFlushTrace go test ./framework/tracing/... -v -run TestWaitForFlushes go test ./framework/tracing/... -v -run TestSetObservabilityPlugins go test ./plugins/otel/... -v -run TestInject_GRPCExportHonoursTimeout go test ./plugins/otel/... -v -run TestInject_HTTPExportHonoursTimeout go test ./plugins/otel/... -v -run TestInject_CircuitBreakerStopsDialling go test ./plugins/otel/... -v -run TestBreakerRecoversAfterCooldown go test ./plugins/otel/... -v -run TestResolveExportTimeout # UI cd ui pnpm i pnpm build ``` **New config field:** | Field | Location | Default | Range | Description | |---|---|---|---|---| | `export_timeout` | OTEL profile | `5` | 1–60 s | Maximum duration for a single trace export. The only deadline on gRPC exports. | To validate end-to-end: point an OTEL profile at a black-hole TCP listener (e.g. `nc -l 4317` without reading), send requests, and confirm that request latency is unaffected and that `ObservabilityDropCounts` / `ExportStats` report skipped/failed exports rather than the process stalling. ## Breaking changes - [x] No `NewOtelClientHTTP` gains a `timeout time.Duration` parameter. Any code calling it directly outside this repository will need to pass the timeout explicitly. ## Related issues Closes the goroutine-leak / log-row-drop regression caused by a misconfigured OTEL collector blocking the logging plugin's DB batch writer. ## Security considerations No auth, secrets, or PII changes. The circuit breaker and concurrency cap reduce the blast radius of a misconfigured or malicious collector endpoint by bounding how many goroutines and how much heap it can consume. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
## Summary This PR addresses several operational reliability issues around Postgres connection pool management, materialized view refresh correctness, ClickHouse batch insert lock contention, and observability gaps in governance and logging paths. ## Changes - **Connection pool tuning**: Added `conn_max_idle_time` (default 5m) to both the logs store and config store pools. Without it, the idle cap is the only pool-size control, so any burst above `MaxIdleConns` closes and reopens physical connections on every query. Migration pools are now explicitly pinned to 2 open / 1 idle connection rather than inheriting `database/sql`'s unlimited default. - **Password command caching**: Introduced a `passwordCache` with a configurable `cache_ttl` (default 60s) that collapses concurrent resolutions via `singleflight`. Previously `BeforeConnect` ran the password command on every new physical connection, causing a subprocess fork per connection under pool churn. Failures are never cached so a transient error does not persist for the whole TTL. - **Materialized view refresh timeout**: Added a per-tick `matview_refresh_timeout` (derived as 5× the refresh interval, floored at 5m, capped at 30m). A refresh holds a pooled connection and a session-scoped advisory lock across every view; without a deadline one stuck refresh leaves views permanently stale on all replicas. The advisory lock unlock now runs on a `context.WithoutCancel` context so a deadline-exceeded refresh still releases the lock. If the unlock cannot be confirmed, the pooled connection is discarded so Postgres reclaims the lock. - **Materialized view refresh rotation**: Filter views now start at a rotating offset each tick. If a pass is consistently cut short by its deadline, a fixed order would starve the tail views permanently. `mv_logs_hourly` remains pinned first since dashboards depend on it. - **ClickHouse batch insert chunking**: `BatchCreateIfNotExists` and `BatchCreateMCPToolLogsIfNotExists` now process entries in chunks of 128 (`chRMWBatchChunk`). An unchunked batch of 1000 rows against 128 shards effectively acquires every shard lock, serialising all concurrent updates. Chunking keeps the held-shard set small enough for concurrent writers to interleave. - **`RetryOnNotFound` logging**: Each retry attempt, success after retry, and exhaustion are now logged at Warn. Without these lines, a slow request on an interactive path looks like slow work rather than waiting on a not-found retry loop. - **`ApplyPoolTuning` logging**: When a logger is supplied, the effective pool shape (max open, max idle, idle time, lifetime) is logged once at startup so operators can size against the server's `max_connections` without reading source. - **`StageTimer` utility**: Added a `StageTimer` type in the governance package that accumulates named stage durations for a single pass and emits them as one log line, logging at Info when the total exceeds a threshold and at Debug otherwise. - **Governance reset cycle timing**: `resetExpiredCounters` now wraps every phase with `StageTimer` marks and warns explicitly when the total exceeds `workerInterval`, since a Go ticker drops ticks silently when the receiver is slow. - **Virtual key save and reload timing**: `updateVirtualKey` and `ReloadVirtualKey` now emit per-stage timing summaries via `StageTimer`. The lock-wait stage in the transaction is marked separately from the work stages since time there is contention, not computation. - **Deferred usage watcher cap**: Added `deferredUsageWatchSem` (cap 2048) to bound goroutines parked waiting for trailing usage on large-payload requests. Previously only the DB-update concurrency was limited; the watcher goroutines themselves were unbounded. The retry backoff was also reduced from 2s/4s/8s to 250ms/500ms/1s, and each DB call now carries a 10s context timeout. ## Type of change - [x] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./framework/postgresconn/... go test ./framework/logstore/... go test ./framework/configstore/... go test ./plugins/governance/... go test ./plugins/logging/... ``` New configuration fields: | Field | Location | Default | Description | |---|---|---|---| | `conn_max_idle_time` | logs store / config store | `5m` | How long an idle physical connection is kept before being closed | | `password_command.cache_ttl` | postgres connection | `60s` | How long a resolved password is reused across new physical connections | | `matview_refresh_timeout` | logs store | 5× interval, min 5m | Maximum time for a single materialized view refresh pass | ## Breaking changes - [x] No `ApplyPoolTuning` now accepts a variadic logger argument; all existing call sites compile unchanged. The password command now caches results by default — operators relying on per-connection re-execution (e.g. credentials with a validity window shorter than 60s) should set `cache_ttl` explicitly to a value below their rotation period. ## Security considerations The password command cache stores a resolved credential in memory for up to `cache_ttl`. Operators using short-lived credentials (e.g. AWS RDS IAM tokens valid for 15 minutes) should verify their `cache_ttl` is set below the credential's validity window to avoid connection failures after rotation. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Database writes for budget and rate-limit resets and dumps were issued one row per statement inside a single transaction spanning the entire sweep. Against a remote database this meant one network round trip per row and every row lock held until the whole sweep committed, which blocked interactive virtual-key saves for the full duration of the sweep. This PR replaces the per-row, single-transaction pattern with batched multi-row `UPDATE … FROM (VALUES …)` statements on PostgreSQL and chunked per-row fallbacks on other dialects. ## Changes - Introduced `dumpBatchSize = 500` to bound both how many rows a single batched statement covers and how long any row lock is held. Each chunk is its own transaction, so locks are released between batches rather than held for the entire sweep. - Added `rateLimitDumpRow` and `budgetDumpRow` structs to carry the fields needed for a dump write, replacing the previously inline anonymous struct. - Added `dumpRateLimitBatch`, `writeBudgetBatch`, and `resetRateLimitDimensionBatch` helpers that emit a single `UPDATE … FROM (VALUES …)` on PostgreSQL and fall back to per-row updates elsewhere. `supportsBatchedDump` gates the dialect check. - `writeBudgetBatch` accepts a `usageGuard` parameter (`"<"` for resets, `"<="` for dumps) so both callers share the same two-statement override-then-usage write order without duplicating the guard logic. - Rate-limit reset batches are built per counter dimension (token, request) rather than per row, matching the independent expiry of each dimension. - Rows within each batch are sorted by ID before any transaction opens, so concurrent writers acquire row locks in a consistent order and deadlocks become structurally impossible rather than merely unlikely. - `isDumpDeadlock` centralises the deadlock-detection string checks that were previously duplicated in `DumpRateLimits` and `DumpBudgets`. On a deadlock the affected chunk is skipped and retried on the next cycle; earlier chunks in the same sweep remain committed, which is safe because every write is monotonically guarded. ## Type of change - [ ] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./plugins/governance/... ``` Run a multi-node deployment against a PostgreSQL instance in a separate region and confirm that budget and rate-limit dump cycles complete without blocking interactive virtual-key saves, and that no deadlock errors surface in the logs under concurrent load. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations No auth, secrets, PII, or sandboxing changes. All writes remain guarded by the same monotonic `last_reset` boundary as before; the batching does not relax any consistency guarantee. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary The Vertex `countTokens` endpoint rejects several fields that are valid in `generateContent` requests, causing 400 errors. Previously, the field-stripping logic incorrectly removed `generationConfig` and `systemInstruction`, which the `countTokens` endpoint actually supports and uses to produce accurate token counts. This PR fixes the set of fields being stripped and consolidates the logic into a dedicated helper. ## Changes - Introduced `stripVertexCountTokensUnsupportedFields` to replace the inline ad-hoc field deletions in `CountTokens`. The new helper drops only the fields the `countTokens` endpoint actually rejects: `toolConfig`, `safetySettings`, `cachedContent`, `serviceTier`, and `labels`. - Removed the incorrect stripping of `generationConfig` and `systemInstruction`, both of which are valid and meaningful to the `countTokens` endpoint. - Added unit tests covering: fields that should be preserved, fields that should be dropped, and a nil/empty body edge case. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/providers/vertex/... ``` Verify that `TestStripVertexCountTokensUnsupportedFields` passes, confirming that `systemInstruction`, `tools`, and `generationConfig` survive stripping while `toolConfig`, `safetySettings`, `cachedContent`, `serviceTier`, and `labels` are removed. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations No security implications. This change only affects which JSON fields are forwarded to the Vertex `countTokens` endpoint. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary The Gemini `countTokens` endpoint requires all prompt configuration (system instruction, tools, tool config, generation config) to be nested inside a `generateContentRequest` envelope rather than at the top level. Previously, these fields were simply stripped before sending, meaning token counts were incomplete. This PR rewrites the request body into the correct envelope shape so the full prompt is counted accurately. ## Changes - Introduced `wrapGeminiCountTokensBody` which rewrites any flat `generateContent`-style body into the `generateContentRequest` envelope, handles already-enveloped bodies without double-wrapping, qualifies the model name with the `models/` prefix, and strips fields not accepted by `GenerateContentRequest` (e.g. `labels`, `fallbacks`). - Added `GeminiCountTokensRequest` type that accepts both bare `contents` and the `generateContentRequest` envelope, with a `ToGeminiGenerationRequest` converter that resolves precedence between the two. - Replaced the previous approach of deleting `toolConfig`, `generationConfig`, and `systemInstruction` from the top-level body with the envelope wrapping strategy, which preserves those fields inside the envelope instead. - Extracted `AddModalityTokens` as a shared helper that maps Gemini modality strings (`audio`, `image`, `video`, text) to the neutral `ResponsesResponseInputTokens` fields, replacing duplicated inline logic in both `ToBifrostCountTokensResponse` and the Vertex equivalent. - Extended `ToGeminiCountTokensResponse` to round-trip `PromptTokensDetails` per modality and to respect `TotalTokens` when present. - Updated `VertexCountTokensResponse` to include `PromptTokensDetails` and `TotalBillableCharacters`, and wired `AddModalityTokens` into the Vertex count tokens response converter. - Added `SetRawJSONField` to provider utils to insert pre-encoded JSON verbatim without re-marshaling. - Updated the GenAI HTTP transport to parse count tokens requests into `GeminiCountTokensRequest` and route them through the new converter, including raw body passthrough for explicit Gemini requests. - Added comprehensive tests for `wrapGeminiCountTokensBody` and `ToGeminiGenerationRequest` covering envelope wrapping, double-wrap prevention, model prefix normalization, unsupported field stripping, and content fallback behavior. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/providers/gemini/... ./core/providers/vertex/... ./transports/bifrost-http/... ``` Send a `countTokens` request with a system instruction and tools attached and verify the returned token count reflects the full prompt rather than only the `contents` array. Confirm that a body already wrapped in `generateContentRequest` is not double-wrapped by inspecting the outgoing request payload. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations No auth, secrets, or PII implications. The change only affects how request bodies are serialized before being forwarded to the Gemini and Vertex APIs. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Extends the Gemini provider's Google Search grounding support to cover image search: `searchTypes` (web + image), image grounding chunks, image search queries, and search localization via `latLng`. Also fixes a bug where multi-source grounding supports were fanned out into one support per cited chunk instead of being regrouped by segment. ## Changes - Added `SearchTypes`, `WebSearch`, and `ImageSearch` types to `types.go`, with camelCase/snake_case `UnmarshalJSON` support. `GoogleSearch` now carries a `SearchTypes` field. - Added `GroundingChunkImage` to represent image-search grounding chunks (with `sourceUri`, `imageUri`, `title`, `domain`). `GroundingChunk` now includes an `Image` field alongside `Web`. - Added `ImageSearchQueries` to `GroundingMetadata` to keep image queries separate from web queries. - Introduced `groundingChunkSource` helper that normalizes both web and image chunks into a `ResponsesWebSearchToolCallActionSearchSource`, replacing scattered inline chunk-to-source conversions throughout `responses.go`. - `search_content_types` on `ResponsesToolWebSearch` now maps `"text"` → `WebSearch` and `"image"` → `ImageSearch` when converting to/from Gemini's `searchTypes`. - Search localization (`latLng` from `toolConfig.retrievalConfig`) is now round-tripped through `ResponsesToolWebSearchUserLocation.Latitude/Longitude` so it survives the Gemini → Bifrost Responses → Gemini hop. - `ImageQueries` added to `ResponsesWebSearchToolCallAction` and `ImageURL`/`Domain` added to `ResponsesWebSearchToolCallActionSearchSource` to carry image-specific grounding data through the neutral schema. - Fixed `buildGroundingMetadataFromWebSearch`: annotations were previously emitting one `GroundingSupport` per `(segment, chunk)` pair. They are now regrouped by segment key so multi-source supports correctly list all cited chunk indices rather than duplicating the segment. - `emitWebSearchFromGroundingMetadata` and `emitAnnotationsFromGroundingSupports` updated to handle image chunks via `groundingChunkSource` and to propagate `ImageQueries`. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/providers/gemini/... ``` New tests cover: - `TestGeminiGoogleSearchToolRoundTrip` — verifies `searchTypes`, `excludeDomains`, `timeRangeFilter`, and `latLng` survive a full Gemini → Bifrost Responses → Gemini round trip. - `TestGeminiGoogleSearchToolRoundTripSnakeCase` — same for snake_case input; asserts unselected search types are not synthesized. - `TestGeminiImageGroundingRoundTrip` — verifies image chunks keep their asset URL and domain, and that `imageSearchQueries` stay separate from `webSearchQueries`. - `TestGeminiGroundedRoundTripMergesMultiSourceSupports` — non-streaming path: four supports over four chunks, three citing two sources, must come back as four supports (not seven). - `TestGeminiGroundedStreamRoundTripMergesMultiSourceSupports` — streaming counterpart of the above. - `TestToBifrostCountTokensResponseCachedContent` — locks cached-token accounting so cache modalities are never folded into the breakdown a second time. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations No auth, secrets, or PII implications. `latLng` coordinates flow through the existing tool config path and are not persisted. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
… fails closed) (#5653) ## Summary Adds a `user_id` query parameter to the virtual keys list endpoint, allowing callers to filter virtual keys by their assigned user. Because the virtual key–user relationship lives in an enterprise-only table, the OSS build fails closed (matches no keys) rather than silently returning unfiltered results. The enterprise store overrides the base implementation to honour the filter correctly. ## Changes - Added `user_id` query parameter to the `GET /virtual-keys` OpenAPI spec, documented as enterprise-only and combined with `customer_id`/`team_id` using OR logic. - Refactored the assignment filter clause construction in `GetVirtualKeysPaginated` from a chain of if/else branches into a slice-based OR builder, making it straightforward to add the `user_id` clause. In the OSS build, a non-empty `user_id` injects `1 = 0` to ensure no rows are returned. - Added `UserID` to `VirtualKeyQueryParams` with a comment clarifying its OSS behaviour. - Wired `user_id` through the HTTP handler query arg parsing and into the params struct. - Added `user_id` URL state, filter prop threading, and a `handleUserFilterChange` callback in the virtual keys page. - Rendered the `UserPicker` component in the virtual keys table filter bar when a picker is registered (enterprise builds). The picker is hidden entirely in OSS builds where no picker is registered. An "or" label appears between the team/customer filters and the user filter when multiple assignment filters are active. - Extended `UserPickerProps` with `placeholder`, `className`, and `triggerClassName` to support use as a filter control. - Added `user_id` to `GetVirtualKeysParams` and the RTK Query API call. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [x] UI (React) - [x] Docs ## How to test ```sh # Core/Transports go test ./framework/configstore/... ./transports/bifrost-http/... # UI cd ui pnpm i pnpm build ``` **OSS build — user_id filter fails closed:** ``` GET /virtual-keys?user_id=some-user-id # Expected: empty result set (0 keys returned), no error ``` **Enterprise build — user_id filter returns matching keys:** ``` GET /virtual-keys?user_id=<valid-user-id> # Expected: only virtual keys assigned to that user are returned GET /virtual-keys?user_id=<valid-user-id>&team_id=<valid-team-id> # Expected: virtual keys assigned to that user OR belonging to that team ``` **UI (enterprise build):** Navigate to Governance → Virtual Keys. A user picker filter should appear in the filter bar. Selecting a user filters the table; clearing it restores the full list. The "or" label should appear between the team/customer filter and the user filter when both are active. **UI (OSS build):** The user picker filter should not appear in the filter bar. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations The OSS implementation explicitly returns no results when `user_id` is supplied, preventing any accidental data exposure from an unimplemented filter being silently ignored. The enterprise override is responsible for enforcing its own access control on the user-scoped query. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Fixes a regression where `:countTokens` for both Gemini (`generativelanguage`) and Vertex (`aiplatform`) reported only the `contents` token count because `systemInstruction`, `generationConfig`, and `toolConfig` were unconditionally stripped from the request body. In the reported case, a ~7.4k-token system prompt plus tool declarations counted as only 17 tokens on Vertex. The root cause is that the two endpoints require opposite request shapes: - **Gemini**: `systemInstruction`, `tools`, `toolConfig`, and `generationConfig` are rejected as `Unknown name` at the top level and must be wrapped inside a `generateContentRequest` envelope. Top-level `contents`/`model` are silently ignored when the envelope is present. - **Vertex**: These fields are accepted flat; only `toolConfig` (and a few others) must be stripped. There is no envelope. The `/genai` ingress must now parse both shapes and emit whichever the resolved provider requires. The Bifrost-only `fallbacks` routing field must be honoured for routing but stripped before the upstream call. ## Changes - Added E2E test suite **34. Gemini/Vertex countTokens full-prompt accounting** covering: - **34.1 / 34.6** – Contents-only baseline for Gemini and Vertex respectively, recording `totalTokens` as a collection variable for relative assertions in subsequent cases. - **34.2 / 34.7** – Flat `systemInstruction` is counted (not stripped) for both providers. - **34.3** – Flat `tools`, `toolConfig`, and `generationConfig` are wrapped into the Gemini envelope rather than forwarded verbatim (which would produce a hard 400). - **34.8** – Vertex keeps `tools` and `generationConfig` flat but strips `toolConfig`. - **34.4 / 34.9** – A `generateContentRequest` envelope is passed through for Gemini and unwrapped for Vertex. - **34.5** – The Bifrost `fallbacks` field is stripped before the upstream call and does not alter the counted token total. - Assertions are relative (count must exceed the contents-only baseline) rather than absolute, so they remain valid across model version updates. A dropped field collapses the count back to baseline, which is exactly the regression signature these cases catch. - HTTP 400 is intentionally **not** in the infra guard for the tools/toolConfig/fallbacks cases — a 400 from upstream is itself the regression signature for those scenarios. ## Type of change - [x] Bug fix ## Affected areas - [x] Providers/Integrations ## How to test Import the updated `provider-harness.json` collection into Postman or Newman and run group **34** with valid `genaiKey`, `genaiModel`, `vertexModel`, and `baseUrl` collection variables set. ```sh newman run tests/e2e/api/collections/provider-harness.json \ --folder "34. Gemini/Vertex countTokens full-prompt accounting (PR #5620)" \ --env-var baseUrl=<your-base-url> \ --env-var genaiKey=<your-api-key> \ --env-var genaiModel=gemini-2.5-pro \ --env-var vertexModel=gemini-2.5-pro ``` Each case should return HTTP 2xx with `totalTokens` strictly greater than the baseline recorded in 34.1/34.6. Any case that returns an `Unknown name` error or a token count equal to the baseline indicates the regression has re-appeared. ## Breaking changes - [x] No ## Related issues Closes #5620 ## Security considerations No auth, secrets, or PII changes. Test requests use a scoped no-op tool declaration (`probe_context`) with no real side effects. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
…pages (#5660) ## Summary Workspace pages were missing consistent padding and were relying on parent-level padding that wasn't always applied correctly. This PR standardizes layout spacing across workspace pages by adding `no-padding-parent` and `p-4` classes where needed, and fixes a few overflow/height issues in specific views. ## Changes - Added `no-padding-parent p-4` to the wrapper divs in Adaptive Routing, Caching, Client Settings, Proxy, Security, Guardrails Providers, MCP Settings, and Config pages so each page manages its own padding independently. - Added `no-padding-parent` (without `p-4`) to Logging and Observability pages, which handle their own internal spacing. - Moved padding into `ObservabilityView` directly via `p-4` on its root flex container. - Added `py-6` to `LoggingView`'s root container to restore vertical spacing. - Removed the fixed `h-[calc(100vh-50px)]` and `overflow-y-auto` from `SecurityView`, allowing the layout to flow naturally within the parent scroll context. - Changed the sticky save button padding in `SecurityView` from `pt-2` to `py-2` for balanced spacing. - Wrapped `MCPView` in a properly padded container div and added a trailing newline. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh cd ui pnpm i || npm i pnpm build || npm run build ``` Navigate to each affected workspace page (Adaptive Routing, Caching, Client Settings, Logging, Proxy, Security, Guardrails Providers, MCP Settings, Observability) and verify that content is consistently padded and no pages appear flush against the edges or have unexpected scroll behavior. ## Screenshots/Recordings Before/after screenshots recommended for Security, Observability, and Logging pages where layout behavior changed most significantly. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…patibility (#5071) The custom pprof server wrote profile payloads without setting a Content-Type, so net/http sniffed the gzipped protobuf and labeled responses application/x-gzip. Pyroscope/Grafana Alloy scrapers require application/octet-stream and rejected every profile with "invalid profile data: unexpected Content-Type application/x-gzip". Explicitly set Content-Type (and X-Content-Type-Options: nosniff plus Content-Disposition) before the first write on the Lookup, CPU profile, and trace handlers, matching the standard net/http/pprof behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Bumps the Bifrost Helm chart to v2.1.32, introducing per-model governance controls on access profile provider configs, additional OAuth scopes for SCIM/Okta, and validation for the Bedrock batch job role ARN. ## Changes - Extended `bifrost.accessProfiles[].provider_configs[]` with three new fields: `blacklisted_models` (a denylist that wins over `allowed_models`; `["*"]` blocks all models), `weight` (load-balancer seed weight; `null` opts out of weighted routing), and `model_budgets[]` (per-model budget groups, each carrying `budgets` and an optional `rate_limit`). These pass through into `access_profiles[].provider_configs[]` in config.json and are validated in both `values.schema.json` and `transports/config.schema.json`. - Added `bifrost.scim.config.additionalScopes` — an array of extra OAuth scopes appended to the base `openid/profile/email/offline_access` set. This is needed for Okta Custom Authorization Servers where claims like `groups` are gated behind a scope that Bifrost does not request by default. The field is validated in both `values.schema.json` and `transports/config.schema.json` so configs are validated at startup. - Documented and validated `bifrost.providers.<provider>.keys[*].bedrock_key_config.batch_role_arn` — the service role ARN passed to Bedrock batch jobs for S3 access, which takes priority over any `role_arn` sent in the request. - Added the v2.1.32 changelog entry and registered it in the docs navigation. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [x] Docs ## How to test Deploy the updated Helm chart and verify the new fields are accepted: ```sh # Validate the Helm chart schema helm lint helm-charts/bifrost # Install/upgrade with additionalScopes and model governance configured helm upgrade --install bifrost helm-charts/bifrost \ --set bifrost.scim.config.additionalScopes[0]=groups # Confirm the generated config passes schema validation at startup by # checking Bifrost logs for no schema validation errors on boot. ``` **New config fields:** | Field | Type | Description | |---|---|---| | `bifrost.accessProfiles[].provider_configs[].blacklisted_models` | `[]string` | Denylist of models blocked even if matched by `allowed_models`; `["*"]` blocks all | | `bifrost.accessProfiles[].provider_configs[].weight` | `number\|null` | Load-balancer seed weight; `null` opts out of weighted routing | | `bifrost.accessProfiles[].provider_configs[].model_budgets[]` | `object[]` | Per-model budget groups with `budgets` and optional `rate_limit` | | `bifrost.scim.config.additionalScopes` | `[]string` | Extra OAuth scopes beyond `openid/profile/email/offline_access` | | `bifrost.providers.<provider>.keys[*].bedrock_key_config.batch_role_arn` | `string` | Service role ARN for Bedrock batch job S3 access | ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations `additionalScopes` expands the OAuth token scopes requested from Okta. Operators should ensure only necessary scopes are added to follow least-privilege principles. `batch_role_arn` is an IAM role ARN and should be scoped to the minimum permissions required for Bedrock batch S3 access. `blacklisted_models` and `weight` affect routing and access control — misconfiguration could inadvertently expose or block model access. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
Add gemini-3.5-flash-lite to vertexFlexModels so service_tier flex propagates X-Vertex-AI-LLM-Shared-Request-Type on the global endpoint. Co-authored-by: gstvortiz1 <300627164+gstvortiz1@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: clarify SDK virtual key placeholders * docs: clarify virtual key usage and Codex model catalogs - document prefer_vk handling for duplicate virtual key headers - expand the Codex CLI model catalog example - correct the Gemini API key placeholder --------- Co-authored-by: Raggav Subramani <raggavsubramani@Raggavs-MacBook-Pro.local>
…deferred priority check (#5678) ## Summary When reloading routing rules from a config file, creates and updates were applied one-by-one, causing the unique-priority-per-scope constraint to be checked after each individual write. This meant valid permutations — such as swapping the priorities of two existing rules — would fail with a spurious duplicate-priority error mid-sync, even though the final state was valid. ## Changes - Added `SyncRoutingRules` to `RDBConfigStore` and the `ConfigStore` interface, which accepts batches of rules to create and update, applies them all within a single transaction, and defers the unique-priority-per-scope check until every rule has been written. Genuine end-state duplicates still produce an error. - Replaced the sequential `CreateRoutingRule` / `UpdateRoutingRule` loop in `updateGovernanceConfigInStore` with a single `SyncRoutingRules` call, so config-file reloads that permute priorities no longer fail on transient intermediate collisions. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test Verify that swapping the priorities of two existing routing rules in a config file reload succeeds without a duplicate-priority error, and that a config file with a genuine end-state priority collision still returns an error. ```sh go test ./framework/configstore/... go test ./transports/bifrost-http/... ``` ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations No auth, secrets, PII, or sandboxing implications. The deferred constraint check ensures the same invariant is enforced as before — only the point in the transaction at which it is evaluated has changed. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Cost recomputation was systematically mis-pricing requests due to three compounding issues: (1) the `SearchLogs` projection used for billing omitted modality output payloads and, on hybrid object-storage-backed stores, returned rows whose `token_usage` was blanked at write time — causing `DeserializeFields` to rebuild a lossy stub where `PromptTokens` is inclusive of cache buckets, inflating cache-heavy requests by 2–4x; (2) the `Log` table had no columns for the served billing tier (`service_tier`, `speed`, `inference_geo`), so every row repriced at standard rates regardless of what was actually served; and (3) several modality-specific pricing inputs (OCR pages, 1h cache-write split, server-side fallback model) were dropped during response reconstruction. A user observed approximately 3x overbilling in production on a cache-heavy Anthropic request. ## Changes - **New `SearchLogsForBilling` interface method** on `LogStore` that selects the full billing projection (including modality output payloads) and, on `HybridLogStore`, concurrently hydrates offloaded payloads from object storage. Rows whose payloads cannot be recovered (content-hidden, missing object) are returned as an `unpriceable` ID list rather than silently handed back as stubs. - **`billingSelectColumns` / `billingPayloadColumns`** added to `RDBLogStore`. The list projection (`listSelectColumns`) deliberately stays free of unbounded output blobs to avoid reintroducing the Cloud Run 32 MB body-limit failure. The billing projection extends it with `speech_output`, `transcription_output`, `image_generation_output`, `video_generation_output`, and `ocr_output`. - **`IsUsageDegraded` flag** on `Log`. `DeserializeFields` marks the stub it builds from denormalized columns as degraded; `calculateCostForLog` refuses to price a degraded row and returns `errPricingInputsUnavailable`. The flag clears when a real `token_usage` payload is subsequently parsed (the hybrid hydration path calls `DeserializeFields` twice). - **New denormalized columns**: `cached_write_tokens`, `service_tier`, `speed`, `inference_geo` on the `Log` table, added via `migrationAddBillingFidelityColumns`. `cached_write_tokens` is backfilled from existing `token_usage` JSON via `ensureBillingFidelityBackfill` (deferred out of the migration to avoid blocking pod startup). The tier columns cannot be backfilled — pre-migration rows reprice at standard rates. - **`applyServedTierToEntry`** captures the served tier at write time from both the response envelope (non-streaming) and the usage struct (streaming, where no envelope is accumulated). Called from `applyNonStreamingOutputToEntry`, `applyStreamingOutputToEntry`, and `applyRealtimeOutputToEntry`. - **`servedTierFromLog` / `buildResponseForRequestType`** restore the tier onto the reconstructed `BifrostResponse` so `CalculateCost`'s `tierFromResponse` sees the correct multipliers. - **OCR billing fix**: `resp.OCRResponse` now has `Pages`, `UsageInfo`, and `DocumentAnnotation` patched back in from `OCROutputParsed`. Previously the reconstructed response reported zero pages and the entire request priced to nothing. - **1h cache-write split fix**: `CachedWriteTokenDetails` is now forwarded through `buildResponseForRequestType` for the Responses path, preventing 1h cache writes from billing at the cheaper 5m rate. - **Server-side fallback model fix**: `logEntry.ServerSideFallbackModel` is now set on `RoutingInfo.ServerSideFallbackModel` so `resolvePricing` prices at the model that actually ran. - **Alias-without-canonical fix**: `ResolvedKeyAlias.ModelID` is now populated whenever `logEntry.Alias` is set, independent of whether a canonical name exists, so per-deployment override pricing resolves correctly. - **`Unpriceable` counter** added to `CostRecalcJobMeta`, `RecalculateCostResult`, and the API status struct, distinguishing "nothing to charge" from "refused to write a number known to be wrong". - **`costRecalcBatchSize` reduced from 1000 to 200** to bound the object-storage fan-out (one `Get` per offloaded row per batch). - **`waitForOffload` test helper** waits for both the object upload and the `has_object` DB flag commit, fixing a race in `TestHybrid_ContentHiddenStripsDBRowAndSkipsHydration` where `processUpload` writes the flag after the `Put`. ## Type of change - [x] Bug fix - [x] Feature ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Plugins ## How to test ```sh go test ./framework/logstore/... ./plugins/logging/... ``` Key test files: - `framework/logstore/billingprojection_test.go` — projection separation and end-to-end modality output delivery - `framework/logstore/hybridbilling_test.go` — hydration, content-hidden reporting, fetch-failure reporting, non-offloaded passthrough - `plugins/logging/costfidelity_test.go` — cache breakdown parity, degraded-usage refusal, tier preservation, 1h cache-write pricing, server-side fallback model, unpriceable accounting ## Breaking changes - [x] No The `SearchLogsForBilling` method is additive to the `LogStore` interface. Existing `fakeRecalcStore` test doubles require a `SearchLogsForBilling` implementation (see `costrecalc_test.go` for the pattern: delegate to `SearchLogs` and return a nil unpriceable list). ## Security considerations None. No auth, secrets, or PII surface changes. The new columns store provider-echoed tier strings (`"priority"`, `"flex"`, `"fast"`, `"us"`) that are already present in the response payloads. ## Checklist - [x] I added/updated tests where appropriate - [x] I verified builds succeed (Go and UI)
## Summary Anthropic rejects messages that contain a document block without an accompanying text block, returning a validation error: *"A text block must be included when using documents."* This PR fixes that by automatically prepending a single-space placeholder text block whenever a message contains one or more document blocks but no text block. ## Changes - When converting a Bifrost chat request to an Anthropic request, document-only message content is detected and a minimal placeholder text block (`" "`) is prepended so the request passes Anthropic's validation. - Messages that already include a text block alongside document blocks are left unchanged. - Two tests cover both cases: a document-only message receiving the placeholder, and a message with an existing text block not receiving an extra one. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/providers/anthropic/... ``` Expected: all tests pass, including the two new tests: - `TestToAnthropicChatRequest_DocumentOnlyMessageGetsPlaceholderTextBlock` - `TestToAnthropicChatRequest_DocumentWithTextDoesNotGetPlaceholder` ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations None. The placeholder text block is a single space and does not expose any sensitive data. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Briefly explain the purpose of this PR and the problem it solves. ## Changes - What was changed and why - Any notable design decisions or trade-offs ## Type of change - [ ] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test Describe the steps to validate this change. Include commands and expected outcomes. ```sh # Core/Transports go version go test ./... # UI cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` If adding new configs or environment variables, document them here. ## Screenshots/Recordings If UI changes, add before/after screenshots or short clips. ## Breaking changes - [ ] Yes - [ ] No If yes, describe impact and migration instructions. ## Related issues Link related issues and discussions. Example: Closes #123 ## Security considerations Note any security implications (auth, secrets, PII, sandboxing, etc.). ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary This PR cuts the `core v1.7.5`, `framework v1.5.5`, `transports v1.6.7`, and associated plugin releases. It delivers a broad set of provider correctness fixes, new routing and governance capabilities, streaming reliability improvements, and database connection controls. ## Changes - **Virtual Key Budget Overrides**: Temporary budget overrides for virtual keys are now supported across the DB schema, governance store, admin APIs, and UI. Overrides add `override_amount` on top of `max_limit` and run for a finite number of reset cycles or until removed, controlled via `override_mode`, `override_cycles_total`, and `override_anchor_reset`. - **Stream Truncation Detection**: Added `SSETruncation` interface and EOF handler support across all providers so upstream stream death surfaces as an error instead of a clean `[DONE]`. Fixes silent discard of dead streams. - **Empty Stream Nil Channel**: `*StreamRequest` now returns a closed non-nil channel for empty streams instead of `(nil, nil)`, preventing consumers from hanging on a nil-channel receive (thanks [@kharkevich](https://github.com/kharkevich)!). - **User Scope for Routing and Pricing**: Routing rules and pricing overrides can now be scoped to individual users via a `user_id` CEL variable, an enterprise user picker in the pricing overrides UI, and streaming request-type options in the routing rules UI. - **Background Model Catalog Refresh**: Provider model lists are now re-fetched in the background on a configurable `live_models_sync_interval` (default 1 hour). Provider reload no longer wipes the catalog before refetching, so a transient failure cannot empty it. - **Responses Retrieve Stream**: Added a retrieve-stream method for the Responses API across core, framework, governance, and logging plugins. - **Routing Info Headers**: Routing info headers are now emitted for streaming responses, inference and integration APIs, and error/passthrough paths, with corrected behavior on streaming fallbacks. - **Bedrock Mantle**: Added count-tokens API support, merged additional tools into the tools list, and set structured outputs to false for Claude models. - **Connector Latency Data**: Bifrost latency and overhead duration are now exported to connectors (OTel and others). - **Database Connection Controls**: Added `conn_max_idle_time` (default 5m) to config and logs stores, `cache_ttl` (default 60s) for password-command credential resolution, and `matview_refresh_timeout` to bound a single materialized view refresh pass. - **Matview Improvements**: Added cached tokens to the materialized view, enabled on-the-fly refresh, corrected hybrid matview count to prevent boundary bucket over-counting, added `customer_id`/`business_unit_id` to `scopeProjection`, and reverted the read-path shape check that gated reads during rolling deploys. - **OTel Export Timeout**: Added an `export_timeout` setting (default 5s) to bound slow or unreachable collector exports, which previously had no timeout on gRPC exports. - **Multinode Override Counts**: Corrected override counts for multinode setups, resolving high CPU in governance rate-limit reset. - **vLLM Responses Streaming**: vLLM responses-stream chunks and completion events are now forwarded instead of silently discarded, and stream truncation is handled correctly. - **Gemini Tool Schema Constraints**: Valid integer constraints in tool schemas are no longer rejected with a 400 INVALID_ARGUMENT. - **HuggingFace Model IDs**: Backfilled model IDs no longer duplicate the inference-provider segment. - **Azure Responses API Version**: Removed the default preview api-version for GA Azure Responses endpoints. - **GenAI API**: Fixed image search for the GenAI search tool, honored `IncludeServerSideToolInvocations`, and followed redirects for downloads in GenAI passthrough. - **Tool Search Wire Shape**: Preserved the `tool_search` Responses API wire shape (thanks [@devonpmack](https://github.com/devonpmack)!). - **Replicate Image Generation**: Input images are now handled in image generation requests. - **SDK Compatibility**: `strict: null` is now converted to `false`. - **New Guardrails**: Added Lakera and Repello Argus as guardrail integrations with configuration docs and UI branding. - **Access Profile Config Schema**: `config.schema.json` now accepts `blacklisted_models`, `weight`, and `model_budgets` on access profile provider configs. - **Async Entity Selectors**: Teams, customers, and virtual keys are loaded through async selector components; the customer list returns a server-computed `virtual_key_count` instead of a full preload. - **Go 1.26.5**: Upgraded the toolchain and all builder images. ## Type of change - [x] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [x] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [x] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go version # should report go1.26.5 go test ./... # UI cd ui pnpm i pnpm test pnpm build ``` **New config knobs to validate:** - `live_models_sync_interval` — background provider model-list refresh interval (default `1h`, `0` disables). - `conn_max_idle_time` — idle physical connection lifetime for config and logs stores (default `5m`). - `cache_ttl` — TTL for password-command credential resolution (default `60s`). - `matview_refresh_timeout` — maximum duration for a single materialized view refresh pass. - `export_timeout` (OTel plugin) — maximum duration for a single OTel export call (default `5s`). - `override_mode`, `override_cycles_total`, `override_anchor_reset` — virtual key budget override fields. ## Breaking changes - [ ] Yes - [x] No ## Related issues - Closes [#4215](#4215): HuggingFace models show provider ID twice in `/v1/models` - Closes [#4851](#4851): Governance rate-limit reset causes high CPU in `BumpRateLimitUsage`/`updateRateLimitReferences` - Closes [#5329](#5329): `/api/logs` returns incorrect `total_count` for time ranges ≥ 24 hours - Closes [#5433](#5433): `/genai` endpoint rejects valid `minLength`/`maxLength` in tool schemas - Closes [#5504](#5504): vLLM streaming Responses API hangs forever, chunks silently discarded - Closes [#5546](#5546): Upstream SSE stream death swallowed into a clean `[DONE]` - Closes [#5551](#5551): `transports/bifrost-http/lib` test package does not compile (`MockConfigStore` missing `UpdateBudgetOverride`) - Closes [#5552](#5552): Models added after boot stay invisible until restart - Closes [#5554](#5554): Provider reload wipes the live model catalog before refetching - Closes [#5555](#5555): `*StreamRequest` returns `(nil, nil)` for empty streams, consumers hang forever ## Security considerations - `cache_ttl` for password-command credential resolution reduces how frequently credentials are re-fetched from external commands; operators should set an appropriate TTL for their secret rotation cadence. - The `user_id` virtual keys filter is enterprise-only and fails closed on OSS builds. - Budget override fields are gated behind the governance store and admin APIs; no public-facing exposure without authentication. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
## Summary Plugins running inside Bifrost hooks had no way to look up model pricing or capability metadata without taking a direct dependency on the framework package, which core cannot import. This PR wires a `ModelInfoProvider` interface into the request context so every plugin hook — pre-request, pre-LLM, post-LLM, streaming, realtime, and MCP tool execution — can call `ctx.GetModelInfo(provider, model)` and `ctx.CalculateCost(resp)` without any extra construction-time wiring. ## Changes - **`core/schemas/modelcatalog.go`** — introduces the `ModelInfoProvider` interface (declared in core, implemented in framework) mirroring how `Tracer` is wired: interface here, concrete type there, handle stamped onto the context per request. - **`core/schemas/bifrost.go`** — adds `BifrostContextKeyModelCatalog` context key and `ModelCatalog ModelInfoProvider` field to `BifrostConfig`. - **`core/schemas/context.go`** — adds `GetModelInfo` and `CalculateCost` accessor methods on `BifrostContext`. Both are inert (return nil / 0) when no catalog is configured, keeping plugins portable between the HTTP gateway and bare Go SDK embeddings. - **`core/bifrost.go`** — stores the catalog from `BifrostConfig` at `Init`, adds `setModelCatalogOnContext` which stamps (or clears) the handle on every entry point: `handleRequest`, `handleStreamRequest`, `RunPreRequestHooks`, `RunStreamPreHooks`, `RunRealtimeTurnPreHooks`, `ExecuteChatMCPTool`, and `ExecuteResponsesMCPTool`. The clear-on-nil path ensures long-lived realtime contexts don't serve a catalog that was subsequently removed. - **`framework/modelcatalog/modelinfo.go`** — adds `GetModelInfo` and `CalculateRequestCost` on `*ModelCatalog`, and extracts `ApplyModelInfo` as an exported function. All reference-typed fields (pointer scalars, slices, maps, `Architecture`) are deep-cloned on the way out so third-party plugin code cannot accidentally mutate catalog state or race the pricing sync. - **`transports/bifrost-http/handlers/inference.go`** — replaces the inline enrichment block in `enrichListModelsResponse` with a call to `modelcatalog.ApplyModelInfo`, so the list-models endpoint and `ctx.GetModelInfo` always use identical mapping logic. - **`transports/bifrost-http/handlers/middlewares.go`** — stamps the catalog onto the context in `TransportInterceptorMiddleware` so `HTTPTransportPreHook` sees it before the inference path runs. - **`transports/bifrost-http/server/server.go`** — passes `ModelCatalog` through to `BifrostConfig` at bootstrap. - **Docs** — `plugins.mdx` documents the plugin-to-Bifrost communication model; `writing-go-plugin.mdx` adds full reference entries for both accessors; `writing-wasm-plugin.mdx` notes that these accessors are not reachable from WASM. Notable design decisions: - The interface is declared in `core/schemas` and implemented in `framework/modelcatalog` to preserve the existing dependency direction (framework → core, never core → framework). - `CalculateRequestCost` is named differently from `ModelCatalog.CalculateCost` to avoid a signature collision on the concrete type while still letting the plugin-facing wrapper apply the full governance pricing chain. - The clear-on-nil behaviour in `setModelCatalogOnContext` is intentional: reused WebSocket contexts re-enter the stamping path on every message, so skipping the clear would leave a stale handle after `SetModelCatalog(nil)`. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (React) - [x] Docs ## How to test ```sh go test ./core/... ./core/schemas/... ./framework/modelcatalog/... ./transports/bifrost-http/... ``` Two new integration tests in `core/modelcataloghooks_test.go` drive a real `ChatCompletionRequest` through a probe plugin and assert: - `TestModelCatalogReachesPluginHooksOnRealRequest` — all three hook phases (`PreRequestHook`, `PreLLMHook`, `PostLLMHook`) receive the catalog handle and return the expected model info and cost. - `TestModelCatalogAbsentLeavesHooksInert` — with no catalog wired, all accessors return zero values and no panic occurs. Unit tests in `core/schemas/modelcatalog_test.go` cover delegation, empty-argument guards, plugin-scope visibility, and derived-context inheritance. Unit tests in `framework/modelcatalog/modelinfo_test.go` cover pricing population, deprecation reporting, unknown model handling, provider-value preservation, and deep-clone correctness for all mutable fields. ## Breaking changes - [ ] Yes - [x] No ## Security considerations The `ModelInfoProvider` handle is stamped onto the request context and read back via a typed assertion. No credentials or secrets pass through this path. The deep-clone in `ApplyModelInfo` prevents a plugin from mutating shared catalog state, which would otherwise be a data-race vector on the pricing sync goroutine. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
Contributor
|
Important Review skippedToo many files! This PR contains 484 files, which is 184 over the limit of 300. To get a review, narrow the scope: Usage-priced reviews support at most 300 files. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (33)
📒 Files selected for processing (485)
You can disable this status message by setting the Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✨ Features
override_amounton top ofmax_limitand runs either for a finite number of reset cycles or until explicitly removed, configured via the newoverride_mode,override_cycles_total, andoverride_anchor_resetfields.user_idCEL variable in routing rules, an enterprise user picker in the pricing overrides UI, and streaming request-type options in the routing rules UI.user_idFilter: Added auser_idfilter to the virtual keys list (enterprise-only; OSS fails closed).config.schema.jsonnow acceptsblacklisted_models(a denylist that wins overallowed_models), aweightseed for weighted routing, andmodel_budgetsfor per-model budgets and rate limits on access profile provider configs. These keys previously failed schema validation because the object rejects unknown properties.RankingLimitfilter withall/limitquery params to cap or remove ranking row limits, and dashboard PDF/CSV exports now use uncapped snapshots.cache_hit_typesfilter to dashboard URL state and query params.TeamSelector/CustomerSelector/VirtualKeySelectorcomponents instead of preloading full lists, and the customer list returns a server-computedvirtual_key_countrather than a fullVirtualKeyspreload.live_models_sync_interval(default 1 hour,0disables), so models an upstream starts serving after boot no longer stay invisible until restart.RestartLiveModelRefresheris exported for custom boot paths.SSETruncationinterface and EOF handler support across all providers, so upstream stream death surfaces as an error instead of a clean[DONE].ModelReasoningschema field and provider-qualified model ID resolution for model-parameters lookup, plus a requiredmodelquery param and 404 response ongetModelParameters.batch_role_arnto Bedrock key config, a service role ARN passed to Bedrock batch jobs for S3 access that takes priority over anyrole_arnin the request.export_timeoutsetting (default 5s) bounding how long a slow or unreachable collector can hold an export goroutine, which previously had no timeout on gRPC exports.conn_max_idle_time(default 5m) to both the config and logs stores,cache_ttl(default 60s) for password-command credential resolution so the command no longer runs on every new physical connection, andmatview_refresh_timeoutbounding a single materialized view refresh pass.config.schema.jsonnow acceptsadditionalScopes, requesting extra OAuth scopes on top of the baseopenid/profile/email/offline_accessset, for Custom Authorization Servers that gate claims such asgroupsbehind a scope Bifrost does not request by default.updateTeamnow acceptscustomer_id/customer_idsattachment and returns anUpdateTeamResponseschema.ProviderConfigCardcomponent with sharedbudgetOutlinehelpers.CopyableIdcomponent to customer, team, and virtual key detail sheets.🐞 Fixed
*StreamRequestnow returns a closed non-nil channel for empty streams instead of(nil, nil), which previously hung consumers on a nil-channel receive (thanks @kharkevich!)application/octet-streamfor scraper compatibility (thanks @tcx4c70!)tool_searchResponses API wire shape (thanks @devonpmack!)customer_id/business_unit_idto the matviewscopeProjectionso team-data DAC scope resolves without column errors.IncludeServerSideToolInvocations, and followed redirects for downloads in GenAI passthrough.output_textuser blocks.[DONE]marker emitted for error frames.strict: nullis now converted tofalse.no-padding-parentand consistent padding across workspace pages.FullPageLoaderon initial fetch instead of flashing an empty state.pickTopSerieswithcomputeDisplaySeriesso chart and legend series order stay in sync.parseTrialExpirynow supports RFC3339 timestamps, and banner background colors were updated.hideSearchIconprop toModelMultiselect.🔧 Maintenance
🐙 Closed GitHub Issues
/v1/models, which breaks requestsBumpRateLimitUsage/updateRateLimitReferences/api/logsreturns an incorrecttotal_countfor time ranges of 24 hours or longer/genaiendpoint rejects validminLength/maxLengthin tool schemas (400 INVALID_ARGUMENT)[DONE], so dead streams appear successfultransports/bifrost-http/libtest package does not compile on dev (MockConfigStoremissingUpdateBudgetOverride)*StreamRequestreturns(nil, nil)for empty streams, so consumers hang forever on a nil-channel receive