feat[bc|notask]: migrate AI SDK provider to v7 and fix dynamic tools#3370
feat[bc|notask]: migrate AI SDK provider to v7 and fix dynamic tools#3370danielAsaboro wants to merge 10 commits into
Conversation
github-actions
Bot
added
the
community-contribution
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
simon-iribarren
left a comment
simon-iribarren
left a comment
There was a problem hiding this comment.
This is a breaking public change, so the title should use [bc] rather than [api|bc]. Please also document the Node 22+, AI SDK 7/provider-v4, and ESM-only requirements in the README. Full CI is currently skipped because the PR lacks the verified label.
| ): T & { tools_compact: boolean } { | ||
| return { | ||
| ...(request.generationParams ?? ({} as T)), | ||
| tools_compact: requestUsesCompactTools(request.tools, options) |
There was a problem hiding this comment.
Blocking: tools_compact does not currently cross the native addon boundary. parseGenerationParams() in AddonJs.hpp never reads this field, while LlamaModel only configures the tools-compaction controller at model load. Since this PR also removes the load-time setting, tool-bearing dynamic requests will no longer enable compaction. Please add native per-request support and an integration test through the real addon boundary.
There was a problem hiding this comment.
Fixed in d58e8851. tools_compact now crosses the JS/native boundary and is applied per request, with isolated continuous-batch controllers. Added real-addon coverage for enablement, explicit disable/reset, and mixed batches.
|
|
||
| async function throwResponseError(operation: string, response: Response): Promise<never> { | ||
| const detail = (await response.text()).slice(0, 1_000) | ||
| throw new Error(`QVAC ${operation} failed (${response.status}): ${detail}`) |
There was a problem hiding this comment.
These new HTTP adapter paths throw plain Error objects for non-2xx responses. AI SDK retry behavior and consumer error inspection rely on structured provider errors such as APICallError, including status, response body/headers, and retryability. Please use the provider-utils error helpers or equivalent structured QVAC errors across the audio, files, and file-reference adapters.
There was a problem hiding this comment.
Fixed in d58e8851. Audio, upload, and file-reference failures now use provider-utils response handlers and surface structured APICallError metadata, including status, body, headers, request values, and retryability.
danielAsaboro
changed the title
There was a problem hiding this comment.
Pull request overview
This PR upgrades @qvac/ai-sdk-provider to AI SDK 7 / provider v4, expands the provider surface to include native files + speech + transcription, and fixes a llama.cpp dynamic-tools regression by making tools_compact a per-request decision (explicitly false for toolless prompts) across single and batch completions. It also preserves uploaded MIME types in the CLI ephemeral-file store.
Changes:
- Migrate the AI SDK provider package to AI SDK 7/provider v4 (Node 22+, ESM-only) and compose native QVAC files/transcription/speech capabilities over the OpenAI-compatible transport.
- Make
tools_compactrequest-scoped in the SDK and addon (single + continuous batch), including per-request enable/disable and safe handling for toolless prompts. - Preserve multipart MIME types through the CLI ephemeral file store and validate via HTTP e2e tests.
Reviewed changes
Copilot reviewed 36 out of 37 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/sdk/test/unit/tools-compact.test.ts | Adds unit coverage for request-scoped tools_compact computation and mixed-batch behavior. |
| packages/sdk/test/unit/llm-plugin-transform.test.ts | Asserts tools_compact is not set at model-load transform time. |
| packages/sdk/test/bare/tools-compact.test.ts | Bare-runtime regression test ensuring toolless dynamic requests succeed with tools_compact: false. |
| packages/sdk/server/bare/plugins/llamacpp-completion/transform.ts | Removes model-load-time mapping from tools mode to tools_compact. |
| packages/sdk/server/bare/plugins/llamacpp-completion/ops/tools-compact.ts | Introduces shared helpers for computing/merging request-scoped tools_compact. |
| packages/sdk/server/bare/plugins/llamacpp-completion/ops/completion-stream.ts | Applies request-scoped tools_compact in single completion requests and preserves other per-request params. |
| packages/sdk/server/bare/plugins/llamacpp-completion/ops/batch-completion-stream.ts | Applies request-scoped tools_compact per batch slot and always forwards generation params. |
| packages/sdk/schemas/tools.ts | Updates docs to reflect request-scoped tools_compact behavior for dynamic tools. |
| packages/llm-llamacpp/test/unit/test_tools_compact_controller.cpp | Extends controller unit tests for enable/disable and unsupported behavior. |
| packages/llm-llamacpp/test/integration/tools-compact.test.js | Adds integration coverage for per-request override and mixed-slot continuous batch isolation. |
| packages/llm-llamacpp/test/integration/generation-params.test.js | Adds validation that tools_compact overrides must be boolean at the JS boundary. |
| packages/llm-llamacpp/index.js | Enforces runtime type checking for generationParams.tools_compact. |
| packages/llm-llamacpp/index.d.ts | Exposes tools_compact?: boolean as a per-request generation param override. |
| packages/llm-llamacpp/addon/src/model-interface/ToolsCompactController.hpp | Enables controller to start disabled and supports per-request toggling. |
| packages/llm-llamacpp/addon/src/model-interface/ToolsCompactController.cpp | Implements setEnabled() and ensures disabling clears boundaries and validation is gated by enabled state. |
| packages/llm-llamacpp/addon/src/model-interface/TextLlmContext.hpp | Adds virtual hook to toggle tools compaction per request. |
| packages/llm-llamacpp/addon/src/model-interface/TextLlmContext.cpp | Rebuilds chat templates when the effective tools-compaction state changes. |
| packages/llm-llamacpp/addon/src/model-interface/MtmdLlmContext.hpp | Adds per-request tools-compaction toggle for multimodal contexts. |
| packages/llm-llamacpp/addon/src/model-interface/MtmdLlmContext.cpp | Mirrors template rebuild logic for multimodal contexts. |
| packages/llm-llamacpp/addon/src/model-interface/LlmContext.hpp | Adds tools_compact override plumbing + a dedicated setter separate from sampler overrides. |
| packages/llm-llamacpp/addon/src/model-interface/LlamaModel.hpp | Tracks a load-time default and supports request overrides. |
| packages/llm-llamacpp/addon/src/model-interface/LlamaModel.cpp | Applies per-request tools_compact, logs unsupported overrides, and wires defaults into schedulers. |
| packages/llm-llamacpp/addon/src/model-interface/ContinuousBatchScheduler.hpp | Carries a tools-compaction default through the scheduler. |
| packages/llm-llamacpp/addon/src/model-interface/ContinuousBatchScheduler.cpp | Instantiates per-seq tools controllers with request-specific enabled state and warnings on unsupported models. |
| packages/llm-llamacpp/addon/src/addon/AddonJs.hpp | Parses tools_compact as an optional boolean generation override from JS. |
| packages/cli/test/e2e/http/files-validation.test.ts | Verifies /v1/files/:id/content returns the original uploaded Content-Type. |
| packages/cli/src/serve/routes/files.ts | Stores upload MIME type in the ephemeral file record. |
| packages/ai-sdk-provider/test/provider.test.ts | Expands provider tests for v4 surface, local audio endpoints, files upload/reference resolution, and APICallError shape. |
| packages/ai-sdk-provider/test/managed-provider.test.ts | Verifies managed-mode provider exposes the same expanded surface. |
| packages/ai-sdk-provider/src/types.ts | Updates provider types to include files/transcription/speech v4 contracts. |
| packages/ai-sdk-provider/src/provider.ts | Composes custom provider capabilities and wraps language models for local file reference resolution. |
| packages/ai-sdk-provider/src/managed/index.ts | Reuses createExternalQvac for managed-mode provider construction. |
| packages/ai-sdk-provider/src/files.ts | Implements AI SDK v4 Files adapter backed by QVAC serve’s ephemeral store. |
| packages/ai-sdk-provider/src/file-reference-model.ts | Resolves { qvac: <id> } file references via local /files/:id/content before model calls. |
| packages/ai-sdk-provider/src/audio-models.ts | Adds native transcription + speech model adapters targeting QVAC local audio endpoints. |
| packages/ai-sdk-provider/README.md | Updates docs for Node 22+, AI SDK 7/provider v4, and new files/audio capabilities. |
| packages/ai-sdk-provider/package.json | Bumps engine/peer deps and adds provider/provider-utils runtime deps. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -13,15 +18,58 @@ import type { | |||
| // pointed at a `qvac serve openai` endpoint the caller runs themselves. This is | |||
| // the v1 behaviour, kept byte-for-byte identical. | |||
| export function createExternalQvac(options: QvacExternalOptions = {}): QvacProvider { | |||
There was a problem hiding this comment.
Fixed in 3a5fe82b. The comment now describes external mode as composing the OpenAI-compatible language-model surface with QVAC native files, transcription, speech, and file-reference resolution.
|
Addressed all review feedback in
Verification passed locally:
Could a maintainer please apply |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 37 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
packages/ai-sdk-provider/src/provider.ts:20
- The comment says external mode is kept “byte-for-byte identical” to the v1 behavior, but this function now composes additional native capabilities (files/transcription/speech) and wraps language models to resolve local file references. This documentation is now misleading for readers trying to understand what differs from the older wrapper.
// External mode synchronously composes the OpenAI-compatible language-model
// surface with QVAC's native files, transcription, and speech capabilities.
// Language models also resolve QVAC file references through the caller-managed
// `qvac serve openai` endpoint.
| const headers: Record<string, string> = { ...DEFAULT_HEADERS, ...options.headers } | ||
| if (options.apiKey ?? DEFAULT_API_KEY) { | ||
| headers['Authorization'] = `Bearer ${options.apiKey ?? DEFAULT_API_KEY}` | ||
| } | ||
| const baseURL = options.baseURL ?? DEFAULT_BASE_URL |
| const prepare = async ( | ||
| call: LanguageModelV4CallOptions | ||
| ): Promise<LanguageModelV4CallOptions> => ({ | ||
| ...call, | ||
| prompt: await resolvePrompt(call.prompt, options, call.abortSignal) | ||
| }) |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 37 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
packages/ai-sdk-provider/src/provider.ts:25
createExternalQvaccan end up sending two Authorization headers if the caller passesheaders: { authorization: ... }(lowercase) and this code also addsAuthorization(capitalized). Because header names are case-insensitive, this can lead to an ambiguous/combined Authorization value and break authentication. Normalize by deleting any pre-existing authorization header (any casing) and then setting exactly oneauthorizationheader based on the resolved apiKey.
const headers: Record<string, string> = { ...DEFAULT_HEADERS, ...options.headers }
if (options.apiKey ?? DEFAULT_API_KEY) {
headers['Authorization'] = `Bearer ${options.apiKey ?? DEFAULT_API_KEY}`
}
|
hey @danielAsaboro, thanks for the AI SDK v7 work! one thing, the request-scoped could you kindly carve those bits out and keep this PR focused on the AI SDK 7 migration? also curious, were you relying on dynamic tools or did you just hit the bug along the way? happy to re-review once it's split |
|
Hi, @opaninakuffo Thanks for the context ...that makes sense. I was relying on dynamic tools in Leash rather than just encountering the code incidentally. We serve Qwen3 with tools: true / toolsMode:"dynamic" and expose a focused, per-turn subset of a larger tool registry. The regression surfaced on real tool-less paths - including raw completions, mesh-forward/mobile borrowing, and reasoning-only subagents. This is often where a model loaded in dynamic mode received no tools and failed I implemented the request-scoped fix before I was aware that #3373 and #3380 were removing the SDK and addon surfaces entirely. Given that direction, I agree it doesn’t make sense to deepen the feature here. I’ll carve all tools_compact / dynamic-tools changes and their tests out of #3370, update the title and description, and keep the PR focused on the AI SDK 7/provider-v4 migration. I’ll rerun the relevant provider and CLI checks and request another review afterward. On the Leash side, I’ll retain the existing 0.13.x workaround while needed and migrate to static tool placement when the removal ships. Thank you |
Summary
This migrates
@qvac/ai-sdk-providerfrom AI SDK 6/provider v3 to AI SDK 7/provider v4 and fixes the llama.cpp dynamic-tools regression where toolless requests failed withtools_compact requires non-empty tools.It also preserves uploaded MIME types in the CLI ephemeral-file store and exposes the same AI SDK 7 provider capabilities in external and managed modes.
API usage
The callable provider is shorthand for selecting a language model. The explicit language-model methods remain available, and other model types use their capability-specific methods:
These methods select model aliases served by QVAC; they do not load every capability through the callable language-model shorthand.
Managed mode exposes the same surface after starting the configured local models:
Changes
@qvac/ai-sdk-providerto AI SDK 7,@ai-sdk/openai-compatible3, provider v4 contracts, Node 22+, and ESM-only operation.qvac(modelId)as the language-model shorthand while retaininglanguageModel()andchatModel()and adding native files, transcription, and speech adapters./v1/files/:id/content.tools_compactfrom model-load configuration and computes it for each request. It istrueonly for a dynamic-tools request with a non-empty tool set and explicitlyfalseotherwise.Behavior / compatibility
@qvac/ai-sdk-providernow requires Node.js 22 or newer, AI SDK 7, and provider v4 packages.require()is not supported.qvac('alias'),qvac.languageModel('alias'), andqvac.chatModel('alias')select the same language-model capability. Existing explicitchatModel()calls do not need to change for model selection.embeddingModel()/textEmbeddingModel()andimageModel(); transcription and speech usetranscriptionModel()andspeechModel().tools_compact: false.Package requirement change:
BEFORE:
{ "engines": { "node": ">=20.0.0" }, "peerDependencies": { "ai": "^6.0", "@ai-sdk/openai-compatible": "^2.0" } }AFTER:
{ "engines": { "node": ">=22.0.0" }, "peerDependencies": { "ai": "^7.0", "@ai-sdk/openai-compatible": "^3.0" } }Testing
npm install; format; lint with zero warnings; typecheck; build; all unit tests (79 passed, 2 skipped); packed-package install/import smoke with AI SDK 7./v1/filesHTTP suite (7 passed), covering exact bytes, MIME header, private cache control, and missing files.toolsMode: "dynamic"completes a normal toolless request withtools_compact: false; unit coverage also verifies dynamic, static, disabled, and toolless requests plus independently evaluated batch prompts.