Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 70 additions & 4 deletions packages/ai-sdk-provider/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@

[Vercel AI SDK](https://ai-sdk.dev) provider for the [QVAC](https://qvac.tether.io) local AI runtime.

QVAC is an open-source, cross-platform ecosystem for **local-first, peer-to-peer AI** — LLMs, embeddings, transcription, translation, speech, OCR, and image generation, all running on the user's own hardware. This package is a thin, branded wrapper around [`@ai-sdk/openai-compatible`](https://www.npmjs.com/package/@ai-sdk/openai-compatible) that points at a running `qvac serve openai` HTTP server and re-exports QVAC's model metadata so callers can introspect typed model constants without an HTTP round-trip.
QVAC is an open-source, cross-platform ecosystem for **local-first, peer-to-peer AI** — LLMs, embeddings, transcription, translation, speech, OCR, and image generation, all running on the user's own hardware. This package implements the native provider contracts for language, embedding, image, file-reference, transcription, and speech operations against a local `qvac serve openai` runtime. It also re-exports QVAC's model metadata so callers can introspect typed model constants without an HTTP round-trip.

> **Status — `0.2.0`.** Two modes:
The provider is OpenAI-compatible by construction: the OpenAI-compatible transport is its fallback provider, while the SDK's native custom-provider composition adds QVAC-specific files, transcription, and speech capabilities. Existing OpenAI-shaped chat, completion, embedding, and image requests keep the same wire protocol.

> **Runtime modes:**
>
> - **External** (default): the package wraps a `qvac serve openai` HTTP endpoint that you run yourself.
> - **Managed** (`mode: 'managed'`): the provider synthesizes an ephemeral config from a model list, then spawns (or reuses) a shared `qvac serve` on a free port and keeps it alive for as long as anything is using it, reaping it automatically once everyone is done. See [Managed mode](#managed-mode) below. Requires the optional [`@qvac/cli`](https://www.npmjs.com/package/@qvac/cli) peer dependency.
Expand All @@ -20,7 +22,14 @@ bun add @qvac/ai-sdk-provider ai @ai-sdk/openai-compatible
# or: npm install @qvac/ai-sdk-provider ai @ai-sdk/openai-compatible
```

`ai` and `@ai-sdk/openai-compatible` are **peer dependencies** — install them alongside.
Runtime requirements:

- **Node.js 22 or newer.** Node 20 is not supported.
- **AI SDK 7** (`ai@^7`) and **OpenAI-compatible provider 3** (`@ai-sdk/openai-compatible@^3`).
- **Provider-v4 contracts.** Custom middleware and direct model integrations must use the AI SDK v4 provider interfaces exposed by these versions.
- **ESM-only imports.** Use `import` / dynamic `import()`; CommonJS `require()` is not supported.

`ai` and `@ai-sdk/openai-compatible` are **peer dependencies** — install those compatible major versions alongside the provider.

---

Expand Down Expand Up @@ -77,8 +86,65 @@ qvac.chatModel('qwen3-600m') // explicit chat model
qvac.completionModel('qwen3-600m') // legacy completion
qvac.textEmbeddingModel('embed-gemma') // text embeddings
qvac.imageModel('flux-schnell') // image generation
qvac.transcriptionModel('whisper') // speech to text
qvac.speechModel('tts') // text to speech
qvac.files() // local ephemeral file uploads
```

### Local file references

`uploadFile` stores bytes in the running QVAC process and returns a `qvac` provider reference. Passing that reference to a QVAC language model resolves the bytes through the same local runtime immediately before inference. The reference never becomes a cloud URL and expires with QVAC's ephemeral file store.

```ts
import { readFileSync } from 'node:fs'
import { generateText, uploadFile } from 'ai'

const { providerReference } = await uploadFile({
api: qvac,
data: readFileSync('./sensor.png'),
mediaType: 'image/png',
filename: 'sensor.png'
})

const { text } = await generateText({
model: qvac('vision'),
messages: [
{
role: 'user',
content: [
{ type: 'text', text: 'Read the sensor display.' },
{ type: 'file', mediaType: 'image/png', data: providerReference }
]
}
]
})
```

### Native speech and transcription

Speech operations use the provider's native model contracts while remaining entirely inside the local QVAC serve:

```ts
import { generateSpeech, transcribe } from 'ai'

const speech = await generateSpeech({
model: qvac.speechModel('tts'),
text: 'The sensor temperature is twenty four degrees.',
voice: 'alloy',
outputFormat: 'wav'
})

const transcript = await transcribe({
model: qvac.transcriptionModel('stt'),
audio: speech.audio.uint8Array,
providerOptions: {
qvac: { prompt: 'Sensor terminology' }
}
})
```

Per-request transcription `prompt` is supported. Language is selected when the QVAC model is loaded, and speech speed, instructions, and language are not consumed by the current local engines; the provider returns structured warnings when callers supply those options.

---

## Managed mode
Expand Down Expand Up @@ -187,7 +253,7 @@ Managed mode runs `qvac serve` as a **shared, self-cleaning daemon** so that ope

- **Startup is gated on model preload.** `qvac serve` does not open its port until every preloaded model is ready, and a cold P2P download can take minutes — hence the generous default `serveStartTimeout`. Raise it for large models.
- **External mode pays nothing.** The managed subsystem (and its `node:child_process` / `@qvac/cli` resolution) is dynamically imported only when `mode: 'managed'` is set.
- **Node 20+ and Bun.** The managed subsystem uses only portable `node:` APIs — no Bun-specific calls.
- **Node 22+ and Bun.** The managed subsystem uses only portable `node:` APIs — no Bun-specific calls.
- **Typed errors.** Managed setup throws structured errors you can `instanceof`-check: `UnknownManagedModelError`, `DuplicateManagedModelError`, `MultipleDefaultManagedModelsError`, `CliNotFoundError`, `ServeStartTimeoutError`, `ServeSpawnFailedError`, `ServeExitedError`, and `PortAllocationFailedError` (all extending `QvacManagedModeError`, with a `.code` from `QvacManagedErrorCode`). They're exported from the package root.

---
Expand Down
14 changes: 9 additions & 5 deletions packages/ai-sdk-provider/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"NOTICE"
],
"engines": {
"node": ">=20.0.0"
"node": ">=22.0.0"
},
"keywords": [
"ai",
Expand Down Expand Up @@ -72,20 +72,24 @@
"test:integration": "tsx --test test/managed-integration.test.ts"
},
"peerDependencies": {
"@ai-sdk/openai-compatible": "^2.0",
"@ai-sdk/openai-compatible": "^3.0",
"@qvac/cli": "^0.6.0 || ^0.7.0 || ^0.8.0",
"ai": "^6.0"
"ai": "^7.0"
},
"peerDependenciesMeta": {
"@qvac/cli": {
"optional": true
}
},
"dependencies": {
"@ai-sdk/provider": "^4.0.0",
"@ai-sdk/provider-utils": "^5.0.0"
},
"devDependencies": {
"@ai-sdk/openai-compatible": "^2.0",
"@ai-sdk/openai-compatible": "^3.0",
"@qvac/registry-client": "^0.5.0",
"@types/node": "^24.2.1",
"ai": "^6.0",
"ai": "^7.0",
"lunte": "^1.8.2",
"prettier": "^3.9.4",
"prettier-config-holepunch": "^2.0.0",
Expand Down
208 changes: 208 additions & 0 deletions packages/ai-sdk-provider/src/audio-models.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
import type { SharedV4Warning, SpeechModelV4, TranscriptionModelV4 } from '@ai-sdk/provider'
import {
createBinaryResponseHandler,
createStatusCodeErrorResponseHandler,
postFormDataToApi,
postJsonToApi,
type ResponseHandler
} from '@ai-sdk/provider-utils'

interface QvacAudioOptions {
readonly baseURL: string
readonly headers: Record<string, string>
readonly fetch?: typeof fetch
}

function endpoint(options: QvacAudioOptions, path: string): string {
return `${options.baseURL.replace(/\/+$/, '')}${path}`
}

function requestHeaders(
configured: Record<string, string>,
perCall: Record<string, string | undefined> | undefined
): Record<string, string> {
const headers = { ...configured }
for (const [name, value] of Object.entries(perCall ?? {})) {
if (value !== undefined) headers[name] = value
}
return headers
}

function withoutContentType(headers: Record<string, string>): Record<string, string> {
return Object.fromEntries(
Object.entries(headers).filter(([name]) => name.toLowerCase() !== 'content-type')
)
}

function bytesFor(data: Uint8Array | string): Uint8Array {
return typeof data === 'string' ? Uint8Array.from(Buffer.from(data, 'base64')) : data
}

interface JsonResponse<T> {
readonly body: T
readonly headers: Record<string, string>
}

function jsonResponseHandler<T>(): ResponseHandler<JsonResponse<T>> {
return async ({ response }) => {
const body = (await response.json()) as T
const headers = Object.fromEntries(response.headers.entries())
return {
value: { body, headers },
rawValue: body,
responseHeaders: headers
}
}
}

const binaryResponseHandler = createBinaryResponseHandler()
const binaryWithHeadersHandler: ResponseHandler<{
audio: Uint8Array
headers: Record<string, string>
}> = async (options) => {
const result = await binaryResponseHandler(options)
return {
...result,
value: { audio: result.value, headers: result.responseHeaders ?? {} }
}
}

const statusCodeErrorResponseHandler = createStatusCodeErrorResponseHandler()

export function createQvacTranscriptionModel(
modelId: string,
options: QvacAudioOptions
): TranscriptionModelV4 {
return {
specificationVersion: 'v4',
provider: 'qvac.transcription',
modelId,
async doGenerate({ audio, mediaType, providerOptions, abortSignal, headers }) {
const qvacOptions = providerOptions?.['qvac'] ?? {}
const form = new FormData()
const bytes = bytesFor(audio)
form.append(
'file',
new Blob([bytes.slice().buffer as ArrayBuffer], { type: mediaType }),
'audio-input'
)
form.append('model', modelId)
form.append('response_format', 'json')

const prompt = qvacOptions['prompt']
const language = qvacOptions['language']
const temperature = qvacOptions['temperature']
if (typeof prompt === 'string') form.append('prompt', prompt)
if (typeof language === 'string') form.append('language', language)
if (typeof temperature === 'number') form.append('temperature', String(temperature))

const warnings: SharedV4Warning[] = []
if (language !== undefined) {
warnings.push({
type: 'unsupported',
feature: 'providerOptions.qvac.language',
details: 'QVAC configures transcription language when the model is loaded.'
})
}
if (temperature !== undefined) {
warnings.push({
type: 'unsupported',
feature: 'providerOptions.qvac.temperature',
details: 'The local transcription runtime does not use per-request temperature.'
})
}

const timestamp = new Date()
const transcriptionResponse = await postFormDataToApi({
url: endpoint(options, '/audio/transcriptions'),
headers: withoutContentType(requestHeaders(options.headers, headers)),
formData: form,
failedResponseHandler: statusCodeErrorResponseHandler,
successfulResponseHandler: jsonResponseHandler<{ text?: unknown }>(),
...(abortSignal !== undefined && { abortSignal }),
...(options.fetch !== undefined && { fetch: options.fetch })
})
const { body, headers: responseHeaders } = transcriptionResponse.value
if (typeof body.text !== 'string') throw new Error('QVAC transcription returned no text')

return {
text: body.text,
segments: [],
language: undefined,
durationInSeconds: undefined,
warnings,
response: {
timestamp,
modelId,
headers: responseHeaders,
body
},
providerMetadata: { qvac: { local: true } }
}
}
}
}

export function createQvacSpeechModel(modelId: string, options: QvacAudioOptions): SpeechModelV4 {
return {
specificationVersion: 'v4',
provider: 'qvac.speech',
modelId,
async doGenerate({
text,
voice,
outputFormat,
instructions,
speed,
language,
abortSignal,
headers
}) {
const warnings: SharedV4Warning[] = []
for (const [feature, value] of Object.entries({ instructions, speed, language })) {
if (value !== undefined) {
warnings.push({
type: 'unsupported',
feature,
details: `QVAC local speech synthesis does not use per-request ${feature}.`
})
}
}

const body = {
model: modelId,
input: text,
...(voice !== undefined && { voice }),
...(outputFormat !== undefined && { response_format: outputFormat })
}
const timestamp = new Date()
const speechResponse = await postJsonToApi({
url: endpoint(options, '/audio/speech'),
headers: withoutContentType(requestHeaders(options.headers, headers)),
body,
failedResponseHandler: statusCodeErrorResponseHandler,
successfulResponseHandler: binaryWithHeadersHandler,
...(abortSignal !== undefined && { abortSignal }),
...(options.fetch !== undefined && { fetch: options.fetch })
})
const { audio, headers: responseHeaders } = speechResponse.value

return {
audio,
warnings,
request: { body },
response: {
timestamp,
modelId,
headers: responseHeaders
},
providerMetadata: {
qvac: {
local: true,
mediaType: responseHeaders['content-type'] ?? 'application/octet-stream'
}
}
}
}
}
}
Loading
Loading