diff --git a/.changeset/modern-kings-smoke.md b/.changeset/modern-kings-smoke.md new file mode 100644 index 000000000000..2e64edd76ed3 --- /dev/null +++ b/.changeset/modern-kings-smoke.md @@ -0,0 +1,5 @@ +--- +'@ai-sdk/lmnt': patch +--- + +feat(providers/lmnt): add speech diff --git a/content/docs/02-foundations/02-providers-and-models.mdx b/content/docs/02-foundations/02-providers-and-models.mdx index 871cfec8fd7c..93ff02c78c88 100644 --- a/content/docs/02-foundations/02-providers-and-models.mdx +++ b/content/docs/02-foundations/02-providers-and-models.mdx @@ -41,6 +41,7 @@ The AI SDK comes with a wide range of providers that you can use to interact wit - [Groq Provider](/providers/ai-sdk-providers/groq) (`@ai-sdk/groq`) - [Perplexity Provider](/providers/ai-sdk-providers/perplexity) (`@ai-sdk/perplexity`) - [ElevenLabs Provider](/providers/ai-sdk-providers/elevenlabs) (`@ai-sdk/elevenlabs`) +- [LMNT Provider](/providers/ai-sdk-providers/lmnt) (`@ai-sdk/lmnt`) - [Hume Provider](/providers/ai-sdk-providers/hume) (`@ai-sdk/hume`) - [Rev.ai Provider](/providers/ai-sdk-providers/revai) (`@ai-sdk/revai`) diff --git a/content/docs/03-ai-sdk-core/37-speech.mdx b/content/docs/03-ai-sdk-core/37-speech.mdx index 77c3f4ef0994..edbbdd71e401 100644 --- a/content/docs/03-ai-sdk-core/37-speech.mdx +++ b/content/docs/03-ai-sdk-core/37-speech.mdx @@ -145,6 +145,8 @@ try { | [OpenAI](/providers/ai-sdk-providers/openai#speech-models) | `tts-1` | | [OpenAI](/providers/ai-sdk-providers/openai#speech-models) | `tts-1-hd` | | [OpenAI](/providers/ai-sdk-providers/openai#speech-models) | `gpt-4o-mini-tts` | +| [LMNT](/providers/ai-sdk-providers/lmnt#speech-models) | `aurora` | +| [LMNT](/providers/ai-sdk-providers/lmnt#speech-models) | `blizzard` | | [Hume](/providers/ai-sdk-providers/hume#speech-models) | `default` | Above are a small subset of the speech models supported by the AI SDK providers. For more, see the respective provider documentation. diff --git a/content/providers/01-ai-sdk-providers/140-lmnt.mdx b/content/providers/01-ai-sdk-providers/140-lmnt.mdx new file mode 100644 index 000000000000..435a35ab9b1f --- /dev/null +++ b/content/providers/01-ai-sdk-providers/140-lmnt.mdx @@ -0,0 +1,136 @@ +--- +title: LMNT +description: Learn how to use the LMNT provider for the AI SDK. +--- + +# LMNT Provider + +The [LMNT](https://lmnt.com/) provider contains language model support for the LMNT transcription API. + +## Setup + +The LMNT provider is available in the `@ai-sdk/lmnt` module. You can install it with + + + + + + + + + + + + + +## Provider Instance + +You can import the default provider instance `lmnt` from `@ai-sdk/lmnt`: + +```ts +import { lmnt } from '@ai-sdk/lmnt'; +``` + +If you need a customized setup, you can import `createLMNT` from `@ai-sdk/lmnt` and create a provider instance with your settings: + +```ts +import { createLMNT } from '@ai-sdk/lmnt'; + +const lmnt = createLMNT({ + // custom settings, e.g. + fetch: customFetch, +}); +``` + +You can use the following optional settings to customize the LMNT provider instance: + +- **apiKey** _string_ + + API key that is being sent using the `Authorization` header. + It defaults to the `LMNT_API_KEY` environment variable. + +- **headers** _Record<string,string>_ + + Custom headers to include in the requests. + +- **fetch** _(input: RequestInfo, init?: RequestInit) => Promise<Response>_ + + Custom [fetch](https://developer.mozilla.org/en-US/docs/Web/API/fetch) implementation. + Defaults to the global `fetch` function. + You can use it as a middleware to intercept requests, + or to provide a custom fetch implementation for e.g. testing. + +## Speech Models + +You can create models that call the [LMNT speech API](https://docs.lmnt.com/api-reference/speech/synthesize-speech-bytes) +using the `.speech()` factory method. + +The first argument is the model id e.g. `aurora`. + +```ts +const model = lmnt.speech('aurora'); +``` + +You can also pass additional provider-specific options using the `providerOptions` argument. For example, supplying a voice to use for the generated audio. + +```ts highlight="6" +import { experimental_generateSpeech as generateSpeech } from 'ai'; +import { lmnt } from '@ai-sdk/lmnt'; + +const result = await generateSpeech({ + model: lmnt.speech('aurora'), + text: 'Hello, world!', + providerOptions: { lmnt: { language: 'en' } }, +}); +``` + +### Provider Options + +The LMNT provider accepts the following options: + +- **model** _'aurora' | 'blizzard'_ + + The LMNT model to use. Defaults to `'aurora'`. + +- **language** _'auto' | 'en' | 'es' | 'pt' | 'fr' | 'de' | 'zh' | 'ko' | 'hi' | 'ja' | 'ru' | 'it' | 'tr'_ + + The language to use for speech synthesis. Defaults to `'auto'`. + +- **format** _'aac' | 'mp3' | 'mulaw' | 'raw' | 'wav'_ + + The audio format to return. Defaults to `'mp3'`. + +- **sampleRate** _number_ + + The sample rate of the audio in Hz. Defaults to `24000`. + +- **speed** _number_ + + The speed of the speech. Must be between 0.25 and 2. Defaults to `1`. + +- **seed** _number_ + + An optional seed for deterministic generation. + +- **conversational** _boolean_ + + Whether to use a conversational style. Defaults to `false`. + +- **length** _number_ + + Maximum length of the audio in seconds. Maximum value is 300. + +- **topP** _number_ + + Top-p sampling parameter. Must be between 0 and 1. Defaults to `1`. + +- **temperature** _number_ + + Temperature parameter for sampling. Must be at least 0. Defaults to `1`. + +### Model Capabilities + +| Model | Instructions | +| ---------- | ------------------- | +| `aurora` | | +| `blizzard` | | diff --git a/examples/ai-core/package.json b/examples/ai-core/package.json index 111f9a45d708..3b21ff3ff3f0 100644 --- a/examples/ai-core/package.json +++ b/examples/ai-core/package.json @@ -16,6 +16,7 @@ "@ai-sdk/google": "2.0.0-canary.9", "@ai-sdk/google-vertex": "3.0.0-canary.9", "@ai-sdk/groq": "2.0.0-canary.9", + "@ai-sdk/lmnt": "1.0.0-canary.0", "@ai-sdk/luma": "1.0.0-canary.8", "@ai-sdk/hume": "1.0.0-canary.0", "@ai-sdk/mistral": "2.0.0-canary.8", diff --git a/examples/ai-core/src/generate-speech/lmnt-speed.ts b/examples/ai-core/src/generate-speech/lmnt-speed.ts new file mode 100644 index 000000000000..7645e4345506 --- /dev/null +++ b/examples/ai-core/src/generate-speech/lmnt-speed.ts @@ -0,0 +1,21 @@ +import { lmnt } from '@ai-sdk/lmnt'; +import { experimental_generateSpeech as generateSpeech } from 'ai'; +import 'dotenv/config'; +import { saveAudioFile } from '../lib/save-audio'; + +async function main() { + const result = await generateSpeech({ + model: lmnt.speech('aurora'), + text: 'Hello from the AI SDK!', + speed: 1.5, + }); + + console.log('Audio:', result.audio); + console.log('Warnings:', result.warnings); + console.log('Responses:', result.responses); + console.log('Provider Metadata:', result.providerMetadata); + + await saveAudioFile(result.audio); +} + +main().catch(console.error); diff --git a/examples/ai-core/src/generate-speech/lmnt-voice.ts b/examples/ai-core/src/generate-speech/lmnt-voice.ts new file mode 100644 index 000000000000..848582877a6d --- /dev/null +++ b/examples/ai-core/src/generate-speech/lmnt-voice.ts @@ -0,0 +1,21 @@ +import { lmnt } from '@ai-sdk/lmnt'; +import { experimental_generateSpeech as generateSpeech } from 'ai'; +import 'dotenv/config'; +import { saveAudioFile } from '../lib/save-audio'; + +async function main() { + const result = await generateSpeech({ + model: lmnt.speech('aurora'), + text: 'Hello from the AI SDK!', + voice: 'nova', + }); + + console.log('Audio:', result.audio); + console.log('Warnings:', result.warnings); + console.log('Responses:', result.responses); + console.log('Provider Metadata:', result.providerMetadata); + + await saveAudioFile(result.audio); +} + +main().catch(console.error); diff --git a/examples/ai-core/src/generate-speech/lmnt.ts b/examples/ai-core/src/generate-speech/lmnt.ts new file mode 100644 index 000000000000..ab02da8e2cad --- /dev/null +++ b/examples/ai-core/src/generate-speech/lmnt.ts @@ -0,0 +1,20 @@ +import { lmnt } from '@ai-sdk/lmnt'; +import { experimental_generateSpeech as generateSpeech } from 'ai'; +import 'dotenv/config'; +import { saveAudioFile } from '../lib/save-audio'; + +async function main() { + const result = await generateSpeech({ + model: lmnt.speech('aurora'), + text: 'Hello from the AI SDK!', + }); + + console.log('Audio:', result.audio); + console.log('Warnings:', result.warnings); + console.log('Responses:', result.responses); + console.log('Provider Metadata:', result.providerMetadata); + + await saveAudioFile(result.audio); +} + +main().catch(console.error); diff --git a/examples/ai-core/tsconfig.json b/examples/ai-core/tsconfig.json index e97a028449ab..391312dc301e 100644 --- a/examples/ai-core/tsconfig.json +++ b/examples/ai-core/tsconfig.json @@ -49,6 +49,9 @@ { "path": "../../packages/deepinfra" }, + { + "path": "../../packages/lmnt" + }, { "path": "../../packages/deepseek" }, diff --git a/packages/lmnt/README.md b/packages/lmnt/README.md new file mode 100644 index 000000000000..1fccaa19d5b9 --- /dev/null +++ b/packages/lmnt/README.md @@ -0,0 +1,36 @@ +# AI SDK - LMNT Provider + +The **[LMNT provider](https://sdk.vercel.ai/providers/ai-sdk-providers/lmnt)** for the [AI SDK](https://sdk.vercel.ai/docs) +contains language model support for the LMNT API. + +## Setup + +The LMNT provider is available in the `@ai-sdk/lmnt` module. You can install it with + +```bash +npm i @ai-sdk/lmnt +``` + +## Provider Instance + +You can import the default provider instance `lmnt` from `@ai-sdk/lmnt`: + +```ts +import { lmnt } from '@ai-sdk/lmnt'; +``` + +## Example + +```ts +import { lmnt } from '@ai-sdk/lmnt'; +import { experimental_generateSpeech as generateSpeech } from 'ai'; + +const result = await generateSpeech({ + model: lmnt.speech('aurora'), + text: 'Hello, world!', +}); +``` + +## Documentation + +Please check out the **[LMNT provider documentation](https://sdk.vercel.ai/providers/ai-sdk-providers/lmnt)** for more information. diff --git a/packages/lmnt/package.json b/packages/lmnt/package.json new file mode 100644 index 000000000000..121cdbc6c052 --- /dev/null +++ b/packages/lmnt/package.json @@ -0,0 +1,64 @@ +{ + "name": "@ai-sdk/lmnt", + "version": "1.0.0-canary.0", + "license": "Apache-2.0", + "sideEffects": false, + "main": "./dist/index.js", + "module": "./dist/index.mjs", + "types": "./dist/index.d.ts", + "files": [ + "dist/**/*", + "CHANGELOG.md" + ], + "scripts": { + "build": "tsup --tsconfig tsconfig.build.json", + "build:watch": "tsup --tsconfig tsconfig.build.json --watch", + "clean": "rm -rf dist", + "lint": "eslint \"./**/*.ts*\"", + "type-check": "tsc --noEmit", + "prettier-check": "prettier --check \"./**/*.ts*\"", + "test": "pnpm test:node && pnpm test:edge", + "test:edge": "vitest --config vitest.edge.config.js --run", + "test:node": "vitest --config vitest.node.config.js --run", + "test:node:watch": "vitest --config vitest.node.config.js --watch" + }, + "exports": { + "./package.json": "./package.json", + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.mjs", + "require": "./dist/index.js" + } + }, + "dependencies": { + "@ai-sdk/provider": "2.0.0-canary.7", + "@ai-sdk/provider-utils": "3.0.0-canary.8" + }, + "devDependencies": { + "@types/node": "20.17.24", + "@vercel/ai-tsconfig": "workspace:*", + "tsup": "^8", + "typescript": "5.6.3", + "zod": "3.23.8" + }, + "peerDependencies": { + "zod": "^3.0.0" + }, + "engines": { + "node": ">=18" + }, + "publishConfig": { + "access": "public" + }, + "homepage": "https://sdk.vercel.ai/docs", + "repository": { + "type": "git", + "url": "git+https://github.com/vercel/ai.git" + }, + "bugs": { + "url": "https://github.com/vercel/ai/issues" + }, + "keywords": [ + "ai" + ] +} diff --git a/packages/lmnt/src/index.ts b/packages/lmnt/src/index.ts new file mode 100644 index 000000000000..879f4bf9eb70 --- /dev/null +++ b/packages/lmnt/src/index.ts @@ -0,0 +1,2 @@ +export { createLMNT, lmnt } from './lmnt-provider'; +export type { LMNTProvider, LMNTProviderSettings } from './lmnt-provider'; diff --git a/packages/lmnt/src/lmnt-api-types.ts b/packages/lmnt/src/lmnt-api-types.ts new file mode 100644 index 000000000000..aeae35e1c3c1 --- /dev/null +++ b/packages/lmnt/src/lmnt-api-types.ts @@ -0,0 +1,39 @@ +export type LMNTSpeechAPITypes = { + /** The voice id of the voice to use; voice ids can be retrieved by calls to List voices or Voice info. */ + voice: string; + /** The text to synthesize; max 5000 characters per request (including spaces) */ + text: string; + /** The model to use for synthesis. One of aurora (default) or blizzard. */ + model?: 'aurora' | 'blizzard'; + /** The desired language. Two letter ISO 639-1 code. Does not work with professional clones. Not all languages work with all models. Defaults to auto language detection. */ + language?: + | 'auto' + | 'en' + | 'es' + | 'pt' + | 'fr' + | 'de' + | 'zh' + | 'ko' + | 'hi' + | 'ja' + | 'ru' + | 'it' + | 'tr'; + /** The file format of the audio output */ + format?: 'aac' | 'mp3' | 'mulaw' | 'raw' | 'wav'; + /** The desired output sample rate in Hz */ + sample_rate?: 8000 | 16000 | 24000; + /** The talking speed of the generated speech, a floating point value between 0.25 (slow) and 2.0 (fast). */ + speed?: number; + /** Seed used to specify a different take; defaults to random */ + seed?: number; + /** Set this to true to generate conversational-style speech rather than reading-style speech. Does not work with the blizzard model. */ + conversational?: boolean; + /** Produce speech of this length in seconds; maximum 300.0 (5 minutes). Does not work with the blizzard model. */ + length?: number; + /** Controls the stability of the generated speech. A lower value (like 0.3) produces more consistent, reliable speech. A higher value (like 0.9) gives more flexibility in how words are spoken, but might occasionally produce unusual intonations or speech patterns. */ + top_p?: number; + /** Influences how expressive and emotionally varied the speech becomes. Lower values (like 0.3) create more neutral, consistent speaking styles. Higher values (like 1.0) allow for more dynamic emotional range and speaking styles. */ + temperature?: number; +}; diff --git a/packages/lmnt/src/lmnt-config.ts b/packages/lmnt/src/lmnt-config.ts new file mode 100644 index 000000000000..a852d5b8c8da --- /dev/null +++ b/packages/lmnt/src/lmnt-config.ts @@ -0,0 +1,9 @@ +import { FetchFunction } from '@ai-sdk/provider-utils'; + +export type LMNTConfig = { + provider: string; + url: (options: { modelId: string; path: string }) => string; + headers: () => Record; + fetch?: FetchFunction; + generateId?: () => string; +}; diff --git a/packages/lmnt/src/lmnt-error.test.ts b/packages/lmnt/src/lmnt-error.test.ts new file mode 100644 index 000000000000..abb9f46d6072 --- /dev/null +++ b/packages/lmnt/src/lmnt-error.test.ts @@ -0,0 +1,33 @@ +import { safeParseJSON } from '@ai-sdk/provider-utils'; +import { lmntErrorDataSchema } from './lmnt-error'; + +describe('lmntErrorDataSchema', () => { + it('should parse LMNT resource exhausted error', () => { + const error = ` +{"error":{"message":"{\\n \\"error\\": {\\n \\"code\\": 429,\\n \\"message\\": \\"Resource has been exhausted (e.g. check quota).\\",\\n \\"status\\": \\"RESOURCE_EXHAUSTED\\"\\n }\\n}\\n","code":429}} +`; + + const result = safeParseJSON({ + text: error, + schema: lmntErrorDataSchema, + }); + + expect(result).toStrictEqual({ + success: true, + value: { + error: { + message: + '{\n "error": {\n "code": 429,\n "message": "Resource has been exhausted (e.g. check quota).",\n "status": "RESOURCE_EXHAUSTED"\n }\n}\n', + code: 429, + }, + }, + rawValue: { + error: { + message: + '{\n "error": {\n "code": 429,\n "message": "Resource has been exhausted (e.g. check quota).",\n "status": "RESOURCE_EXHAUSTED"\n }\n}\n', + code: 429, + }, + }, + }); + }); +}); diff --git a/packages/lmnt/src/lmnt-error.ts b/packages/lmnt/src/lmnt-error.ts new file mode 100644 index 000000000000..40540a8bdd90 --- /dev/null +++ b/packages/lmnt/src/lmnt-error.ts @@ -0,0 +1,16 @@ +import { z } from 'zod'; +import { createJsonErrorResponseHandler } from '@ai-sdk/provider-utils'; + +export const lmntErrorDataSchema = z.object({ + error: z.object({ + message: z.string(), + code: z.number(), + }), +}); + +export type LMNTErrorData = z.infer; + +export const lmntFailedResponseHandler = createJsonErrorResponseHandler({ + errorSchema: lmntErrorDataSchema, + errorToMessage: data => data.error.message, +}); diff --git a/packages/lmnt/src/lmnt-provider.ts b/packages/lmnt/src/lmnt-provider.ts new file mode 100644 index 000000000000..ec8e2ef1d6d1 --- /dev/null +++ b/packages/lmnt/src/lmnt-provider.ts @@ -0,0 +1,74 @@ +import { SpeechModelV1, ProviderV1 } from '@ai-sdk/provider'; +import { FetchFunction, loadApiKey } from '@ai-sdk/provider-utils'; +import { LMNTSpeechModel } from './lmnt-speech-model'; +import { LMNTSpeechModelId } from './lmnt-speech-settings'; + +export interface LMNTProvider extends Pick { + ( + modelId: 'aurora', + settings?: {}, + ): { + speech: LMNTSpeechModel; + }; + + /** +Creates a model for speech synthesis. + */ + speech(modelId: LMNTSpeechModelId): SpeechModelV1; +} + +export interface LMNTProviderSettings { + /** +API key for authenticating requests. + */ + apiKey?: string; + + /** +Custom headers to include in the requests. + */ + headers?: Record; + + /** +Custom fetch implementation. You can use it as a middleware to intercept requests, +or to provide a custom fetch implementation for e.g. testing. + */ + fetch?: FetchFunction; +} + +/** +Create an LMNT provider instance. + */ +export function createLMNT(options: LMNTProviderSettings = {}): LMNTProvider { + const getHeaders = () => ({ + 'x-api-key': loadApiKey({ + apiKey: options.apiKey, + environmentVariableName: 'LMNT_API_KEY', + description: 'LMNT', + }), + ...options.headers, + }); + + const createSpeechModel = (modelId: LMNTSpeechModelId) => + new LMNTSpeechModel(modelId, { + provider: `lmnt.speech`, + url: ({ path }) => `https://api.lmnt.com${path}`, + headers: getHeaders, + fetch: options.fetch, + }); + + const provider = function (modelId: LMNTSpeechModelId) { + return { + speech: createSpeechModel(modelId), + }; + }; + + provider.speech = createSpeechModel; + provider.speechModel = createSpeechModel; + + return provider as LMNTProvider; +} + +/** +Default LMNT provider instance. + */ +export const lmnt = createLMNT(); diff --git a/packages/lmnt/src/lmnt-speech-model.test.ts b/packages/lmnt/src/lmnt-speech-model.test.ts new file mode 100644 index 000000000000..54e51bf4f485 --- /dev/null +++ b/packages/lmnt/src/lmnt-speech-model.test.ts @@ -0,0 +1,189 @@ +import { createTestServer } from '@ai-sdk/provider-utils/test'; +import { LMNTSpeechModel } from './lmnt-speech-model'; +import { createLMNT } from './lmnt-provider'; + +const provider = createLMNT({ apiKey: 'test-api-key' }); +const model = provider.speech('aurora'); + +const server = createTestServer({ + 'https://api.lmnt.com/v1/ai/speech/bytes': {}, +}); + +describe('doGenerate', () => { + function prepareAudioResponse({ + headers, + format = 'mp3', + }: { + headers?: Record; + format?: 'aac' | 'mp3' | 'mulaw' | 'raw' | 'wav'; + } = {}) { + const audioBuffer = new Uint8Array(100); // Mock audio data + server.urls['https://api.lmnt.com/v1/ai/speech/bytes'].response = { + type: 'binary', + headers: { + 'content-type': `audio/${format}`, + ...headers, + }, + body: Buffer.from(audioBuffer), + }; + return audioBuffer; + } + + it('should pass the model and text', async () => { + prepareAudioResponse(); + + await model.doGenerate({ + text: 'Hello from the AI SDK!', + }); + + expect(await server.calls[0].requestBody).toMatchObject({ + model: 'aurora', + text: 'Hello from the AI SDK!', + }); + }); + + it('should pass headers', async () => { + prepareAudioResponse(); + + const provider = createLMNT({ + apiKey: 'test-api-key', + headers: { + 'Custom-Provider-Header': 'provider-header-value', + }, + }); + + await provider.speech('aurora').doGenerate({ + text: 'Hello from the AI SDK!', + headers: { + 'Custom-Request-Header': 'request-header-value', + }, + }); + + expect(server.calls[0].requestHeaders).toMatchObject({ + 'x-api-key': 'test-api-key', + 'content-type': 'application/json', + 'custom-provider-header': 'provider-header-value', + 'custom-request-header': 'request-header-value', + }); + }); + + it('should pass options', async () => { + prepareAudioResponse(); + + await model.doGenerate({ + text: 'Hello from the AI SDK!', + voice: 'nova', + outputFormat: 'mp3', + speed: 1.5, + }); + + expect(await server.calls[0].requestBody).toMatchObject({ + model: 'aurora', + text: 'Hello from the AI SDK!', + voice: 'nova', + speed: 1.5, + response_format: 'mp3', + }); + }); + + it('should return audio data with correct content type', async () => { + const audio = new Uint8Array(100); // Mock audio data + prepareAudioResponse({ + format: 'mp3', + headers: { + 'x-request-id': 'test-request-id', + 'x-ratelimit-remaining': '123', + }, + }); + + const result = await model.doGenerate({ + text: 'Hello from the AI SDK!', + outputFormat: 'mp3', + }); + + expect(result.audio).toStrictEqual(audio); + }); + + it('should include response data with timestamp, modelId and headers', async () => { + prepareAudioResponse({ + headers: { + 'x-request-id': 'test-request-id', + 'x-ratelimit-remaining': '123', + }, + }); + + const testDate = new Date(0); + const customModel = new LMNTSpeechModel('aurora', { + provider: 'test-provider', + url: () => 'https://api.lmnt.com/v1/ai/speech/bytes', + headers: () => ({}), + _internal: { + currentDate: () => testDate, + }, + }); + + const result = await customModel.doGenerate({ + text: 'Hello from the AI SDK!', + }); + + expect(result.response).toMatchObject({ + timestamp: testDate, + modelId: 'aurora', + headers: { + 'content-type': 'audio/mp3', + 'x-request-id': 'test-request-id', + 'x-ratelimit-remaining': '123', + }, + }); + }); + + it('should use real date when no custom date provider is specified', async () => { + prepareAudioResponse(); + + const testDate = new Date(0); + const customModel = new LMNTSpeechModel('aurora', { + provider: 'test-provider', + url: () => 'https://api.lmnt.com/v1/ai/speech/bytes', + headers: () => ({}), + _internal: { + currentDate: () => testDate, + }, + }); + + const result = await customModel.doGenerate({ + text: 'Hello from the AI SDK!', + }); + + expect(result.response.timestamp.getTime()).toEqual(testDate.getTime()); + expect(result.response.modelId).toBe('aurora'); + }); + + it('should handle different audio formats', async () => { + const formats = ['aac', 'mp3', 'mulaw', 'raw', 'wav'] as const; + + for (const format of formats) { + const audio = prepareAudioResponse({ format }); + + const result = await model.doGenerate({ + text: 'Hello from the AI SDK!', + providerOptions: { + lmnt: { + format, + }, + }, + }); + + expect(result.audio).toStrictEqual(audio); + } + }); + + it('should include warnings if any are generated', async () => { + prepareAudioResponse(); + + const result = await model.doGenerate({ + text: 'Hello from the AI SDK!', + }); + + expect(result.warnings).toEqual([]); + }); +}); diff --git a/packages/lmnt/src/lmnt-speech-model.ts b/packages/lmnt/src/lmnt-speech-model.ts new file mode 100644 index 000000000000..e0fff2e8f7aa --- /dev/null +++ b/packages/lmnt/src/lmnt-speech-model.ts @@ -0,0 +1,210 @@ +import { SpeechModelV1, SpeechModelV1CallWarning } from '@ai-sdk/provider'; +import { + combineHeaders, + createBinaryResponseHandler, + parseProviderOptions, + postJsonToApi, +} from '@ai-sdk/provider-utils'; +import { z } from 'zod'; +import { LMNTConfig } from './lmnt-config'; +import { lmntFailedResponseHandler } from './lmnt-error'; +import { LMNTSpeechModelId } from './lmnt-speech-settings'; +import { LMNTSpeechAPITypes } from './lmnt-api-types'; + +// https://docs.lmnt.com/api-reference/speech/synthesize-speech-bytes +const lmntSpeechCallOptionsSchema = z.object({ + /** + * The model to use for speech synthesis e.g. 'aurora' or 'blizzard'. + * @default 'aurora' + */ + model: z + .union([z.enum(['aurora', 'blizzard']), z.string()]) + .nullish() + .default('aurora'), + + /** + * The language of the input text. + * @default 'auto' + */ + language: z + .union([z.enum(['auto', 'en']), z.string()]) + .nullish() + .default('auto'), + + /** + * The audio format of the output. + * @default 'mp3' + */ + format: z + .enum(['aac', 'mp3', 'mulaw', 'raw', 'wav']) + .nullish() + .default('mp3'), + + /** + * The sample rate of the output audio in Hz. + * @default 24000 + */ + sampleRate: z + .union([z.literal(8000), z.literal(16000), z.literal(24000)]) + .nullish() + .default(24000), + + /** + * The speed of the speech. Range: 0.25 to 2. + * @default 1 + */ + speed: z.number().min(0.25).max(2).nullish().default(1), + + /** + * A seed value for deterministic generation. + */ + seed: z.number().int().nullish(), + + /** + * Whether to use a conversational style. + * @default false + */ + conversational: z.boolean().nullish().default(false), + + /** + * Maximum length of the output in seconds (up to 300). + */ + length: z.number().max(300).nullish(), + + /** + * Top-p sampling parameter. Range: 0 to 1. + * @default 1 + */ + topP: z.number().min(0).max(1).nullish().default(1), + + /** + * Temperature for sampling. Higher values increase randomness. + * @default 1 + */ + temperature: z.number().min(0).nullish().default(1), +}); + +export type LMNTSpeechCallOptions = z.infer; + +interface LMNTSpeechModelConfig extends LMNTConfig { + _internal?: { + currentDate?: () => Date; + }; +} + +export class LMNTSpeechModel implements SpeechModelV1 { + readonly specificationVersion = 'v1'; + + get provider(): string { + return this.config.provider; + } + + constructor( + readonly modelId: LMNTSpeechModelId, + private readonly config: LMNTSpeechModelConfig, + ) {} + + private getArgs({ + text, + voice = 'ava', + outputFormat = 'mp3', + speed, + providerOptions, + }: Parameters[0]) { + const warnings: SpeechModelV1CallWarning[] = []; + + // Parse provider options + const lmntOptions = parseProviderOptions({ + provider: 'lmnt', + providerOptions, + schema: lmntSpeechCallOptionsSchema, + }); + + // Create request body + const requestBody: Record = { + model: this.modelId, + text, + voice, + response_format: 'mp3', + speed, + }; + + if (outputFormat) { + if (['mp3', 'aac', 'mulaw', 'raw', 'wav'].includes(outputFormat)) { + requestBody.response_format = outputFormat; + } else { + warnings.push({ + type: 'unsupported-setting', + setting: 'outputFormat', + details: `Unsupported output format: ${outputFormat}. Using mp3 instead.`, + }); + } + } + + // Add provider-specific options + if (lmntOptions) { + const speechModelOptions: Omit = { + conversational: lmntOptions.conversational ?? undefined, + length: lmntOptions.length ?? undefined, + seed: lmntOptions.seed ?? undefined, + speed: lmntOptions.speed ?? undefined, + temperature: lmntOptions.temperature ?? undefined, + top_p: lmntOptions.topP ?? undefined, + sample_rate: lmntOptions.sampleRate ?? undefined, + }; + + for (const key in speechModelOptions) { + const value = + speechModelOptions[ + key as keyof Omit + ]; + if (value !== undefined) { + requestBody[key] = value; + } + } + } + + return { + requestBody, + warnings, + }; + } + + async doGenerate( + options: Parameters[0], + ): Promise>> { + const currentDate = this.config._internal?.currentDate?.() ?? new Date(); + const { requestBody, warnings } = this.getArgs(options); + + const { + value: audio, + responseHeaders, + rawValue: rawResponse, + } = await postJsonToApi({ + url: this.config.url({ + path: '/v1/ai/speech/bytes', + modelId: this.modelId, + }), + headers: combineHeaders(this.config.headers(), options.headers), + body: requestBody, + failedResponseHandler: lmntFailedResponseHandler, + successfulResponseHandler: createBinaryResponseHandler(), + abortSignal: options.abortSignal, + fetch: this.config.fetch, + }); + + return { + audio, + warnings, + request: { + body: JSON.stringify(requestBody), + }, + response: { + timestamp: currentDate, + modelId: this.modelId, + headers: responseHeaders, + body: rawResponse, + }, + }; + } +} diff --git a/packages/lmnt/src/lmnt-speech-settings.ts b/packages/lmnt/src/lmnt-speech-settings.ts new file mode 100644 index 000000000000..377c029f980f --- /dev/null +++ b/packages/lmnt/src/lmnt-speech-settings.ts @@ -0,0 +1 @@ +export type LMNTSpeechModelId = 'aurora' | 'blizzard' | (string & {}); diff --git a/packages/lmnt/tsconfig.build.json b/packages/lmnt/tsconfig.build.json new file mode 100644 index 000000000000..8181e407dd6d --- /dev/null +++ b/packages/lmnt/tsconfig.build.json @@ -0,0 +1,7 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + // Disable project configuration for tsup builds + "composite": false + } +} diff --git a/packages/lmnt/tsconfig.json b/packages/lmnt/tsconfig.json new file mode 100644 index 000000000000..987760e4111d --- /dev/null +++ b/packages/lmnt/tsconfig.json @@ -0,0 +1,22 @@ +{ + "extends": "./node_modules/@vercel/ai-tsconfig/ts-library.json", + "compilerOptions": { + "composite": true, + "rootDir": "src", + "outDir": "dist" + }, + "exclude": [ + "dist", + "build", + "node_modules", + "tsup.config.ts" + ], + "references": [ + { + "path": "../provider" + }, + { + "path": "../provider-utils" + } + ] +} diff --git a/packages/lmnt/tsup.config.ts b/packages/lmnt/tsup.config.ts new file mode 100644 index 000000000000..3f92041b987c --- /dev/null +++ b/packages/lmnt/tsup.config.ts @@ -0,0 +1,10 @@ +import { defineConfig } from 'tsup'; + +export default defineConfig([ + { + entry: ['src/index.ts'], + format: ['cjs', 'esm'], + dts: true, + sourcemap: true, + }, +]); diff --git a/packages/lmnt/turbo.json b/packages/lmnt/turbo.json new file mode 100644 index 000000000000..620b8380e744 --- /dev/null +++ b/packages/lmnt/turbo.json @@ -0,0 +1,12 @@ +{ + "extends": [ + "//" + ], + "tasks": { + "build": { + "outputs": [ + "**/dist/**" + ] + } + } +} diff --git a/packages/lmnt/vitest.edge.config.js b/packages/lmnt/vitest.edge.config.js new file mode 100644 index 000000000000..700660e913f5 --- /dev/null +++ b/packages/lmnt/vitest.edge.config.js @@ -0,0 +1,10 @@ +import { defineConfig } from 'vite'; + +// https://vitejs.dev/config/ +export default defineConfig({ + test: { + environment: 'edge-runtime', + globals: true, + include: ['**/*.test.ts', '**/*.test.tsx'], + }, +}); diff --git a/packages/lmnt/vitest.node.config.js b/packages/lmnt/vitest.node.config.js new file mode 100644 index 000000000000..b1d14b21fc11 --- /dev/null +++ b/packages/lmnt/vitest.node.config.js @@ -0,0 +1,10 @@ +import { defineConfig } from 'vite'; + +// https://vitejs.dev/config/ +export default defineConfig({ + test: { + environment: 'node', + globals: true, + include: ['**/*.test.ts', '**/*.test.tsx'], + }, +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index cf7e56e2f895..6a883a8a0c82 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -104,6 +104,9 @@ importers: '@ai-sdk/hume': specifier: 1.0.0-canary.0 version: link:../../packages/hume + '@ai-sdk/lmnt': + specifier: 1.0.0-canary.0 + version: link:../../packages/lmnt '@ai-sdk/luma': specifier: 1.0.0-canary.8 version: link:../../packages/luma @@ -1611,6 +1614,31 @@ importers: specifier: 3.23.8 version: 3.23.8 + packages/lmnt: + dependencies: + '@ai-sdk/provider': + specifier: 2.0.0-canary.7 + version: link:../provider + '@ai-sdk/provider-utils': + specifier: 3.0.0-canary.8 + version: link:../provider-utils + devDependencies: + '@types/node': + specifier: 20.17.24 + version: 20.17.24 + '@vercel/ai-tsconfig': + specifier: workspace:* + version: link:../../tools/tsconfig + tsup: + specifier: ^8 + version: 8.3.0(jiti@2.4.0)(postcss@8.5.3)(tsx@4.19.2)(typescript@5.6.3)(yaml@2.7.0) + typescript: + specifier: 5.6.3 + version: 5.6.3 + zod: + specifier: 3.23.8 + version: 3.23.8 + packages/luma: dependencies: '@ai-sdk/provider': diff --git a/tsconfig.json b/tsconfig.json index 3d04cad6424b..7aacfdaf513b 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -92,6 +92,9 @@ }, { "path": "packages/xai" + }, + { + "path": "packages/lmnt" } ], "include": []