Skip to content

Commit d3b131a

Browse files
committed
feat: wire Speak V2 (Flux) streaming into DeepgramClient + cover new regen surface
Speak V2 (Flux streaming TTS, /v2/speak) shipped in the 2026-07-09 regen but was not usable via DeepgramClient: it was not wrapped by CustomClient, and the generated V2 socket parses every frame as JSON (which drops binary audio). - CustomClient: add WrappedSpeakV2Client + binary-aware WrappedSpeakV2Socket, a speak.v2 getter, the SpeakV2ConnectionArgs type and SpeakV2ClientWithWebSocket interface — mirrors the speak.v1 pair so auth/session/custom-transport apply. - transport.ts: add "speak.v2" to the streaming service union. - examples: 37 (Flux streaming TTS), 38 (listen v2 end-of-turn tuning); registered in the Makefile (targets, run list, TOTAL, help). - tests: speak-v2-tts wire test (binary audio + control messages), agent UpdateListen/ListenUpdated wire test, speak.v2 accessor unit test, and regen-constraints coverage for the new eot_* provider fields, agent listen update types, and Speak V2 types. - AGENTS.md: document the CustomClient/transport streaming-wrapper rule and correct the stale ListenV2CloseStream.Type note (keeps deprecated members).
1 parent a4f58f6 commit d3b131a

10 files changed

Lines changed: 824 additions & 7 deletions

AGENTS.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,9 @@ How to identify:
2222
- The file lives **outside `src/api/`** in a hand-maintained location (e.g., `.claude/`, `docs/`, `scripts/`)
2323

2424
Current permanently frozen files:
25-
- `src/CustomClient.ts` — entirely custom wrapper with WebSocket management, auth providers, session ID handling, `transportFactory` plumbing, and the `reconnect` flag (auto-disabled when a custom transport is in use); no Fern equivalent
25+
- `src/CustomClient.ts` — entirely custom wrapper with WebSocket management, auth providers, session ID handling, `transportFactory` plumbing, and the `reconnect` flag (auto-disabled when a custom transport is in use); no Fern equivalent. **Wraps every streaming client** (`agent.v1`, `listen.v1`, `listen.v2`, `speak.v1`, `speak.v2`) so connections route through `createWebSocketConnection` (auth/session/custom transport) and so binary audio frames are handled correctly. When the generator adds a new streaming resource (e.g. the 2026-07-09 `speak.v2` Flux TTS API), add a matching `Wrapped…Client` + binary-aware `Wrapped…Socket`, a getter on the parent `Wrapped…Client`, and the `…WithWebSocket` interface here — mirror the existing `speak.v1` pair.
2626
- `src/index.ts` — curated re-export file with custom namespace handling
27-
- `src/transport.ts` — hand-maintained pluggable transport interface for SageMaker and other custom streaming transports
27+
- `src/transport.ts` — hand-maintained pluggable transport interface for SageMaker and other custom streaming transports. The `service` union enumerates every streaming resource (`agent.v1` | `listen.v1` | `listen.v2` | `speak.v1` | `speak.v2`); extend it when `CustomClient.ts` gains a new wrapped streaming client.
2828
- `scripts/fix-wire-test-imports.js`, `scripts/revert-wire-test-imports.js` — post-generation import fixup scripts
2929
- `scripts/proxy-server.js` — development proxy server
3030
- `scripts/validate-esm-build.mjs` — ESM build validation
@@ -66,7 +66,7 @@ Current temporarily frozen files:
6666
- `src/api/types/CreateKeyV1Request.ts` — preserve the old exported `CreateKeyV1RequestOne` alias alongside `CreateKeyV1Request` so existing type imports keep compiling.
6767
- `src/api/resources/agent/resources/v1/types/AgentV1Settings.ts` — backward-compat patch for the `AgentV1Settings.Agent` namespace. The 2026-05-06 regen restructured `Agent` from an interface with named sub-types into a union with anonymous shapes plus a string variant (`{...} | string`) for the new agent-by-ID-string flow. We restore the original interface form so existing imports of the sub-types (`Agent.Context`, `Agent.Listen`, `Agent.Think`, `Agent.Speak`, `Agent.Context.Messages.Item`) AND direct field reads (`settings.agent.context`, `.greeting`, etc.) keep compiling in TS strict mode without a `typeof` narrowing guard. The new agent-by-ID-string variant is surfaced as the opt-in `AgentReference = Agent | string` alias for callers who want to pass a string ID at the type level. Mirrors the Python SDK's compat shims for the same regen. Regression coverage in `tests/unit/compat-aliases.test.ts`.
6868
- `src/api/types/DeepgramListenProviderV2.ts` — backward-compat shim recreating the `DeepgramListenProviderV2.LanguageHint` (`string | string[]`) namespace type that the 2026-06-16 regen removed when it replaced the `language_hint` field with `language_hints?: string[]`. The nested `AgentV1SettingsAgentContextListenProviderV2.LanguageHint` alias (below) and the `tests/unit/compat-aliases.test.ts` regression test still reference it, so we recreate the namespace via declaration merge in this generated file and freeze it. Mirrors the Python SDK's `DeepgramListenProviderV2LanguageHint` shim. **Also** re-adds the deprecated optional `language_hint` field (the regen renamed it to `language_hints`) so existing call sites keep compiling — avoids a major-version break. The singular field was never honored by the API, so this is compile-compat only.
69-
- `src/api/resources/listen/resources/v2/types/ListenV2CloseStream.ts` — backward-compat shim. The 2026-06-16 regen corrected the `type` field to the literal `"CloseStream"` and dropped the `ListenV2CloseStream.Type` namespace, which wrongly included `"Finalize"`/`"KeepAlive"` (copied from v1's control-message enum). We restore the `Type` namespace with **only** the valid `"CloseStream"` value so existing `ListenV2CloseStream.Type` references keep compiling, without resurrecting the invalid values — avoids a major-version break.
69+
- `src/api/resources/listen/resources/v2/types/ListenV2CloseStream.ts` — backward-compat shim. The 2026-06-16 regen corrected the `type` field to the literal `"CloseStream"` and dropped the `ListenV2CloseStream.Type` namespace. We restore the `Type` namespace so existing `ListenV2CloseStream.Type` references keep compiling. The namespace keeps `"Finalize"` and `"KeepAlive"` as **deprecated** members (marked `@deprecated`, valid value is `"CloseStream"`) rather than removing them outright — dropping them would break existing references, so they're retained for one more major and the shim also widens the `type` field to `ListenV2CloseStream.Type` to preserve assignability. Regression coverage in `tests/unit/regen-constraints.test.ts` (asserts `CloseStream` plus the deprecated `Finalize`/`KeepAlive` members).
7070
- **Note:** `src/api/resources/agent/resources/v1/types/AgentV1SettingsAgentListenProvider.ts` was *unfrozen* in the 2026-06-16 regen — the generator now emits a type structurally equal to `AgentV1SettingsAgentContextListenProvider` (verified by the `Equals<…>` assertion in `tests/unit/compat-aliases.test.ts`), so the hand-written alias is no longer needed and Fern owns the file again.
7171
- `biome.json` — Fern's generator emits `"vcs": { "enabled": false }`, which makes biome lint every tracked-or-untracked file under the repo, including `examples/browser/deepgram.js` (the bundle produced by `make browser-serve`, which is `.gitignore`d but not biome-ignored). We restore `clientKind: "git"`, `enabled: true`, `useIgnoreFile: true` so generated browser artifacts stay out of lint.
7272
- `src/api/resources/agent/resources/v1/types/AgentV1SettingsAgentContextListenProvider.ts` — backward-compat shim for the listen-provider dedup in the 2026-05-14 regen. The new spec consolidated 5 PascalCase listen-provider types into top-level `DeepgramListenProviderV1` / `DeepgramListenProviderV2` and dropped the nested `AgentV1SettingsAgentContextListenProviderV2.LanguageHint` namespace path. We restore the nested namespace as a one-line alias `type LanguageHint = DeepgramListenProviderV2.LanguageHint` inside the outer namespace so deep import paths keep resolving. Regression coverage in `tests/unit/compat-aliases.test.ts`.

Makefile

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: help examples example-1 example-2 example-3 example-4 example-5 example-6 example-7 example-8 example-9 example-10 example-11 example-12 example-13 example-14 example-15 example-16 example-17 example-18 example-19 example-20 example-21 example-22 example-23 example-24 example-25 example-26 example-27 example-28 example-29 example-30 example-31 example-32 example-33 example-34 example-35 example-36 test test-esm typecheck-tests lint build browser browser-serve
1+
.PHONY: help examples example-1 example-2 example-3 example-4 example-5 example-6 example-7 example-8 example-9 example-10 example-11 example-12 example-13 example-14 example-15 example-16 example-17 example-18 example-19 example-20 example-21 example-22 example-23 example-24 example-25 example-26 example-27 example-28 example-29 example-30 example-31 example-32 example-33 example-34 example-35 example-36 example-37 example-38 test test-esm typecheck-tests lint build browser browser-serve
22

33
# Default target
44
help:
@@ -11,7 +11,7 @@ help:
1111
@printf " \033[1;32mmake test-esm\033[0m - Run ESM build validation tests\n"
1212
@echo ""
1313
@printf "\033[1;33mExample Commands:\033[0m\n"
14-
@printf " \033[1;32mmake examples\033[0m - Run all example scripts (1-32) sequentially\n"
14+
@printf " \033[1;32mmake examples\033[0m - Run all example scripts (1-38) sequentially\n"
1515
@printf " \033[1;32mmake example-N\033[0m - Run a specific example by number (e.g., make example-1)\n"
1616
@printf " \033[1;32mmake browser\033[0m - Run browser tests\n"
1717
@printf " \033[1;32mmake browser-serve\033[0m - Serve the browser examples for manual testing\n"
@@ -57,7 +57,7 @@ help:
5757
# Run all examples
5858
examples:
5959
@printf "\033[1;36mRunning all examples...\033[0m\n\n"; \
60-
TOTAL=36; \
60+
TOTAL=38; \
6161
PASS_COUNT=0; \
6262
FAIL_COUNT=0; \
6363
PASSED_LIST=""; \
@@ -136,6 +136,8 @@ examples:
136136
run_example 34 "examples/34-agent-custom-providers.ts" "Agent Custom Providers"; \
137137
run_example 35 "examples/35-agent-provider-combinations.ts" "Agent Provider Combinations"; \
138138
run_example 36 "examples/36-agent-inject-message.ts" "Agent Inject Message"; \
139+
run_example 37 "examples/37-text-to-speech-streaming-flux.ts" "Text-to-Speech Streaming Flux (v2)"; \
140+
run_example 38 "examples/38-transcription-flux-eot.ts" "Transcription Flux End-of-Turn"; \
139141
\
140142
printf "\n\033[1;36m=========================================\033[0m\n"; \
141143
printf "\033[1;36mSummary Report\033[0m\n"; \
@@ -280,6 +282,12 @@ example-35:
280282
example-36:
281283
pnpm exec tsx examples/36-agent-inject-message.ts
282284

285+
example-37:
286+
pnpm exec tsx examples/37-text-to-speech-streaming-flux.ts
287+
288+
example-38:
289+
pnpm exec tsx examples/38-transcription-flux-eot.ts
290+
283291
lint:
284292
pnpm exec biome lint --skip-parse-errors --no-errors-on-unmatched --max-diagnostics=none
285293
pnpm exec biome format --skip-parse-errors --no-errors-on-unmatched --max-diagnostics=none
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
/**
2+
* Example: Text-to-Speech Streaming with Flux (WebSocket, /v2/speak)
3+
*
4+
* Connect to Deepgram's v2 streaming TTS websocket (Flux voices) and send text
5+
* to synthesize speech. Flux uses the v2 message set: Speak / Flush / Close, and
6+
* emits Connected, SpeechStarted, SpeechMetadata, Flushed and SessionMetadata
7+
* control messages alongside binary audio frames.
8+
*
9+
* Note: Flux models use the `flux-{voice}-{language}` naming (e.g. flux-alexis-en).
10+
* Aura voices are not valid on /v2/speak — use speak.v1 for Aura.
11+
*/
12+
13+
const { DeepgramClient } = require("../dist/cjs/index.js");
14+
15+
const deepgramClient = new DeepgramClient({
16+
apiKey: process.env.DEEPGRAM_API_KEY,
17+
});
18+
19+
async function textToSpeechStreamingFlux() {
20+
try {
21+
// Create a connection object with Flux TTS options (not yet connected)
22+
const deepgramConnection = await deepgramClient.speak.v2.createConnection({
23+
model: "flux-alexis-en",
24+
encoding: "linear16",
25+
sample_rate: "24000",
26+
});
27+
28+
// Set up event handlers before connecting
29+
deepgramConnection.on("open", () => {
30+
console.log("Connection opened");
31+
});
32+
33+
let flushed = false;
34+
35+
deepgramConnection.on("message", (data) => {
36+
// Binary audio frames arrive as ArrayBuffer/Blob (or string in some runtimes)
37+
if (typeof data === "string" || data instanceof ArrayBuffer || data instanceof Blob) {
38+
console.log(
39+
"Audio received (length):",
40+
data instanceof ArrayBuffer ? data.byteLength : data instanceof Blob ? data.size : data.length,
41+
);
42+
// Process audio data here
43+
return;
44+
}
45+
46+
switch (data.type) {
47+
case "Connected":
48+
console.log("Connected:", data.request_id, data.model_name);
49+
break;
50+
case "SpeechStarted":
51+
console.log("Speech started:", data.speech_id);
52+
break;
53+
case "SpeechMetadata":
54+
console.log("Speech metadata:", data);
55+
break;
56+
case "Flushed":
57+
console.log("Flushed:", data.speech_id);
58+
flushed = true;
59+
// Wait a moment for any remaining audio, then close
60+
setTimeout(() => {
61+
deepgramConnection.close();
62+
}, 1000);
63+
break;
64+
case "SessionMetadata":
65+
console.log("Session metadata:", data);
66+
break;
67+
case "Warning":
68+
console.warn("Warning:", data.code, data.description);
69+
break;
70+
case "Error":
71+
console.error("Error:", data.code, data.description);
72+
break;
73+
default:
74+
console.log("Unknown message type:", data);
75+
}
76+
});
77+
78+
deepgramConnection.on("error", (error) => {
79+
console.error("Error:", error);
80+
});
81+
82+
deepgramConnection.on("close", () => {
83+
console.log("Connection closed");
84+
});
85+
86+
// Connect to the websocket
87+
deepgramConnection.connect();
88+
89+
// Wait for connection to open before sending data
90+
try {
91+
await deepgramConnection.waitForOpen();
92+
93+
const text = "Hello, this is a test of Deepgram's Flux streaming text-to-speech API.";
94+
95+
// Send text for synthesis (type: "Speak"), then Flush to finish the turn
96+
deepgramConnection.sendSpeak({ type: "Speak", text });
97+
deepgramConnection.sendFlush({ type: "Flush" });
98+
99+
// Fallback close in case Flushed never comes
100+
setTimeout(() => {
101+
if (!flushed) {
102+
console.log("Timeout waiting for Flushed message, closing connection");
103+
deepgramConnection.close();
104+
}
105+
}, 10000);
106+
107+
// Kill websocket after 1 minute, so we can run these in CI
108+
setTimeout(() => {
109+
deepgramConnection.close();
110+
process.exit(0);
111+
}, 60000);
112+
} catch (error) {
113+
console.error("Error waiting for connection:", error);
114+
deepgramConnection.close();
115+
}
116+
} catch (error) {
117+
console.error("Error setting up connection:", error);
118+
}
119+
}
120+
121+
textToSpeechStreamingFlux();
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
/**
2+
* Example: Live Transcription (V2 / Flux) with End-of-Turn tuning
3+
*
4+
* Flux exposes end-of-turn (EOT) controls on the /v2/listen websocket that let
5+
* you tune how eagerly a turn is finalized:
6+
*
7+
* - eot_threshold: EOT confidence required to finish a turn (0.5 - 0.9, default 0.7)
8+
* - eager_eot_threshold: EOT confidence to fire an *eager* end-of-turn event, enabling
9+
* EagerEndOfTurn / TurnResumed events (0.3 - 0.9)
10+
* - eot_timeout_ms: a turn is finished this many ms after speech regardless of
11+
* EOT confidence (default 5000)
12+
*
13+
* These are sent as query parameters on the connection.
14+
*/
15+
16+
const { DeepgramClient } = require("../dist/cjs/index.js");
17+
const { createReadStream } = require("fs");
18+
19+
const deepgramClient = new DeepgramClient({
20+
apiKey: process.env.DEEPGRAM_API_KEY,
21+
});
22+
23+
async function liveTranscriptionFluxEot() {
24+
try {
25+
const deepgramConnection = await deepgramClient.listen.v2.createConnection({
26+
model: "flux-general-en",
27+
// End-of-turn tuning for Flux
28+
eot_threshold: 0.8,
29+
eager_eot_threshold: 0.5,
30+
eot_timeout_ms: 4000,
31+
});
32+
33+
deepgramConnection.on("open", () => {
34+
console.log("Connection opened (Flux with EOT tuning)");
35+
});
36+
37+
deepgramConnection.on("message", (data) => {
38+
switch (data.type) {
39+
case "Connected":
40+
console.log("Connected:", data);
41+
break;
42+
case "TurnInfo":
43+
console.log("Turn Info:", data);
44+
break;
45+
case "EagerEndOfTurn":
46+
console.log("Eager end-of-turn:", data);
47+
break;
48+
case "TurnResumed":
49+
console.log("Turn resumed:", data);
50+
break;
51+
case "FatalError":
52+
console.error("Fatal Error:", data);
53+
deepgramConnection.close();
54+
break;
55+
default:
56+
console.log("Message:", data);
57+
}
58+
});
59+
60+
deepgramConnection.on("error", (error) => {
61+
console.error("Error:", error);
62+
});
63+
64+
deepgramConnection.on("close", () => {
65+
console.log("Connection closed");
66+
});
67+
68+
deepgramConnection.connect();
69+
70+
try {
71+
await deepgramConnection.waitForOpen();
72+
73+
const audioStream = createReadStream("./examples/spacewalk.wav");
74+
75+
audioStream.on("data", (chunk) => {
76+
deepgramConnection.sendMedia(chunk);
77+
});
78+
79+
audioStream.on("end", () => {
80+
console.log("Audio stream ended");
81+
setTimeout(() => {
82+
deepgramConnection.sendCloseStream({ type: "CloseStream" });
83+
}, 3000);
84+
});
85+
86+
// Kill websocket after 60 seconds so this can run in CI
87+
setTimeout(() => {
88+
console.log("Timeout reached, closing connection");
89+
deepgramConnection.close();
90+
process.exit(0);
91+
}, 60000);
92+
} catch (error) {
93+
console.error("Error waiting for connection:", error);
94+
deepgramConnection.close();
95+
}
96+
} catch (error) {
97+
console.error("Error setting up connection:", error);
98+
}
99+
}
100+
101+
liveTranscriptionFluxEot();

0 commit comments

Comments
 (0)