Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 135 additions & 0 deletions docs/codedocs/agent-conversations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
---
title: "Agent Conversations"
description: "How the agent WebSocket client composes listen, think, and speak providers into a live conversational session."
---

`AgentWebSocketClient` is the most compositional API in the SDK. Instead of representing one product endpoint, it packages a multi-stage conversation pipeline: audio input and output configuration, speech recognition, reasoning, speech generation, and optional function calling. The key request type is `Deepgram.Models.Agent.v2.WebSocket.SettingsSchema`.

The important implementation files are `Deepgram/Clients/Agent/v2/Websocket/Client.cs`, `Deepgram/Models/Agent/v2/WebSocket/Settings.cs`, `Deepgram/Models/Agent/v2/WebSocket/Agent.cs`, `Deepgram/Models/Agent/v2/WebSocket/Provider.cs`, and `Deepgram/Models/Agent/v2/WebSocket/FunctionCallResponseSchema.cs`.

## Why this concept exists

Agent sessions need more than a transcription stream or a TTS stream. They need a single live session that knows how to listen, think, and speak, and they often need to integrate other providers like OpenAI-style LLMs or custom endpoints. The SDK turns that complex payload into a typed model with enough escape hatches to support provider-specific configuration.

## How it relates to other concepts

- It shares the same WebSocket lifecycle and subscription model as [Streaming Transcription](/docs/streaming-transcription).
- It internally models STT, LLM, and TTS configuration together, so it overlaps conceptually with both speech and text features.
- It often pairs with `Deepgram.Microphone.Microphone` because agent sessions usually start from live audio capture.

## How it works internally

`SettingsSchema` contains top-level `Audio` and `Agent` objects. `Audio.Input` and `Audio.Output` carry formats like `linear16`, sample rate, and container settings. `Agent` contains nested `Listen`, `Think`, and `Speak` records, each of which exposes a `Provider`. The interesting part is `Provider.cs`: it inherits from `DynamicObject`, stores arbitrary JSON in `AdditionalProperties`, and converts C# dynamic member names to `snake_case` before serialization. That is how example code can write values like `settingsConfiguration.Agent.Think.Provider.Model = "gpt-4o-mini"` even though `Provider` only declares `Type` explicitly.

In `Deepgram/Clients/Agent/v2/Websocket/Client.cs`, `Connect(SettingsSchema, ...)` first opens the socket with `base.Connect`, then serializes `SettingsSchema` to JSON. Before sending the payload, the client removes empty nested provider objects from `agent.think`, `agent.speak`, and `agent.listen`. That cleanup avoids shipping empty `{}` objects for sections the caller did not configure. After the cleaned JSON is sent, the client can optionally start a keepalive loop and then publish events such as `WelcomeResponse`, `ConversationTextResponse`, `AgentThinkingResponse`, `AudioResponse`, `FunctionCallRequestResponse`, and `AgentAudioDoneResponse`.

```mermaid
graph TD
A[SettingsSchema] --> B[Audio.Input and Audio.Output]
A --> C[Agent.Listen]
A --> D[Agent.Think]
A --> E[Agent.Speak]
C --> F[Provider dynamic properties]
D --> F
E --> F
A --> G[Serialize to JSON]
G --> H[Remove empty provider objects]
H --> I[Send settings payload after socket connect]
I --> J[Receive welcome, text, function, and audio events]
```

## Basic usage

```csharp
using Deepgram;
using Deepgram.Models.Agent.v2.WebSocket;

var client = ClientFactory.CreateAgentWebSocketClient();

var settings = new SettingsSchema();
settings.Agent.Greeting = "Hello! How can I help?";
settings.Agent.Listen.Provider.Type = "deepgram";
settings.Agent.Listen.Provider.Model = "nova-3";
settings.Agent.Think.Provider.Type = "open_ai";
settings.Agent.Think.Provider.Model = "gpt-4o-mini";
settings.Agent.Speak.Provider.Type = "deepgram";
settings.Agent.Speak.Provider.Model = "aura-2-thalia-en";

await client.Subscribe(new EventHandler<ConversationTextResponse>((_, e) =>
{
Console.WriteLine(e.ToString());
}));

await client.Connect(settings);
await client.SendInjectUserMessage("Summarize today's support queue.");
```

## Advanced usage

```csharp
using Deepgram;
using Deepgram.Models.Agent.v2.WebSocket;

var client = ClientFactory.CreateAgentWebSocketClient();

var settings = new SettingsSchema
{
Experimental = true,
Tags = new List<string> { "support-bot", "dotnet" }
};

settings.Audio.Input.Encoding = "linear16";
settings.Audio.Input.SampleRate = 24000;
settings.Audio.Output.Encoding = "linear16";
settings.Audio.Output.SampleRate = 24000;
settings.Audio.Output.Container = "none";

settings.Agent.Think.Provider.Type = "open_ai";
settings.Agent.Think.Provider.Model = "gpt-4o-mini";
settings.Agent.Think.Functions = new List<Function>
{
new Function
{
Name = "lookup_ticket",
Description = "Fetch a ticket by id"
}
};

await client.Subscribe(new EventHandler<FunctionCallRequestResponse>(async (_, e) =>
{
var result = new FunctionCallResponseSchema
{
FunctionCallId = e.FunctionCallId,
Output = "{"status":"open","priority":"high"}"
};

var payload = System.Text.Encoding.UTF8.GetBytes(result.ToString());
await client.SendMessageImmediately(payload);
}));

await client.Connect(settings);
```

<Callout type="warn">The provider objects in the agent schema are intentionally dynamic. That makes the API flexible, but it also means property names are not compile-time validated the way normal strongly typed models are. A misspelled provider-specific property will still serialize, and the server is the component that ultimately rejects or ignores it.</Callout>

<Accordions>
<Accordion title="Strongly typed shell vs dynamic provider payloads">
The agent schema is a deliberate compromise. `SettingsSchema`, `Audio`, `Agent`, `Listen`, `Think`, and `Speak` are strongly typed enough to guide the overall structure, while `Provider` remains dynamic so the SDK does not need a new class every time a provider adds a proprietary field. That flexibility is powerful for experimentation and third-party integrations, but it reduces editor-time safety compared with the rest of the SDK. When you need reliability, build small helper methods in your own codebase that set known-good provider keys in one place.

```csharp
settings.Agent.Think.Provider.Type = "open_ai";
settings.Agent.Think.Provider.Model = "gpt-4o-mini";
settings.Agent.Listen.Provider.Keyterms = new List<string> { "Deepgram" };
```

</Accordion>
<Accordion title="Injected text vs live microphone audio">
`SendInjectUserMessage` is the simplest way to drive an agent because it lets you send a structured text message without managing raw audio. Live microphone audio feels more natural and better matches production voice experiences, but it adds capture, timing, and playback complexity. The trade-off is development speed versus realism: injected text is excellent for testing prompts and function calls, while microphone streaming is the right path for end-to-end voice validation. Teams usually move faster if they stabilize agent behavior with injected text first and add audio transport second.

```csharp
await client.SendInjectUserMessage("What can you do?");
client.SendBinary(audioChunk);
```

</Accordion>
</Accordions>
99 changes: 99 additions & 0 deletions docs/codedocs/api-reference/agent-websocket-client.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
---
title: "AgentWebSocketClient"
description: "Configure live agent sessions, subscribe to agent events, and send text or audio into the conversation."
---

Source files: `Deepgram/AgentWebSocketClient.cs`, `Deepgram/Clients/Interfaces/v2/IAgentWebSocketClient.cs`, `Deepgram/Clients/Agent/v2/Websocket/Client.cs`.

Import paths:

- `using Deepgram;`
- `using Deepgram.Models.Agent.v2.WebSocket;`

Constructor:

```csharp
public AgentWebSocketClient(
string apiKey = "",
DeepgramWsClientOptions? deepgramClientOptions = null)
```

## Public methods

### Connection lifecycle

```csharp
Task<bool> Connect(
SettingsSchema options,
CancellationTokenSource? cancelToken = null,
Dictionary<string, string>? addons = null,
Dictionary<string, string>? headers = null)

Task<bool> Stop(CancellationTokenSource? cancelToken = null, bool nullByte = false)
```

### Subscriptions

```csharp
Task<bool> Subscribe(EventHandler<OpenResponse> eventHandler)
Task<bool> Subscribe(EventHandler<AudioResponse> eventHandler)
Task<bool> Subscribe(EventHandler<AgentAudioDoneResponse> eventHandler)
Task<bool> Subscribe(EventHandler<AgentStartedSpeakingResponse> eventHandler)
Task<bool> Subscribe(EventHandler<AgentThinkingResponse> eventHandler)
Task<bool> Subscribe(EventHandler<ConversationTextResponse> eventHandler)
Task<bool> Subscribe(EventHandler<FunctionCallRequestResponse> eventHandler)
Task<bool> Subscribe(EventHandler<UserStartedSpeakingResponse> eventHandler)
Task<bool> Subscribe(EventHandler<WelcomeResponse> eventHandler)
Task<bool> Subscribe(EventHandler<CloseResponse> eventHandler)
Task<bool> Subscribe(EventHandler<UnhandledResponse> eventHandler)
Task<bool> Subscribe(EventHandler<ErrorResponse> eventHandler)
Task<bool> Subscribe(EventHandler<SettingsAppliedResponse> eventHandler)
Task<bool> Subscribe(EventHandler<InjectionRefusedResponse> eventHandler)
Task<bool> Subscribe(EventHandler<PromptUpdatedResponse> eventHandler)
Task<bool> Subscribe(EventHandler<SpeakUpdatedResponse> eventHandler)
```

### Send and helper methods

```csharp
Task SendKeepAlive()
Task SendInjectUserMessage(string content)
Task SendInjectUserMessage(InjectUserMessageSchema injectUserMessageSchema)
Task SendClose(bool nullByte = false, CancellationTokenSource? _cancellationToken = null)
void SendBinary(byte[] data, int length = Constants.UseArrayLengthForSend)
void SendMessage(byte[] data, int length = Constants.UseArrayLengthForSend)
Task SendBinaryImmediately(byte[] data, int length = Constants.UseArrayLengthForSend, CancellationTokenSource? _cancellationToken = null)
Task SendMessageImmediately(byte[] data, int length = Constants.UseArrayLengthForSend, CancellationTokenSource? _cancellationToken = null)
WebSocketState State()
bool IsConnected()
```

## Main schema types

| Type | Important fields | Notes |
|-----------|------|-------------|
| `SettingsSchema` | `Experimental`, `Tags`, `MipOptOut`, `Audio`, `Agent` | Initial settings payload sent immediately after connect. |
| `Agent` | `Language`, `Listen`, `Think`, `Speak`, `Greeting` | Conversation behavior. |
| `Input` | `Encoding`, `SampleRate` | Input audio format. |
| `Output` | `Encoding`, `SampleRate`, `Bitrate`, `Container` | Output audio format. |
| `InjectUserMessageSchema` | `Type`, `Content` | Text injection without microphone audio. |
| `FunctionCallResponseSchema` | `FunctionCallId`, `Output` | Response message for function-calling flows. |

## Example

```csharp
var client = ClientFactory.CreateAgentWebSocketClient();

var settings = new SettingsSchema();
settings.Agent.Think.Provider.Type = "open_ai";
settings.Agent.Think.Provider.Model = "gpt-4o-mini";
settings.Agent.Listen.Provider.Type = "deepgram";
settings.Agent.Listen.Provider.Model = "nova-3";
settings.Agent.Speak.Provider.Type = "deepgram";
settings.Agent.Speak.Provider.Model = "aura-2-thalia-en";

await client.Connect(settings);
await client.SendInjectUserMessage("What is still unresolved in our queue?");
```

Related pages: [Agent Conversations](/docs/agent-conversations), [Guides: Build a Voice Agent](/docs/guides/build-a-voice-agent).
64 changes: 64 additions & 0 deletions docs/codedocs/api-reference/analyze-client.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
title: "AnalyzeClient"
description: "Text analysis methods for intent, sentiment, topic, and summary workflows."
---

Source files: `Deepgram/AnalyzeClient.cs`, `Deepgram/Clients/Interfaces/v1/IAnalyzeClient.cs`, `Deepgram/Clients/Analyze/v1/Client.cs`.

Import paths:

- `using Deepgram;`
- `using Deepgram.Models.Analyze.v1;`

Constructor:

```csharp
public AnalyzeClient(
string apiKey = "",
DeepgramHttpClientOptions? deepgramClientOptions = null,
string? httpId = null)
```

## Public methods

```csharp
Task<SyncResponse> AnalyzeUrl(UrlSource source, AnalyzeSchema? analyzeSchema, CancellationTokenSource? cancellationToken = default, Dictionary<string, string>? addons = null, Dictionary<string, string>? headers = null)
Task<SyncResponse> AnalyzeText(TextSource source, AnalyzeSchema? analyzeSchema, CancellationTokenSource? cancellationToken = default, Dictionary<string, string>? addons = null, Dictionary<string, string>? headers = null)
Task<SyncResponse> AnalyzeFile(byte[] source, AnalyzeSchema? analyzeSchema, CancellationTokenSource? cancellationToken = default, Dictionary<string, string>? addons = null, Dictionary<string, string>? headers = null)
Task<SyncResponse> AnalyzeFile(Stream source, AnalyzeSchema? analyzeSchema, CancellationTokenSource? cancellationToken = default, Dictionary<string, string>? addons = null, Dictionary<string, string>? headers = null)
Task<AsyncResponse> AnalyzeFileCallBack(byte[] source, string? callBack, AnalyzeSchema? analyzeSchema, CancellationTokenSource? cancellationToken = default, Dictionary<string, string>? addons = null, Dictionary<string, string>? headers = null)
Task<AsyncResponse> AnalyzeFileCallBack(Stream source, string? callBack, AnalyzeSchema? analyzeSchema, CancellationTokenSource? cancellationToken = default, Dictionary<string, string>? addons = null, Dictionary<string, string>? headers = null)
Task<AsyncResponse> AnalyzeUrlCallBack(UrlSource source, string? callBack, AnalyzeSchema? analyzeSchema, CancellationTokenSource? cancellationToken = default, Dictionary<string, string>? addons = null, Dictionary<string, string>? headers = null)
Task<AsyncResponse> AnalyzeTextCallBack(TextSource source, string? callBack, AnalyzeSchema? analyzeSchema, CancellationTokenSource? cancellationToken = default, Dictionary<string, string>? addons = null, Dictionary<string, string>? headers = null)
```

## Main schema parameters

| Property | Type | Default | Description |
|-----------|------|---------|-------------|
| `Language` | `string?` | `null` | Input language. Current schema comments document English support. |
| `Summarize` | `bool?` | `null` | Generate a summary. |
| `Sentiment` | `bool?` | `null` | Detect sentiment shifts. |
| `Topics` | `bool?` | `null` | Detect topics. |
| `Intents` | `bool?` | `null` | Detect intent segments. |
| `CustomIntent` | `List<string>?` | `null` | Custom intents. |
| `CustomTopic` | `List<string>?` | `null` | Custom topics. |
| `CallBack` | `string?` | `null` | Async callback URL. |

## Example

```csharp
var client = ClientFactory.CreateAnalyzeClient();

var response = await client.AnalyzeText(
new TextSource("The customer sounds frustrated but still wants to renew."),
new AnalyzeSchema
{
Language = "en",
Sentiment = true,
Intents = true,
Summarize = true
});
```

Related pages: [Client Factory and Options](/docs/client-factory-and-options), [Types](/docs/types).
85 changes: 85 additions & 0 deletions docs/codedocs/api-reference/auth-client.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
---
title: "AuthClient"
description: "Grant temporary Deepgram tokens using the SDK's auth client."
---

Source files: `Deepgram/AuthClient.cs`, `Deepgram/Clients/Interfaces/v1/IAuthClient.cs`, `Deepgram/Clients/Auth/v1/Client.cs`.

Import paths:

- `using Deepgram;`
- `using Deepgram.Models.Auth.v1;`

Constructor:

```csharp
public AuthClient(
string apiKey = "",
DeepgramHttpClientOptions? deepgramClientOptions = null,
string? httpId = null)
```

## Public methods

```csharp
Task<GrantTokenResponse> GrantToken(
CancellationTokenSource? cancellationToken = default,
Dictionary<string, string>? addons = null,
Dictionary<string, string>? headers = null)

Task<GrantTokenResponse> GrantToken(
GrantTokenSchema grantTokenSchema,
CancellationTokenSource? cancellationToken = default,
Dictionary<string, string>? addons = null,
Dictionary<string, string>? headers = null)
```

## Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `grantTokenSchema.TtlSeconds` | `int?` | `null` | Token TTL in seconds, validated to the range `1` to `3600`. |

Behavior notes from `Deepgram/Clients/Auth/v1/Client.cs`:

- The parameterless `GrantToken()` posts an empty body to `auth/grant`.
- The overload with `GrantTokenSchema` throws `ArgumentNullException` if you pass `null`.
- Both methods return `GrantTokenResponse`, which is the SDK's typed wrapper around the temporary JWT response.

Use this client when you want short-lived credentials in frontends or worker processes without distributing a long-lived API key to every runtime. The auth client still requires normal SDK authentication to mint the temporary token in the first place.

Response usage pattern:

- Request a token in a backend or trusted worker.
- Hand the short-lived token to a less-trusted runtime that should not hold the main API key.
- Build `DeepgramHttpClientOptions` or `DeepgramWsClientOptions` with `accessToken` instead of `apiKey`.

That pattern lines up with the credential-precedence logic in the options classes, where `AccessToken` always wins over `ApiKey` when both are present.

Example of consuming the minted token:

```csharp
var grant = await client.GrantToken(new GrantTokenSchema { TtlSeconds = 120 });

var wsOptions = new Deepgram.Models.Authenticate.v1.DeepgramWsClientOptions(
accessToken: grant.AccessToken);

var live = ClientFactory.CreateListenWebSocketClient(options: wsOptions);
```

That flow is especially useful when your application has a trusted service layer that should keep the primary API key private while still enabling downstream live or REST requests.

## Example

```csharp
var client = ClientFactory.CreateAuthClient();

var token = await client.GrantToken(new GrantTokenSchema
{
TtlSeconds = 300
});

Console.WriteLine(token);
```

Related pages: [Library and ClientFactory](/docs/api-reference/library-and-client-factory).
Loading