Skip to content

Commit 6034b15

Browse files
committed
Add the VoiceAgent core: facade, funnel, turn detection, clause chunking
Phase 3 of the AWS.Speech.MEAI stack. Composes the Polly TTS and Transcribe STT clients from PR 2 and any MEAI IChatClient into a full-duplex voice loop that emits one ordered stream of VoiceAgentUpdate values. Public API (all net8-only, all [Experimental("MEAI001")]): - VoiceAgent (IAsyncDisposable): provider-neutral ctor + Create(...) factory that constructs Amazon Transcribe / Bedrock / Polly clients from the default chains, honoring VoiceAgentOptions.Credentials and Region. Owned AWS clients are disposed on DisposeAsync. GetService exposes the underlying MEAI clients. - VoiceAgentOptions: ModelId, Voice, Instructions, Language, sample rates, Tools, Region, Credentials, Backend (Pipeline default; NovaSonic reserved for PR 5), plus pre-built STT/chat/TTS overrides. - VoiceAgentUpdate readonly record struct and VoiceAgentUpdateKind (UserTranscriptPartial/Final, TurnStarted, AssistantText, AssistantAudio, TurnComplete, Cancelled). - VoiceAgentBackend (Pipeline, NovaSonic). Create throws NotSupportedException for NovaSonic; wired in PR 5. Internal implementation: - VoiceAgentPipeline drives the loop. A single-reader output channel funnels updates from two producers (an STT consumer and a reasoning-plus-TTS worker) so the caller sees one ordered stream. A separate user-turn channel enqueues each finalized user utterance for the reasoning worker to drain. - ClauseChunker returns the first punctuation boundary at or past a minimum clause length so completed clauses can be spoken while the model is still writing later ones. This keeps time-to-first-audio low. - History is maintained in-process; Instructions seeds a system message. - On teardown or cancellation, the linked CTS cancels both producers, the writer closes, and the first non-cancellation failure is rethrown via ExceptionDispatchInfo. Not in this PR (stack roadmap): - Barge-in, the Nova Sonic backend, AsIRealtimeClient(), and AddVoiceAgent DI (PR 5). - Live device sample and integration tests (PRs 6 and 7). Testing: 5 clause-chunker tests + 6 pipeline tests via inline STT/chat/TTS stubs (single-turn ordering, clause chunking, whitespace-final gate, system-message seeding, null-arg + disposal + GetService). All Speech tests pass on net8.0 (24) and net472 (7, unchanged). All 243 net8.0 + 173 net472 Bedrock/Nova tests still pass unchanged.
1 parent 8f0b241 commit 6034b15

9 files changed

Lines changed: 966 additions & 0 deletions

File tree

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
#if NET8_0_OR_GREATER
5+
using System.Text;
6+
7+
namespace AWS.Speech.MEAI;
8+
9+
/// <summary>
10+
/// Finds clause boundaries in streaming assistant text so the pipeline can send completed clauses to
11+
/// text-to-speech while the model is still writing later ones. This keeps time-to-first-audio low.
12+
/// </summary>
13+
internal static class ClauseChunker
14+
{
15+
/// <summary>Minimum clause length in characters; guards against flushing single-word fragments.</summary>
16+
internal const int MinClauseLength = 12;
17+
18+
/// <summary>
19+
/// Returns the length of the prefix of <paramref name="buffer"/> that should be flushed as a clause,
20+
/// or 0 if no boundary meets the minimum length yet.
21+
/// </summary>
22+
/// <remarks>
23+
/// Walks from <see cref="MinClauseLength"/> - 1 onwards so short fragments never trigger a flush.
24+
/// Returns at the first boundary that qualifies so first-audio latency stays low.
25+
/// </remarks>
26+
public static int NextClauseBoundary(StringBuilder buffer)
27+
{
28+
for (int i = MinClauseLength - 1; i < buffer.Length; i++)
29+
{
30+
switch (buffer[i])
31+
{
32+
case '.':
33+
case '?':
34+
case '!':
35+
case ',':
36+
case ';':
37+
case ':':
38+
case '\n':
39+
return i + 1;
40+
}
41+
}
42+
return 0;
43+
}
44+
}
45+
#endif

src/AWS.Speech.MEAI/VoiceAgent.cs

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
#if NET8_0_OR_GREATER
5+
using Amazon;
6+
using Amazon.BedrockRuntime;
7+
using Amazon.Polly;
8+
using Amazon.Runtime;
9+
using Amazon.TranscribeStreaming;
10+
using Microsoft.Extensions.AI;
11+
using System;
12+
using System.Collections.Generic;
13+
using System.Diagnostics.CodeAnalysis;
14+
using System.IO;
15+
using System.Threading;
16+
using System.Threading.Tasks;
17+
18+
namespace AWS.Speech.MEAI;
19+
20+
/// <summary>
21+
/// Composes an <see cref="ISpeechToTextClient"/>, an <see cref="IChatClient"/>, and an
22+
/// <see cref="ITextToSpeechClient"/> into a full-duplex voice loop that emits one ordered stream of
23+
/// <see cref="VoiceAgentUpdate"/>s.
24+
/// </summary>
25+
/// <remarks>
26+
/// The default backend is <see cref="VoiceAgentBackend.Pipeline"/>: Amazon Transcribe streaming for
27+
/// STT, Amazon Bedrock via <c>AWS.Bedrock.MEAI</c> for reasoning, and Amazon Polly for TTS. Call
28+
/// <see cref="Create"/> for the one-line default-chain factory, or use the constructor to compose any
29+
/// MEAI clients you already have. Barge-in, the Nova Sonic backend swap, the <c>AsIRealtimeClient()</c>
30+
/// adapter, and DI registration are wired in a later phase.
31+
/// </remarks>
32+
[Experimental("MEAI001")]
33+
public sealed class VoiceAgent : IAsyncDisposable
34+
{
35+
private readonly ISpeechToTextClient _stt;
36+
private readonly IChatClient _chat;
37+
private readonly ITextToSpeechClient _tts;
38+
private readonly VoiceAgentOptions _options;
39+
private readonly List<IDisposable> _ownedResources;
40+
private int _disposed;
41+
42+
/// <summary>Initializes a provider-neutral <see cref="VoiceAgent"/> around any MEAI clients.</summary>
43+
/// <exception cref="ArgumentNullException">A client is <see langword="null"/>.</exception>
44+
public VoiceAgent(ISpeechToTextClient stt, IChatClient chat, ITextToSpeechClient tts, VoiceAgentOptions? options = null)
45+
: this(stt, chat, tts, options ?? new VoiceAgentOptions(), ownedResources: null)
46+
{
47+
}
48+
49+
private VoiceAgent(ISpeechToTextClient stt, IChatClient chat, ITextToSpeechClient tts,
50+
VoiceAgentOptions options, List<IDisposable>? ownedResources)
51+
{
52+
_stt = stt ?? throw new ArgumentNullException(nameof(stt));
53+
_chat = chat ?? throw new ArgumentNullException(nameof(chat));
54+
_tts = tts ?? throw new ArgumentNullException(nameof(tts));
55+
_options = options;
56+
_ownedResources = ownedResources ?? new List<IDisposable>();
57+
}
58+
59+
/// <summary>Creates a <see cref="VoiceAgent"/> with the default AWS credential and region chains.</summary>
60+
/// <remarks>
61+
/// Constructs an Amazon Transcribe streaming client, an Amazon Bedrock runtime client (adapted via
62+
/// <c>AmazonBedrockRuntimeExtensions.AsIChatClient</c>), and an Amazon Polly client, honoring
63+
/// <see cref="VoiceAgentOptions.Credentials"/> and <see cref="VoiceAgentOptions.Region"/> when set.
64+
/// A pre-built client on the options object wins over the default AWS client for that leg. The
65+
/// returned agent owns any clients it constructed and disposes them on <see cref="DisposeAsync"/>.
66+
/// </remarks>
67+
/// <exception cref="NotSupportedException">The requested backend is not yet available in this preview.</exception>
68+
public static VoiceAgent Create(Action<VoiceAgentOptions>? configure = null)
69+
{
70+
var options = new VoiceAgentOptions();
71+
configure?.Invoke(options);
72+
73+
if (options.Backend != VoiceAgentBackend.Pipeline)
74+
{
75+
throw new NotSupportedException(
76+
$"The {options.Backend} backend is not available yet in this preview. Use VoiceAgentBackend.Pipeline.");
77+
}
78+
79+
var owned = new List<IDisposable>();
80+
81+
var stt = options.SpeechToTextClient;
82+
if (stt is null)
83+
{
84+
var transcribe = CreateTranscribeClient(options.Credentials, options.Region);
85+
owned.Add(transcribe);
86+
stt = transcribe.AsISpeechToTextClient(options.Language, options.InputSampleRateHertz);
87+
}
88+
89+
var chat = options.ChatClient;
90+
if (chat is null)
91+
{
92+
var bedrock = CreateBedrockClient(options.Credentials, options.Region);
93+
owned.Add(bedrock);
94+
chat = bedrock.AsIChatClient(options.ModelId);
95+
}
96+
97+
var tts = options.TextToSpeechClient;
98+
if (tts is null)
99+
{
100+
var polly = CreatePollyClient(options.Credentials, options.Region);
101+
owned.Add(polly);
102+
tts = polly.AsITextToSpeechClient(options.Voice, Engine.Neural, options.OutputSampleRateHertz);
103+
}
104+
105+
return new VoiceAgent(stt, chat, tts, options, owned);
106+
}
107+
108+
/// <summary>Runs the voice loop over the caller's microphone PCM stream.</summary>
109+
/// <param name="microphonePcm">
110+
/// Input audio at <see cref="VoiceAgentOptions.InputSampleRateHertz"/>, 16-bit signed
111+
/// little-endian mono PCM. The agent never disposes this stream.
112+
/// </param>
113+
/// <param name="cancellationToken">Stops the loop; the returned enumerable then completes.</param>
114+
/// <returns>One ordered stream of <see cref="VoiceAgentUpdate"/>s.</returns>
115+
/// <exception cref="ArgumentNullException"><paramref name="microphonePcm"/> is <see langword="null"/>.</exception>
116+
/// <exception cref="ObjectDisposedException">The agent has been disposed.</exception>
117+
public IAsyncEnumerable<VoiceAgentUpdate> RunAsync(Stream microphonePcm, CancellationToken cancellationToken = default)
118+
{
119+
if (microphonePcm is null) throw new ArgumentNullException(nameof(microphonePcm));
120+
ThrowIfDisposed();
121+
122+
return VoiceAgentPipeline.RunAsync(_stt, _chat, _tts, _options, microphonePcm, cancellationToken);
123+
}
124+
125+
/// <summary>Returns the underlying MEAI client for the requested service type, or <see langword="null"/>.</summary>
126+
public object? GetService(System.Type serviceType, object? serviceKey = null)
127+
{
128+
if (serviceType is null) throw new ArgumentNullException(nameof(serviceType));
129+
if (serviceKey is not null) return null;
130+
131+
if (serviceType == typeof(ISpeechToTextClient)) return _stt;
132+
if (serviceType == typeof(IChatClient)) return _chat;
133+
if (serviceType == typeof(ITextToSpeechClient)) return _tts;
134+
return serviceType.IsInstanceOfType(this) ? this : null;
135+
}
136+
137+
/// <inheritdoc/>
138+
public ValueTask DisposeAsync()
139+
{
140+
if (Interlocked.Exchange(ref _disposed, 1) != 0) return default;
141+
142+
Exception? first = null;
143+
foreach (var resource in _ownedResources)
144+
{
145+
try { resource.Dispose(); }
146+
catch (Exception ex) { first ??= ex; }
147+
}
148+
_ownedResources.Clear();
149+
150+
if (first is not null)
151+
{
152+
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(first).Throw();
153+
}
154+
return default;
155+
}
156+
157+
private void ThrowIfDisposed()
158+
{
159+
if (Volatile.Read(ref _disposed) != 0)
160+
{
161+
throw new ObjectDisposedException(nameof(VoiceAgent));
162+
}
163+
}
164+
165+
private static AmazonTranscribeStreamingClient CreateTranscribeClient(AWSCredentials? credentials, RegionEndpoint? region) =>
166+
(credentials, region) switch
167+
{
168+
(null, null) => new AmazonTranscribeStreamingClient(),
169+
(null, _) => new AmazonTranscribeStreamingClient(region),
170+
(_, null) => new AmazonTranscribeStreamingClient(credentials),
171+
_ => new AmazonTranscribeStreamingClient(credentials, region),
172+
};
173+
174+
private static AmazonBedrockRuntimeClient CreateBedrockClient(AWSCredentials? credentials, RegionEndpoint? region) =>
175+
(credentials, region) switch
176+
{
177+
(null, null) => new AmazonBedrockRuntimeClient(),
178+
(null, _) => new AmazonBedrockRuntimeClient(region),
179+
(_, null) => new AmazonBedrockRuntimeClient(credentials),
180+
_ => new AmazonBedrockRuntimeClient(credentials, region),
181+
};
182+
183+
private static AmazonPollyClient CreatePollyClient(AWSCredentials? credentials, RegionEndpoint? region) =>
184+
(credentials, region) switch
185+
{
186+
(null, null) => new AmazonPollyClient(),
187+
(null, _) => new AmazonPollyClient(region),
188+
(_, null) => new AmazonPollyClient(credentials),
189+
_ => new AmazonPollyClient(credentials, region),
190+
};
191+
}
192+
#endif
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
#if NET8_0_OR_GREATER
5+
using System.Diagnostics.CodeAnalysis;
6+
7+
namespace AWS.Speech.MEAI;
8+
9+
/// <summary>Selects which backend drives a <see cref="VoiceAgent"/>.</summary>
10+
[Experimental("MEAI001")]
11+
public enum VoiceAgentBackend
12+
{
13+
/// <summary>The STT + <c>IChatClient</c> + TTS composition (default).</summary>
14+
Pipeline,
15+
16+
/// <summary>Amazon Bedrock Nova Sonic single-model speech-to-speech (wired in a later phase).</summary>
17+
NovaSonic,
18+
}
19+
#endif
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
#if NET8_0_OR_GREATER
5+
using Amazon;
6+
using Amazon.Polly;
7+
using Amazon.Runtime;
8+
using Microsoft.Extensions.AI;
9+
using System.Collections.Generic;
10+
using System.Diagnostics.CodeAnalysis;
11+
12+
namespace AWS.Speech.MEAI;
13+
14+
/// <summary>Configuration for a <see cref="VoiceAgent"/>.</summary>
15+
[Experimental("MEAI001")]
16+
public sealed class VoiceAgentOptions
17+
{
18+
/// <summary>The Amazon Bedrock model ID for the reasoning step (for example, an Anthropic Claude model).</summary>
19+
public string? ModelId { get; set; }
20+
21+
/// <summary>The Amazon Polly voice to synthesize with. Defaults to <see cref="VoiceId.Matthew"/>.</summary>
22+
public VoiceId Voice { get; set; } = VoiceId.Matthew;
23+
24+
/// <summary>The system prompt, applied at the start of the conversation history.</summary>
25+
public string? Instructions { get; set; }
26+
27+
/// <summary>The Amazon Transcribe language code. Defaults to <c>en-US</c>.</summary>
28+
public string Language { get; set; } = "en-US";
29+
30+
/// <summary>Input PCM sample rate in hertz. Defaults to 16000.</summary>
31+
public int InputSampleRateHertz { get; set; } = 16000;
32+
33+
/// <summary>Output PCM sample rate in hertz. Defaults to 16000; Amazon Polly PCM supports only 8000 or 16000.</summary>
34+
public int OutputSampleRateHertz { get; set; } = 16000;
35+
36+
/// <summary>Enables barge-in (wired in a later phase).</summary>
37+
public bool EnableBargeIn { get; set; } = true;
38+
39+
/// <summary>End-of-utterance debounce (wired with the barge-in tuning pass in a later phase).</summary>
40+
public int EndOfUtteranceSilenceMs { get; set; } = 700;
41+
42+
/// <summary>Tools passed through to the reasoning <c>IChatClient</c> via <see cref="ChatOptions.Tools"/>.</summary>
43+
public IList<AITool>? Tools { get; set; }
44+
45+
/// <summary>The AWS region for the constructed AWS clients. <see langword="null"/> uses the default region chain.</summary>
46+
public RegionEndpoint? Region { get; set; }
47+
48+
/// <summary>The AWS credentials for the constructed AWS clients. <see langword="null"/> uses the default credential chain.</summary>
49+
public AWSCredentials? Credentials { get; set; }
50+
51+
/// <summary>Which backend drives the agent. Defaults to <see cref="VoiceAgentBackend.Pipeline"/>.</summary>
52+
public VoiceAgentBackend Backend { get; set; } = VoiceAgentBackend.Pipeline;
53+
54+
/// <summary>Pre-built speech-to-text client. When set, wins over the default AWS client chain.</summary>
55+
public ISpeechToTextClient? SpeechToTextClient { get; set; }
56+
57+
/// <summary>Pre-built chat client. When set, wins over the default AWS client chain.</summary>
58+
public IChatClient? ChatClient { get; set; }
59+
60+
/// <summary>Pre-built text-to-speech client. When set, wins over the default AWS client chain.</summary>
61+
public ITextToSpeechClient? TextToSpeechClient { get; set; }
62+
}
63+
#endif

0 commit comments

Comments
 (0)