Skip to content

feat: add ConversationAudioObserver for mic and agent audio - #215

Merged
renal128 merged 2 commits into
nextfrom
audio-observers
Aug 4, 2026
Merged

feat: add ConversationAudioObserver for mic and agent audio#215
renal128 merged 2 commits into
nextfrom
audio-observers

Conversation

@renal128

@renal128 renal128 commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

What

  • Introduce ConversationAudioObserver for observing decoded PCM from agent output or the local microphone.
  • Add durable observer APIs to ConversationClient:
    • addAgentAudioObserver / removeAgentAudioObserver
    • addMicAudioObserver / removeMicAudioObserver
  • Keep LiveKit AudioRenderer / AudioTrack fully internal behind AudioObserverRegistry.
  • Reattach client observers for each new session and on mid-session track changes.

Why

Apps need raw audio for visualization, recording, or custom analysis without:

  • LiveKit types in the public SDK surface
  • main-actor audio processing
  • SwiftUI churn from high-rate @Published updates

Usage

final class Recorder: ConversationAudioObserver, @unchecked Sendable {
    func didReceive(_ buffer: AVAudioPCMBuffer) {
        // Copy samples synchronously, then process them away from this callback.
    }
}

let client = ConversationClient()
let recorder = Recorder()

client.addMicAudioObserver(recorder)
client.addAgentAudioObserver(recorder)

Behavior

  • didReceive(_:) receives a borrowed, read-only buffer on the audio callback path.
  • Consumers own any buffering, copying, analysis, or visualization work outside the callback.
  • Cross-session durability is owned by ConversationClient; Conversation itself is single-use.
  • Use separate observers when you need independent state per audio stream.

Testing

  • swift test --filter 'AudioObserverRegistryTests|ConversationAudioObserverTests|ConversationClientTests'

Note

Medium Risk
New public audio callback API runs on the real-time audio path and changes teardown ordering around WebRTC disconnect; behavior is covered by unit tests but misuse in observers could affect audio performance.

Overview
Adds a public ConversationAudioObserver protocol so apps can tap decoded PCM from agent output and local mic for visualization, recording, or analysis without LiveKit types on the SDK surface.

ConversationClient exposes durable add/remove APIs for agent and mic observers; registrations are re-bound on each new session when bind runs. Conversation owns per-session AudioObserverRegistry instances that bridge observers to internal LiveKit AudioRenderer adapters, reattach on track swaps, and reset on teardown.

WebRTCConnectionManager surfaces onTracksChanged from LiveKit publish/subscribe/unpublish events and types tracks as AudioTrackProtocol internally. Session end / startup failure paths tear down observers before disconnect() and use isTearingDown so stale track callbacks cannot re-attach after cleanup.

Reviewed by Cursor Bugbot for commit ffd0362. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread Sources/ElevenLabs/Public/Conversation/Conversation.swift Outdated
Comment thread Sources/ElevenLabs/Internal/Audio/ConversationAudioLevelAnalyzer.swift Outdated
Comment thread Sources/ElevenLabs/Public/Conversation/ConversationAudioLevelMonitor.swift Outdated
Comment thread Sources/ElevenLabs/Public/Conversation/ConversationAudioLevelMonitor.swift Outdated
@renal128
renal128 force-pushed the audio-observers branch 2 times, most recently from 48cf638 to d1acd8d Compare July 20, 2026 17:07
@renal128
renal128 force-pushed the audio-observers branch 2 times, most recently from 8b49bd5 to ab9f9cd Compare July 20, 2026 22:32
@renal128
renal128 force-pushed the audio-observers branch 3 times, most recently from f0fb4ba to 51cd5ec Compare July 21, 2026 12:28
Comment thread Sources/ElevenLabs/Internal/Conversation/Conversation.swift
Comment thread Sources/ElevenLabs/Internal/Conversation/Conversation.swift
@renal128
renal128 force-pushed the audio-observers branch 3 times, most recently from 3e01a48 to 648be4b Compare July 21, 2026 22:44
Comment thread Sources/ElevenLabs/Internal/Conversation/Conversation.swift
@renal128
renal128 force-pushed the audio-observers branch 2 times, most recently from db18f3c to a772bc5 Compare July 22, 2026 13:21

@jacksonh jacksonh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very useful.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 27ed8e2. Configure here.

Comment thread Sources/ElevenLabs/Internal/Conversation/Conversation.swift
Base automatically changed from conversation-client to next August 4, 2026 18:38
renal128 and others added 2 commits August 4, 2026 19:38
Expose durable mic and agent audio observers with optional level analysis while keeping real-time callbacks off the main actor.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@renal128
renal128 merged commit c109e50 into next Aug 4, 2026
2 checks passed
@renal128
renal128 deleted the audio-observers branch August 4, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants