feat: drive the widget orb from mic and agent audio - #225
Merged
Conversation
3 tasks
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
renal128
force-pushed
the
widget-orb
branch
2 times, most recently
from
July 26, 2026 13:59
3abcb03 to
cc508d0
Compare
renal128
force-pushed
the
widget-orb
branch
2 times, most recently
from
July 26, 2026 16:53
97750a6 to
a727cf1
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b8a6979. Configure here.
Contributor
|
Only thing I'd say on this one is I'd add a demo video to the PR |
jacksonh
approved these changes
Aug 3, 2026
Port the procedural Metal orb from ElevenLabs components-swift (Apache-2.0), trimmed to the volume-driven view, and drive it from the SDK's mic and agent audio observers: a lock-protected RMS monitor per stream, sampled at 30 Hz so PCM-rate audio never re-renders SwiftUI. The mic button reuses the same level. Co-authored-by: Cursor <cursoragent@cursor.com>
Make mic input drive angular motion and agent output drive shape changes while hardening shader edges and Int16 sampling. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the placeholder orb with the Metal orb from components-swift (Apache-2.0), reacting to live audio.
Rendering/Orb.swift,Rendering/VisualizerAgentState.swift,Resources/OrbShader.metal: ported and trimmed to what the widget needs. The LiveKit dependency is gone, so the orb runs on plain audio levels.ConversationAudioLevelMonitor: aConversationAudioObserver(from feat: add ConversationAudioObserver for mic and agent audio #215) that keeps a normalized loudness level with fast attack and slow release, computed on the audio thread for both float32 and int16 buffers.OrbAudioLevels: samples the mic and agent monitors on a timer and publishes the two levels the orb renders from.Test plan
swift build,swift test(newConversationAudioLevelMonitorTests),swiftformat . --strictxcodebuildofExamples/DemoAppfor the iOS simulatororb-demo.mov
Stacked on #224.