Playback reference audio capture explainer - #1377
Playback reference audio capture explainer#1377Nishitha Burman Dey (nishitha-burman) wants to merge 9 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove link to issue that had explainer
Removed mention of echo cancellation from audio processing.
Clarified API options for playback reference access and adjusted language for consistency. Added details on permission model and potential playback scopes.
Clarify API shape descriptions for playback reference tracks.
Steven Becker (SteveBeckerMSFT)
left a comment
Steven Becker (SteveBeckerMSFT)
left a comment
There was a problem hiding this comment.
I got through most of the explainer, but not all of it.
| the device is playing back. | ||
|
|
||
| Browsers today expose microphone input through `getUserMedia()`, but they do | ||
| not expose an independent, application-consumable signal of what the browser |
There was a problem hiding this comment.
signal of what the browser or system is playing out of the speakers
Should we also be able to capture a specific application's audio? For the browser case, should we be able to capture a specific tab or alternatively the entire browser window?
Maybe this is clarified later.
| Browsers today expose microphone input through `getUserMedia()`, but they do | ||
| not expose an independent, application-consumable signal of what the browser | ||
| or system is playing out of the speakers. That signal is called a **playback | ||
| reference**. This document proposes a constrained playback reference: a |
There was a problem hiding this comment.
a purpose-scoped
How is this primitive purpose scoped? What does that mean?
| or system is playing out of the speakers. That signal is called a **playback | ||
| reference**. This document proposes a constrained playback reference: a | ||
| purpose-scoped primitive that lets an application obtain a time-aligned | ||
| reference of playback audio so it can run deterministic or ML-based processing |
There was a problem hiding this comment.
a time-aligned reference
What does this mean? What is the reference?
| reference**. This document proposes a constrained playback reference: a | ||
| purpose-scoped primitive that lets an application obtain a time-aligned | ||
| reference of playback audio so it can run deterministic or ML-based processing | ||
| on the raw microphone signal. |
There was a problem hiding this comment.
For the microphone case, we should be more specific this is about echo cancellation.
What other scenarios and use cases, that don't involve microphones, does this capability unlock? A few come to mind:
- You could record a specific application, a specific browser tab's or the entire system's audio using something like MediaRecorder.
- You could also audio processing of the above using Web Audio.
| microphone capture. Ending the associated microphone capture ends the | ||
| playback reference. | ||
| - **Does it create a new exposure surface beyond what exists today?** | ||
| Microphone access without echo cancellation already indirectly exposes |
There was a problem hiding this comment.
Microphone access without echo cancellation already indirectly exposes
played-back audio through acoustic coupling.
This is not always true. The user may be using headphones or a headset so their audio does not leak into the microphone.
| - **How does the user stay aware and in control?** Access requires a | ||
| user-initiated request, distinct playback-reference permission, a persistent | ||
| active-capture indicator, and a way to stop reference access while preserving | ||
| microphone capture. |
There was a problem hiding this comment.
Access requires a user-initiated request, distinct playback-reference permission, a persistent
active-capture indicator, and a way to stop reference access while preserving microphone capture.
In our example above, should we sketch out what might look like using some screenshots?
I think this new permission will generate a lot of interest and discussion.
|
|
||
| The microphone remains available if the user declines playback-reference | ||
| access. This lets the voice experience continue with microphone-only capture | ||
| and, where appropriate, browser-provided audio processing. |
There was a problem hiding this comment.
if the user declines playback-reference
Would the microphone stream fall back to browser based echo cancellation?
| }, | ||
| playback: { | ||
| scope: "system", | ||
| optional: true, |
There was a problem hiding this comment.
optional: true
scope should use a constraint like getUserMedia() and getDisplayMedia().
scope: "system" is optional.
scope: { required: "system" } would not be optional.
| ```js | ||
| if (!playbackReferenceTrack) { | ||
| await microphoneTrack.applyConstraints({ | ||
| echoCancellation: true, |
There was a problem hiding this comment.
nit: You need to specify echoCancellation: "all" to cancel system audio
|
|
||
| ```js | ||
| const result = | ||
| await navigator.mediaDevices.getPlaybackReference({ |
There was a problem hiding this comment.
Like we discussed, apps like Teams and Meet will want to bundle their camera and microphone access with a signal getUserMedia() call, which also informs the permission UI. getPlaybackReference() lacks camera.
| same capabilities. The microphone is required. Playback-reference access is | ||
| optional and can be shown as a preselected checkbox: | ||
|
|
||
| <img src="playback-reference-permission.png" |
There was a problem hiding this comment.
We also might want to think about a permission experience involving the new <camera> and <microphone> elements. See:
https://chromestatus.com/feature/5153829504024576?gate=6067694366490624
2 participants
Explainer for proposal to allow web developers to capture playback reference audio without screen share.