Playback Reference Stream for Chromium Web Applications

[huozcan-ms]

Playback Reference Stream for Chromium Web Applications

---

1. Introduction

Web applications increasingly depend on advanced, real-time audio processing: conversational AI and voice agents, accessibility experiences, meeting and calling clients, transcription, and custom machine-learning (ML) audio enhancement. All of these need to separate what the user is saying from what the device is playing back.

Browsers today expose microphone input through getUserMedia(), but they do not expose an independent, application-consumable signal of what the browser (or the system) is playing out of the speakers. That signal is called a playback reference. Without it:

• Applications cannot perform their own deterministic echo cancellation. They are limited to the browser's built-in echo canceller, which they cannot see into, tune, or replace.
• Applications cannot reliably distinguish near-end user speech from system-generated audio (media, other participants, a screen reader, or a narrator).

The browser's built-in WebRTC echo cancellation is helpful for the common human-to-human calling case, but it is insufficient for these newer scenarios because:

• It does not expose the underlying microphone or reference signals to the application.
• It does not allow an application to implement its own deterministic, model-based cancellation.
• It is not designed as a full-duplex canceller and has known quality limits under difficult conditions.
• Its behavior is bundled: echo cancellation is coupled with automatic gain control (AGC) and noise suppression (NS), and these cannot be enabled or disabled independently.

This document proposes a constrained playback reference stream: a purpose-scoped primitive that lets an application obtain a time-aligned reference of playback audio so it can run its own deterministic and ML-based processing on the raw microphone signal.

---

2. Goals

• Provide applications with a time-aligned playback reference suitable for echo cancellation and related signal processing.
• Enable deterministic, application-defined echo cancellation and speech separation on top of the raw microphone signal.
• Improve speech quality, transcription accuracy, interruption handling, and overall user experience for browser and WebView-hosted RTC and voice-agent applications.
• Give applications access to the playback reference without requiring screen sharing.
• Keep the capability narrowly scoped, mediated by the browser, and aligned with user expectations.

---

3. Non-goals

• This is not unrestricted system-audio capture. The capability is scoped to a reference for processing, not a general recording tap.
• This is not a replacement for the browser's built-in echo cancellation. Applications that do not need custom processing can continue to rely on the built-in path.
• This does not bypass user awareness or browser mediation. The user remains in control, and the browser remains the arbiter of access.

---

4. Motivating use cases

Scenario 1: Browser-based real-time communication

A user is on a browser-based call. Local playback (the far-end participants, shared media, notification sounds) leaks into the microphone capture and is heard as echo by the other side. Application-level cancellation cannot reliably remove this leakage without a reference for exactly what was played out.

Scenario 2: Accessibility (screen reader and narrator)

A user relies on a screen reader or the OS narrator while using a voice experience. The narrator audio plays out of the device and is picked up by the microphone. Because the narrator is a system-level source, the application has no reference for it and cannot cancel it. The result is that assistive audio contaminates the captured speech and, in agent scenarios, is transcribed and acted on as if it were the user. This is a real accessibility regression, not a theoretical one.

Scenario 3: Advanced application-level and ML-based processing

Modern voice agents and communication clients run their own ML audio pipelines (custom echo cancellation, voice isolation, speaker separation, noise suppression). These models require two inputs: the raw microphone signal and a clean playback reference. Today the application can obtain neither in a form that is deterministic and time-aligned, so it is forced to run on top of browser-processed audio, which degrades the model's input and its output.

---

5. Why existing APIs are insufficient

• getUserMedia() provides the microphone input, optionally with browser echo cancellation, but provides no playback reference. When echo cancellation is enabled, the signal is already altered; when it is disabled, echo is present but there is still no reference with which to remove it.
• getDisplayMedia() can expose system audio, but only as a side effect of screen sharing. It couples audio access to visual capture, adds a prominent capture-sharing UX, and is not suitable for audio-only, always-on communication scenarios.

Neither API gives an application a purpose-built, time-aligned playback reference for signal processing.

---

6. Why cascading browser processing with an application pipeline is not a solution

A natural objection is: "Let the browser or system cancel the loopback first, and let the application run its ML processing on top. Why is a reference stream needed at all?" This section answers that objection directly. It is the crux of the proposal.

The short version: cascading two adaptive, nonlinear audio pipelines violates the assumptions each pipeline was designed under. The browser stage transforms the signal in ways the application stage cannot see, invert, or compensate for, and the application stage can no longer model the echo it is supposed to remove. The problems below are not tuning issues. They are structural.

6.1 The browser cannot give you "loopback cancellation only"

In Chromium, echo cancellation is bundled with automatic gain control and noise suppression, and these cannot be toggled independently. So "let the browser cancel the loopback, then the app processes on top" is not actually available. In practice the application always inherits AEC and AGC and NS, not a clean loopback cancel. Every failure mode below follows from that fact.

6.2 Signal-domain corruption: the application pipeline receives damaged input

• Nonlinear, time-varying echo removal breaks the app's echo model. The browser echo canceller is nonlinear and adapts over time. It removes most of the linear echo and leaves a nonlinearly distorted residual. An application canceller that assumes a linear echo path can no longer model that residual, and with no reference signal it cannot recover it.
• Unknown gain from AGC cannot be inverted. The bundled AGC applies time-varying gain to the microphone signal before the application sees it. Any reference the application does have no longer matches the scaled near-end signal, and the applied gain is neither exposed nor logged, so it cannot be undone.
• Noise suppression pushes ML models out of distribution. The bundled NS reshapes spectral content and the noise floor. ML models trained on natural microphone input then receive out-of-distribution audio, which degrades separation, voice isolation, and transcription accuracy even when each stage performs acceptably in isolation.

6.3 Adaptation and timing failures

• Double convergence at call start. Two adaptive filters converge at the same time when a call begins. This lengthens the window in which echo leaks through, which directly harms first-audio-response latency, a metric that materially differentiates leading voice agents.
• Delay-estimation breakage. The browser stage introduces a variable processing and buffering delay. The application canceller's delay estimator now sees a shifting offset between reference and capture, causing misalignment and intermittent echo bursts. A first-class reference stream must carry a timing and clock contract precisely to avoid this.
• Playback glitches read as echo-path changes. Buffer underruns or dropped playback frames create discontinuities that the application canceller interprets as a change in the echo path, forcing costly re-convergence and audible artifacts.

6.4 Double-talk and duplex behavior

• Compounded half-duplex behavior. The browser canceller is not designed as a full-duplex canceller. Placing a second suppressor on top compounds the tendency toward half-duplex behavior, which breaks barge-in and interruption. Interruption handling is the highest-impact live-site issue for voice agents, so this is not a corner case.
• Double-talk detector failure. The browser stage alters the energy and spectral statistics of the signal. The application's double-talk detector, tuned on raw microphone input, then misfires and clips near-end speech during simultaneous talk.

6.5 System integrity and testability

• Compounded compute and thermal cost. Running two full audio-processing stacks compounds overhead. Web-based processing already carries roughly 15 to 20 percent additional compute relative to native for comparable work, and doubling the pipeline risks dropouts and glitches on lower-end devices.
• Multi-source attribution is impossible with a single mixed capture. When several sources play at once (other call participants, media, a narrator), a single mixed capture path gives neither stage a correct per-source reference. This is a plausible root cause for observed cases where audio from a separate application leaks into a voice agent even with system-wide cancellation enabled. A labeled or per-source reference resolves it; cascading cannot.
• Non-reproducibility and eval gaps. The double-processed configuration is not a tested or expected configuration and cannot be evaluated deterministically. Regressions ship silently because the production pipeline does not match what any evaluation harness measures. This directly undercuts the determinism that these applications require.

6.6 Summary of the argument

Cascading fails because the browser stage is nonlinear, time-varying, bundled (AEC plus AGC plus NS), and opaque. The application stage cannot see what was done, cannot invert it, and can no longer model the echo it must remove. The correct primitive is a deterministic, time-aligned playback reference that the application can subtract itself, not a second pipeline stacked on an uncontrollable first stage.

---

7. Proposed approach

Introduce a browser-mediated playback reference stream: a MediaStream carrying the audio the browser (and, where policy permits, the system) is playing out, delivered in a form suitable for echo cancellation and related processing.

Key properties:

• Time-aligned with capture. The reference carries timestamps in the same clock domain as the microphone capture, so the application can align reference and capture without guessing the delay.
• Defined fidelity. The reference represents the rendered playback signal at a specified point in the pipeline (see Section 8), with a defined sample rate, channel layout, and gain state.
• Purpose-scoped. It is intended for local, real-time processing. It is gated by permission and browser mediation and is not a general capture or recording capability.
• Lifecycle-bound to the microphone. Access to the reference is tied to an active microphone capture and is revoked when the microphone is muted or the capture ends.

7.1 API shape (conceptual, non-normative)

The description below communicates the shape of the capability, not final naming or a normative interface. The final surface, whether a new method or an extension of an existing capture primitive, is an open question (Section 10).

At a high level, the flow has three steps:

1. Acquire raw microphone capture. The application requests microphone input with the built-in processing turned off, that is, echo cancellation, automatic gain control, and noise suppression disabled, so it receives an unmodified near-end signal.
2. Acquire the playback reference. The application requests a playback reference that is time-aligned to the microphone capture clock. The request specifies the properties the application depends on: the tap point or fidelity (for example, the rendered output versus a mixed signal), sample rate, and channel count.
3. Process locally. The application feeds the raw microphone capture and the time-aligned reference into its own canceller and downstream ML pipeline. Muting the microphone terminates the reference (see Section 9, Security and privacy).

The essential contract is: raw microphone in one hand, time-aligned playback reference in the other, application-owned processing in between.

---

8. Detailed design considerations

• Reference fidelity and tap point. The specification must define where in the audio path the reference is taken. A post-mix, resampled, or post-volume signal models a different signal than what reached the speaker and undermines cancellation. The reference should be defined relative to the rendered output, with sample rate, channel layout, and pre- or post-volume state specified.
• Clock domain and timestamps. Reference and capture must share a clock, or the reference must carry timestamps convertible to the capture clock, so the application can align them without drift.
• Discontinuity signaling. Playback gaps, underruns, and device changes should be signaled so the application can handle them rather than misinterpret them as echo-path changes.
• Source scope. An initial version may expose a single mixed same-origin or same-instance playback reference.
• Interaction with built-in processing. The specification must define how the reference behaves when the application also requests browser echo cancellation. The intended primary mode is: application requests raw capture (built-in processing off) plus reference, and does its own cancellation.

---

9. Security and privacy considerations

This capability exposes information about what the device is playing back, so it must be scoped and mediated carefully. The following is a self-review in the style of the W3C Security and Privacy Questionnaire.

• What information is exposed? A processed reference of playback audio, intended for local signal processing. It is not a general recording of arbitrary system audio.
• Is it scoped to the minimum necessary? Yes. The default and primary intent is a same-origin or same-application-instance rendered reference, not cross-origin or arbitrary system capture.
• How does the user stay aware and in control? Access is gated by permission and browser mediation. The browser remains the arbiter and can constrain stream format, quality, and lifecycle.
• What is the lifecycle guarantee? Access to the reference is bound to an active microphone capture. Muting the microphone immediately terminates the reference signal. Ending capture ends reference access.
• Does it create a new exposure surface beyond what exists today? Microphone access without echo cancellation already indirectly exposes played-back audio through acoustic coupling. This proposal seeks a programmatic, controlled, and higher-quality version of a capability that acoustically already leaks, while keeping the user and browser in control.
• Abuse and fingerprinting considerations. The reference should not become a side channel for cross-origin content or for silent, always-on system capture. Constraints on source scope, permission, and lifecycle are intended to prevent this. This area needs further design work and standards input.
• Permission friction tradeoff. Overly heavy permission prompts could make real-time, always-on use cases impractical. The permission model, and how it composes with the existing microphone permission, is a core open design question.

---

10. Considered alternatives

1. Continue relying on built-in browser echo cancellation. Helps the common case, but does not expose reference signals and does not enable application-defined deterministic processing. It cannot serve the accessibility and ML scenarios above.
2. Use getDisplayMedia() with system audio. Can expose system audio in screen-sharing scenarios, but couples audio to visual capture, adds capture-sharing UX friction, and is unsuitable for audio-only, always-on communication.
3. Cascade browser processing with an application pipeline. Structurally unsound: the application stage cannot see, invert, or compensate for the bundled, nonlinear, time-varying browser stage, and cannot model the echo it must remove.
4. Do nothing. Browser-based communication and conversational-AI experiences remain constrained relative to native platforms whenever playback and microphone capture interact.

---

11. Open questions and future work

• Should this be a new API or an extension of an existing media-capture primitive?
• What permission and user-awareness model is appropriate, and how does it compose with the microphone permission?
• Is a single mixed playback reference sufficient for a first version, or is per-source or labeled reference needed early?
• Should browsers restrict use of the reference to processing scenarios, and if so, how is that enforced?
• How should the capability interact with the built-in browser audio processing chain, including the bundling of AEC with AGC and NS?
• Should there be a path to decouple AEC from AGC and NS in the built-in chain, independent of this proposal, since that bundling is a root cause of the cascading problems in Section 6?