Audio does not work properly on the Quest

[sebjf]

Echo cancellation does not appear to work in the Oculus Quest; when creating an audio chat session other users can hear themselves through a remote peer's Quest, and two Quests will generate a feedback loop.

While on the surface an AEC issue, this is really to do with how Android handles audio, and what assumptions WebRtc makes about how the app should work.

Relevant threads:

• https://stackoverflow.com/questions/35350320/android-acousticechocanceler-doesnt-seem-to-work-on-most-devices

[sebjf]

Echo cancellation at the highest level is controlled by the AudioOptions (WebRtcVoiceEngine::ApplyOptions()). If echo cancellation is on (it should be by default), then the voice engine will check for audio device echo cancellation support, and if found disable AEC as if the AudioOptions had opted against it.

  if (options.echo_cancellation) {
    // Check if platform supports built-in EC. Currently only supported on
    // Android and in combination with Java based audio layer.
    // TODO(henrika): investigate possibility to support built-in EC also
    // in combination with Open SL ES audio.
    const bool built_in_aec = adm()->BuiltInAECIsAvailable();
    if (built_in_aec) {
      // Built-in EC exists on this device. Enable/Disable it according to the
      // echo_cancellation audio option.
      const bool enable_built_in_aec = *options.echo_cancellation;
      if (adm()->EnableBuiltInAEC(enable_built_in_aec) == 0 &&
          enable_built_in_aec) {
        // Disable internal software EC if built-in EC is enabled,
        // i.e., replace the software EC with the built-in EC.
        options.echo_cancellation = false;
        RTC_LOG(LS_INFO)
            << "Disabling EC since built-in EC will be used instead";
      }
    }
  }

[sebjf]

It appears that this is a known problem on Android. A solution is to disable the Android AEC and rely on WebRTCs implementation. We can trivially confirm that the in-built AEC is taking precedence using logcat.

2021-05-07 17:23:08.322 1863-14738/? I/libjingle: [164040:878] [14738] (webrtcvoiceengine.cc:722): Disabling EC since built-in EC will be used instead

Based on the SO question, we need to use the WebRtcAudioUtils class and JavaAudioDeviceModule class. It is not clear how these Java classes interact with the native code. Does WebRtc start a java vm on Android or the other way round?

Note that the Audio Utils has a list of blacklisted devices, but it is empty and the comments say...

It is recommended to maintain a list of blacklisted models outside
      // this package and instead call setWebRtcBasedAcousticEchoCanceler(true)
      // from the client for devices where the built-in AEC shall be disabled.

e.g.

WebRtcAudioUtils.setWebRtcBasedAcousticEchoCanceler(true);
WebRtcAudioUtils.setWebRtcBasedAutomaticGainControl(true);
WebRtcAudioUtils.setWebRtcBasedNoiseSuppressor(true);

A commentor on SO also said that setUseHardwareAcousticEchoCanceler(false) and setUseHardwareNoiseSuppressor(false) must also be set (on the Java ADM). But is this true, and if so where to get the ADM ref in C# (or even C)?

setWebRtcBasedAcousticEchoCanceler sets a static member of the Audio Utils class (useWebRtcBasedAcousticEchoCanceler). This in turn is checked by public static boolean WebRtcAudioEffects::canUseAcousticEchoCanceler().

The class WebRtcAudioManager checks this automatically when it initialises its default parameters. The WebRtcAudioManager manages the underlying Android audio device (as per its comments). Perhaps then we need to simply ensure the static calls to the WebRtcAudioUtils complete before an instance of this is created?

It appears as though the recording stream is managed by a WebRtcAudioRecord instance. This, after creating the Android object, will call WebRtcAudioEffects::enable(int audioSession) passing in the object. It is here that the echo cancellation is added (or not). This class (the one in voiceengine) however is deprecated; it is used by the sdk/api's LegacyAudioDeviceModule, which has the comment

It will return a null native AudioDeviceModule pointer, leading to an internal object being
created inside WebRTC that is controlled by static calls to the classes under the voiceengine
package. Please use the new JavaAudioDeviceModule instead of this class.

However, there is another WebRtcAudioRecord in the sdk/android. This seems to be the one that the audio device actually creates (via AudioRecordWrapper.cpp according to logcat?).
Regarding the JavaAudioDeviceModule::setUseHardwareAcousticEchoCanceler, this class is in the api directory, and sets a local flag. The flag is passed to the SDK WebRtcAudioRecord constructor, that takes a number of arguments.
If the arguments conflict (i.e. the passed instruction is to use HW AEC, an illegal argument exception is thrown).
However, the value is initialised based on the isBuiltInAcousticEchoCancelerSupported() method, which checks with WebRtcAudioEffects. The method it checks though is isAcousticEchoCancelerSupported, which calls isEffectTypeAvailable, which ignores any manual flags and simply checks with the cached effects.

The simplest resolution then is likely to be to modify the WebRtcAudioEffects::getAvailableEffects() method to return false for HW AEC (much like we do for Windows).
Alternatively, we could turn it off in the JavaAudioDeviceModule, at the point the WebRtcAudioRecord object is created.

[sebjf]

An additional problem is that the WebRtc audio streams do not play nicely with Unity.

Android has the concept of audio stream routing, where streams of different types can be routed to different endpoints (e.g. earpiece, speakers) and possibly take precedence over each other (e.g. voice over music).

The default Java Audio Device in WebRtc assumes the app is acting as a call app and that the device is in COMMUNICATION_MODE. This is incompatible with Unity. One consequence is that the volume controls disappear when in COMMUNICATION_MODE. Another is that the streams from Unity audio sources go silent when a voice channel is established.

Relevant threads:

• https://bugs.chromium.org/p/webrtc/issues/detail?id=7313
• https://groups.google.com/g/discuss-webrtc/c/Pqag6R7QV2c

Another consideration is how spatialised audio from Unity will interact with AEC.

On Android, apps cannot access the mixer output, and so it is likely that hardware AEC will be the only option on the Quest. Therefore we should fix it rather than disable it as above.

Branch dev-androidecho contains an echo/mixed example: a cube in the room that when activated will play an audio file locally. This can be used to verify that local & remote audio is being mixed, and that the local audio source is being cancelled correctly.

To confirm, this issue does not present on Windows.