(Question) Implementing Spatial Audio

[alex-schroedsen]

In the last month I have been working on finding a process for implementing spatial audio (5.1.2 channels) for both DirectSound3D and OpenAL APIs. My next goal is to make this work for XAudio2 games. The current setup/signal path I have is:

1. [Game using OpenAL Soft outputs 3rd order ambisonic signal to JACK backend on Windows 8.1/10]
2. [ASIO enabled VST host hooked into JACKRouter ASIO driver does ambisonic decoding using IEM ALLRAD ambisonic Decoder]
3. [JACK outputs decoded 5.1.2 audio to an ASIO enabled audio interface]

Software used:

1. JACK2 Windows x86-64 https://jackaudio.org/faq/jack_on_windows.html
2. IEM Ambisonics Suite https://plugins.iem.at/
3. Blue Cat's Patchwork https://www.bluecataudio.com/Products/Product_PatchWork/ (free alternative vsthost untested https://www.hermannseib.com/english/vsthost.htm)

I am planning to try using the old XACT libraries from WINE 4.2 to adapt XAudio2 into OpenAL Soft, but I'm confident that your implementation will offer far better compatibility. My questions are:

1. What types of speaker configurations can FAudio mix for?
2. Does FAudio mix in ambisonic audio? Can it output ambisonic audio?
3. How does FAudio decide what speaker configuration and device it outputs to?
4. Is there a configuration file/interface somewhere that can override the system speaker configuration?
5. Is there a diagnostic log that can be printed somewhere to troubleshoot functionality of FAudio? Specifically to know if the settings I request are being put into effect.
6. Does FAudio support the JACK audio API/driver?

Thank you for putting your time and effort into inventing, maintaining and improving this awesome project!

[flibitijibibo]

1. FAudio supports any channel format that SDL supports - I think you can make an 8-channel stream but it might assume 7.1, might be worth bringing up at that repo.
2. Probably not, XAudio2 mostly just takes in source samples and applies arbitrary matrix coefficients, the coefficients are influenced usually by X3DAudio/F3DAudio but sometimes studios have in-house 3D implementations.
3. Same idea, XAudio2 mostly doesn't care what the channel mask is, X3DAudio does however. SDL sort of cares but I think it's mostly influenced by what backends support.
4. Not at the moment, but a custom F3DAudio/X3DAudio would be able to do that. I believe there's an Hrtf mod for X3DAudio that does this actually?
5. Would probably be addressed by 4's solution; F3DAudio.c does have debug info though.
6. Yes, as long as the SDL binary used by FAudio supports it as well.

[alex-schroedsen]

Good morning flibitijibibo,

Thank you for getting back to me so quickly with answers to every question. I really appreciate the benefit of your knowledge in helping me towards my goal. I have been learning about SDL and xaudio2 over the last couple weeks. I'm still not confident in knowing a lot about either. I've read that xaudio2 is a (topological-layer) lower level API than OpenAL in general. I have spent the last two weekends working on cross-compiling the old wine-4.2 xaudio implementation. It has not reached a working state yet, but I feel like I'm just a dependency or compiler flag off from getting things working. In the meantime I have also had the idea of cross-compiling the Linux version of SDL2 in order to give SDL, JACK support on Windows. I am hoping to then be able to spoof a channel mask in JACK and get SDL to output to a 5.1.2 channel mask. I don't know if this is going to work. I'll report back with my findings.

On the topic of FAudio, as I've been investigating the role of xaudio2_7.dll, I've also very commonly found winmm.dll in use. In the case of OpenAL Soft, the DLL receives all data that's needed for the audio presentation. For xaudio2_7.dll it sometimes looks like the audio endpoint resides in winmm.dll (Sonic Mania). I am wondering if you can clarify, does Microsoft native xaudio2_7.dll output audio directly to WASAPI?

Another question, you mentioned that XAudio2 mostly takes in audio samples and applies arbitrary matrix coefficients. Does FAudio (xaudio2 or x3daudio) do audio mixing for positional audio or is mixing done by SDL-mixer? I started reading Microsoft's educational materials on XAudio2, it describes sources in virtualized space and listening points/virtual microphones. Are there a set collection of channel masks that exist or is there a call that can define a custom channel mask at runtime? Are these channel masks defined in x3daudio, thus that's why you suggested a custom x3daudio?

I have also had the fortune of being able to chat with the creator of the Wine-4.2 xaudio implementation, Andrew Eikum of Codeweavers. He also suggested that I go with FAudio rather than the Wine-4.2 xaudio implementation. He suggested that I look into building a OpenAL backend for FAudio, passing through positional data and other audio calls in order to implement spatial audio. I had mentioned that involving OpenAL Soft is an important aspect of this goal in order to obtain ambisonic output. I was reading through closed issues and saw the bounty list for implementing FAudio in Wine to begin with. I have set a bounty for Andrew of Codeweavers, however would you be interested in creating a bounty here for an OpenAL Soft backend in the event that Andrew doesn't accept my bounty.

I consider myself a novice reverse engineer, which has gotten me this far in this goal, but I am completely inexperienced when it comes to actual C, C++ and XAudio2, OpenAL programming. This is why I'm trying to cross-compile existing code rather than create something new.

The OpenAL Soft Issue page can be found here along with Andrew's comments kcat/openal-soft#997