proposal: PipeWire backend#19
Conversation
|
You shuld take a look at mixxxdj/mixxx#10999 which provides a stable jitter free host time reference for audio syncronization. This should be suitable for audio resampling to sound devices with drifting clock. |
|
|
||
| ## Alternatives | ||
|
|
||
| An alternative is to use a different crossplatform library which has fine grained API for different backends. One such contender is [libsoundio](https://github.com/andrewrk/libsoundio). The issue is that this library is not maintained, and does not support PipeWire on linux, only PulseAudio and JACK. |
There was a problem hiding this comment.
I think we can sort out this option.
There was a problem hiding this comment.
By sort out you mean remove this option?
|
|
||
| ## How | ||
|
|
||
| * The proposal will be implemented as a PipeWire client which listens for all node/port/link objects (similar to existing PipeWire patchbays), and creates SoundDevices accordingly. Since this happens for all source/sink available, not just soundcard source/sink, we can route any source/sink to Mixxx. This will work with the current routing UI in Sound Hardware preference page. Same mechanism will be used to update the routing UI to reflect any changes made by an external patchbay. |
There was a problem hiding this comment.
Here we need more details.
We have two use cases:
- Normal Linux "Souncard" based approach. -> The hardware GUI can remain.
- The Ubuntu Studio based approach. The wiring is outsourced to an external patch bay application.
Which use case do you want to focus one (first)
There is another degree of freedom JACK Support on windows and MacOS. Pipewire does support the JACK API. So we may use Pipewire via the JACK API on all platforms.
There was a problem hiding this comment.
By the first usecase you mean the Mixxx GUI for selecting inputs and outputs? If that's the case, I would prefer to focus on this first, arguing that Mixxx should provide a completely self sufficient experience first, without relying on external tools (later we can integrate them).
I agree that JACK support can be used on Windows and MacOS, but we need to figure out if anyone is using JACK over WASAPI (on Windows) or over whatever MacOS native API is. Are there any advantages of our dedicated JACK implementation over the PortAudio backend, apart from better patchbay routing and naming conventions? Can we do a poll somewhere, to ask what is preferable? I also read that PipeWire has a better security/permisson model, paraphrasing from their website:
PipeWire was designed with a powerful security model that makes interacting with audio and video devices > from containerized applications easy, with support for Flatpak applications being the primary goal.
I'll have to look a bit to see how Flatpak is different on JACK vs PipeWire, and how it would affect end user.
There was a problem hiding this comment.
So you propose to not implement the jack API of Pipewire? Would be an option.
I wonder if we need the jack API later anyway because if Ubuntu Studio.
Yes a poll would be nice if anyone is even interested in jack on windows and macOS and what the favorite alternative would be.
Maybe it is for your project enough, to keep that in mind and make a later introduction of these alternatives easy.
There was a problem hiding this comment.
Ubuntu Studio 24.04 LTS has PipeWire by default. Is there any other reason we need JACK API for Ubuntu Studio? Older versions compat?
There was a problem hiding this comment.
Ther is no reason to use the jack API on Ubuntu Studio. Implementating the jack API in Mixxx would allow to use original Jack server as well. But I don't see a reason to do it on Linux because Pipewire outperforms Jack.
All native Jack tools the user might want to user are compatible via the replacement library pipewire-jack.
|
|
||
| * The proposal will be implemented as a PipeWire client which listens for all node/port/link objects (similar to existing PipeWire patchbays), and creates SoundDevices accordingly. Since this happens for all source/sink available, not just soundcard source/sink, we can route any source/sink to Mixxx. This will work with the current routing UI in Sound Hardware preference page. Same mechanism will be used to update the routing UI to reflect any changes made by an external patchbay. | ||
|
|
||
| * On Linux systems, PipeWire would show up in the Sound API option in the preference panel, among other audio APIs offered by PortAudio. The existing soundio code would be refactored into enumerators for different audio backends, similar to Controller enumerators. |
There was a problem hiding this comment.
This requires an internal abstraction of PortAudio and Pipwire devices, which might be unnecessary complexity.
For the first approach a Pipwire (JACK API) only solution would be OK for me.
If we in addition find out how to run Mixxx directly with the ALSA callbacks, (via pipwire or bypassing it) we can retire Portaudio on Linux entirely.
I can Imagine to have a ALSA and a Pipwire/Jack mode finally, nothing else.
ALSA for the lowest possible latency and Pipewire for routing system sound and other applications.
There was a problem hiding this comment.
Just found this: https://www.omglinux.com/pipewire-gains-a-zero-latency-jack-d-bus-bridge-performance-improvements/
Not exactly sure what it means, but when it manages to remove the extra latency cycle of Jack and early pipe wire, we can even dispose the direct ALSA access.
The test is copy the incoming date directly to the output.
In case of ALSA we receive the captured audio in one buffer and can copy it to the output buffer. The output buffer will reach the DAC, after this single cycle.
Jack was delaying this write back by one cycle in my tests documented here:
https://mixxx.org/news/2021-05-09-jack-zero-latency/
I think the very first task is to repeat the test with the latest pipewire. Than we can decide to add or remove the direct ALSA mode.
There was a problem hiding this comment.
Pipewire is now at V 1.6 in the test it was with Pipewire V 0.3.36
There was a problem hiding this comment.
I think the very first task is to repeat the test with the latest pipewire
I wrote a simple PipeWire program where I copied the input buffer to output, and measured its latency with jack_iodelay, where I got ~23 ms roundtrip latency for 1024 buffer size. I wrote about this in the GSOC channel on Zulip, can you comment on that?
I can try with implementing PipeWire client in Mixxx, is there any case I'm missing with the simple program?
There was a problem hiding this comment.
The simple program is probably good enough.
For a first test.
There was a problem hiding this comment.
Did the test and I can confirm that there is no latency difference between the Portaudio ALSA solution and Pipewire.
There is only a minor concern that we probably give away some usable latency CPU for Mixxx to manage the additional context switch. Not sure how much this is, hopefully negligible for the 1:1 copy case.
This means we can target a solution that is pure pipewire on linux without any Portaudio options.
| exposed, which leads to poor experience on Linux (for example, | ||
| [incorrect naming of JACK ports](https://github.com/mixxxdj/mixxx/issues/5979)). | ||
|
|
||
| * There is no hotplug for SoundDevices |
There was a problem hiding this comment.
You may also mention the port model vs. the soundcard model.
| Goals and use cases for the solution as proposed in [How](#how): | ||
|
|
||
| * Refactor code related to current audio backends, and allow the selection of | ||
| PipeWire among the available backends. |
There was a problem hiding this comment.
With the latest findings, this can even be a compiler switch or a command line parameter. This releases the user form the decision if they should use Pipewire and probably you form corner cases when switching the API.
There was a problem hiding this comment.
WDYM by corner cases, conditional code in CMake files and #ifdef (PIPEWIRE) and #ifdef (PORTAUDIO) macros?
There was a problem hiding this comment.
I have some doubts that a common iterator over Portaudio and Pipewire is an easy thing, because of the opposite abstraction models, but I might be wrong. The #ifdef solution works probably for all users well, but it is finally your project decision.
| * Refactor code related to current audio backends, and allow the selection of | ||
| PipeWire among the available backends. | ||
| * Get feature parity with the current PortAudio backend. Ensure that drift and | ||
| jitter correction is happening properly. |
There was a problem hiding this comment.
| jitter correction is happening properly. | |
| jitter correction of two DAC clocks is happening properly. |
There was a problem hiding this comment.
Pleas also mention smoothing of the waveforms. A precise clock is key for that.
There was a problem hiding this comment.
Can you elaborate, I don't understand WDYM by smoothing of waveforms.
There was a problem hiding this comment.
We have quantization by the audio buffer and we have quantization by the display refresh rate. If we would straight forward pass the audio position to the waveforms we see a jitter, quantization noise. In addition we have two different clocks. The DAC clock and the GPU clock.
We have solved the issue by using the time ALSA reports when the sound reaches the DAC. The VisualPlayposition class calculated which sample is proceeded in the DAC when the next waveform frame is displayed. The Waveform position is moved to that place.
This works nicely with ALSA which reports precise timing. But not that perfectly with other APIs. Ther is a sanitary check implemented which falls back to the CPU clock counting samples when the check fails.
There is a similar issue with Ableton link where two DACs on two different devices needs to be synced. @JoergAtGithub has linked that above.
There was a problem hiding this comment.
PipeWire has similar API for the time the sample will reach the DAC. That should work with current logic.
I assume this is what
Access to DAC timing info of the soundcards
meant in the idea.
| jitter correction is happening properly. | ||
| * Hotplug for audio devices on PipeWire | ||
| * Synchronize routing UI with changes through external patchbays | ||
| * Have a design style that is concise and covers all the essential information. |
There was a problem hiding this comment.
Do you plan significant changes.
There was a problem hiding this comment.
Changes about what? The
Have a design style that is concise and covers all the essential information.
was left by mistake from the template, I'll remove it
| * Hotplug for audio devices on PipeWire | ||
| * Synchronize routing UI with changes through external patchbays | ||
| * Have a design style that is concise and covers all the essential information. | ||
|
|
There was a problem hiding this comment.
Support external patch bays. (Not like now with Jack where only anyway connected ports are exposed)
There was a problem hiding this comment.
That's what I meant by:
Synchronize routing UI with changes through external patchbays
I'll be more explicit.
| mechanism will be used to update the routing UI to reflect any changes made by | ||
| an external patchbay. | ||
|
|
||
| * On Linux systems, PipeWire would show up in the Sound API option in the |
There was a problem hiding this comment.
the Sound API option...
I don't like the idea. Portaudio enumerates all devices from all APIs at start up. The "listening" nature of Pipewire does not fit to this approach. So it is probably more straight forward to have a switch above, that decides if its a Portaudio or a Pipewire build. Even a compiler switch will work for me.
There was a problem hiding this comment.
I encountered similar issue while introducing libremidi along with PortMidi. PortMidi had a queryDevices() which returned all the devices, and libremidi had the "listening" nature. As a compromise, the initial libremidi events are supressed, they are recieved by calling its queryDevices() (which is part of the enumerator interface), and then onwards any new events trigger respective signals, and further queryDevices() calls give devices discovered till then.
So as a part of the refactor, when the PortAudioEnumerator and PipewireEnumerator are split up, both have the queryDevices(), which is used to get the devices when the Sound API switches, and any new device events are handled appropriately by Pipewire signals.
This is more complex than simply having one API at a time, maybe lets keep this in back of the mind, depending on how much additional complexity we have to incur to support this case.
|
|
||
| The tasks to do in order to migrate to the new idea. | ||
|
|
||
| * [ ] Implement support for PipeWire backend. |
There was a problem hiding this comment.
[ ] test the multi soundcard sync of Pipwwire.
|
|
||
| * [ ] Implement support for PipeWire backend. | ||
| * [ ] Add soundcard hotplug support. | ||
| * [ ] Modify PipeWire graph from Mixxx UI |
There was a problem hiding this comment.
Not sure if this is beneficial work. Can't we just add a button to open the system installed Graph editor?
qpwgraph/Helvum than the user feels at home with the known tool.
It is unlikely that the user opens Mixxx for doing this. The GUI in Mixxx shall be optimized to do the easy Mixxx only tasks. The graphical representation seem to be overdone.
Maybe we can put a bit of business logic on top of it. For automatically suggest mappings.
There was a problem hiding this comment.
I wonder who is responsible for rewire the graph after restart.
At one point we have discussed two modes to switch between. The media player mode at home and the DJ mode when connecting the controller soundcard.
There was a problem hiding this comment.
It would be super great if we finally can access the fader in the sound hardware via Mixxx.
There was a problem hiding this comment.
By "Modify PipeWire graph from Mixxx UI", I meant how currently in PortAudio backends, we can set the Mixxx Main/Headphone/Deck 1/2/3/4 outputs and Microphone 1/2/3/4 inputs from the Sound Hardware preference page, that UI is completely sufficient for routing to and from Mixxx. Only case it is lacking is to be able to input from multiple sources, or output to multiple sinks (if we want to address this too, we can add a plus (+) button to add additional source/sink to a single Mixxx input/output).
qpwgraph allows saving and loading patchbays. So if we edit the graph in qpwgraph (or from Mixxx UI, like mentioned above), we can save the patchbay, and qpwgraph sets the connections accordingly. We can include this feature in Mixxx, with above mentioned routing, to be independent of qpwgraph. Is this worth it?.
I do remember the discussion about the two modes, but I cannot recall the specifics or find the discussion, can you explain?
About accessing the analog volume, I still have to read about it, I will add this as TODO.
There was a problem hiding this comment.
Is this worth it?.
IMHO not. When we look at the eco system of different audio application actions, it does not seem to be reasonable that any of these has its own idea of routing and storing the pipe wire graph. And finally may have evrn concurrent settings.
There was a problem hiding this comment.
So I checked, it is possible to access soundcard hardware volume. There is a very nifty program pw-volume for reference. One issue is that PipeWire in Pro Audio mode does not manage setting hardware volume, so if we want that we will have to look other way (use ALSA API?).
There was a problem hiding this comment.
In normal mode, its a bit more complex, because even though we set the parameter on the soundcard, we need to mention the route we want to affect (in desktop case speaker or headphone route), so we need to keep track of the current route from the events.
There was a problem hiding this comment.
Even if this is a manual choice, it would be a great benefit to be able to adjust the hardware volume from the controller.
| * [ ] Implement support for PipeWire backend. | ||
| * [ ] Add soundcard hotplug support. | ||
| * [ ] Modify PipeWire graph from Mixxx UI | ||
| * [ ] Update Mixxx UI from external graph changes |
There was a problem hiding this comment.
Once the list is complete, you need to define mergable PRs with a testable goal and an estimated time frame.
There was a problem hiding this comment.
I am still struggling to understand why would we want to implement pipewire directly, rather than improving the situation with portaudio.
Please make sure to extend this proposal to highlight why this is not a solution (e.g reference issues in PA that highlights the fundamental issues with hotplug support or output naming issues)
Edit: some more context on this struggle. Recent backend extension such as iOS CoreAudio or Android Obee were made via portaudio to lift the maintenance burden on our end. Yet, both these back end have very different capabilities and design, but portaudio's flexibility allowed supporting their subtilities, so it would be good to understand why this does not apply for pipewire
|
I read the PortAudio issues about hotplug and PipeWire. There is nothing hinting at any fundamental issues with the library, just no work is being done. Even the JACK naming issue is without any such fundamental issue. If hotplug and PipeWire support is added to PortAudio, that would be enough to satisfy the proposal (along with some cherry picked extra PipeWire API which we can add in PortAudio as Host API-specific extensions (so a pa_pipewire.h), like callbacks for graph connection changes, to update Mixxx's UI to external changes like done by qpwgraph (edit: does PortAudio allow this kind of specific API? Why wasn't this present in JACK Host API-specific extension?). Only issue I can find is that we might miss on additonal PipeWire API, like providing some client metadata which would help with Mixxx's appearance in other application and routing policies, or getting same data from other applications. |
|
In the blog by @daschuer about JACK latency (which I still don't understand completely), it is mentioned that PortAudio JACK backend has 4 buffers worth of latency. Is that due to the use of JACK through PortAudio? Is it possible for implementing PipeWire support to it while keeping latency of only 2 buffers (theoretical minimum in case of direct PipeWire implementation)? In case not, is that big enough of a deal to warrant a direct implementation, and not through PortAudio? |
|
The blog is from a time where Pipewire was at version 0.35 where It was used in an asynchrony mode. This has been fixed, probably in Pipewire 0.74. Ubuntu Noble 24.04 our minimum Version for Mixxx 2.6 hast Pipewire 1.0.5. where I have confirmed that this problem is fixes. It would be nice if your project will support that version, but it is also likely that we bump our Mixxx 2.7 minimum supported version to Ubuntu Resolute 26.4 which has Pipewire 1.6. Portaudio does not introduce additional latency buffers. Of cause it consumes some CPU for copy the buffer. Portaudio already supports Pipwire via the Jack API and via the unreleased Pulse API. IMHO there is not really a point for implementing a third way using the Pipwire API, that will be limited in the same way like the Pulse and Jack implementation, just using different named functions. The main driver of this project is to allow external patch bays and have hot plug support. So the alternative based on Portaudio is to bring that into Portaudio. |
- mention the port model vs. the soundcard model - mention smoothing of the waveforms (accurate sample playback timing reporting) - be explicit about supporting external patch bays - add goal for testing drift/jitter correction and sync with multiple soundcards
4 participants
I have modified the GSOC proposal to suit the template, I have omitted some info like the diagrams. There is some repetition, and some missing details that I will add back as I figure out.