-
Notifications
You must be signed in to change notification settings - Fork 52
WIP: Final Report, GSOC 2025 #401
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,83 @@ | ||||||
| title: GSoC 2025 Work Product - Resampling Options for Mixxx | ||||||
| authors: Armaan Chowfin | ||||||
| status: draft | ||||||
| tags: gsoc, gsoc-2025 | ||||||
| comments: yes | ||||||
|
|
||||||
| ## Project Description | ||||||
| This GSOC project is derived from [Mixxx issue#9328](https://github.com/mixxxdj/mixxx/issues/9328) | ||||||
| > Mixxx uses a linear resample when scratching. This is blazing fast, but the sound can be improved. Here Mixxx should provide more resample options. This project involves to review the already used resample libraries RubberBand and Soundtouch and compare them with other candidates. The one with the best Sound/CPU load trade of shall be selected. Make sure that it supports on the fly changing of the sample rate without artefacts. This project may also involve to contribute a missing feature to such library. | ||||||
|
|
||||||
| ### Introduction | ||||||
| Mixxx offers the ability to emulate vinyl scratching when records are stored digitally. Here, a MIDI controller's jog wheels can be spun to emulate the motion of the turntable stylus during scratching. The action of scratching causes a sudden acceleration or deceleration in tempo of the loaded record(s). | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
The turntable stylus only follows the carves in the vinyl, This is a minimal movement. Also turntable is moving at a constant speed eg. 33 rounds/min. DJs have a so called slip pad between the vinyl and the turntable. This allow to stop or accelerate the vinyl on the turntable or play it backwards. The vinyl slips on the turntable. |
||||||
|
|
||||||
| On a turntable, scratching is performed by moving the stylus by hand - causing it to follow grooves in the vinyl that correspond to the analog audio waveform. However, while working with dital audio, faithful emulation of scratching requires a highly accurate and fast implementation of tempo ramping. In particular, Mixxx needs to implement fast and accurate resampling of input audio. Sub-optimal resampling often leads to audible distortions during scratching, particularly underflows. While there is technically also a chance for phase distortion due to incorrect interpolation algorithms, it is far less noticeable. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| At present, the Mixxx resampler for scratching uses a fast, handcrafted linear interpolation algorithm. Mixxx also uses the `SoundTouch` and `RubberBand` libraries to perform general audio time-stretching. These libraries implement interpolation algorithms, but their performance is not optimal for fast changing speed and pitch which is the case when scratching. Therefore, my project goal is defined as follows: | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| > To determine whether alternative interpolation algorithms result in a *noticable reduction in scratching artifacts or a latency improvement over linear interpolation* and **providing quantitative supporting evidence** in the form of C++ unit tests under varying scratching scenarios. | ||||||
|
|
||||||
| Contributing to this project involves fundamental ideas in audio engineering, digital signal processing and realtime programming. To make this report accessible to those who have not worked with audio systems, I provide all necessary theoretical background, before describing my implementation. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| #### Analog and Digital Audio | ||||||
| Vibrations of the surrounding air in turn cause the human eardrum to vibrate, and generate a continuous electrical signal. In audio-engineering terms, this electrical signal represents "analog audio", and our ear represents a (biological) "audio interface", i.e. a gateway for audio to enter or exit a processing system. The hearing sense is the result of our brain processing analog audio to create the perception of sound. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| While an analog signal is represented by its amplitude as continuous function in continuous time, the digital representation of that signal is a finite series of amplitude values generated by noting the value of the analog signal at fixed, discrete time intervals. The process of generating a digital audio representation from an analog signal is termed "sampling", and the length of the time interval is termed the "sample-period (its reciprocal - the "sampling rate" - is more commonly used while describing digital audio). In practice, analog audio from a sound source enters a mic, is converted to a continuous electrical signal, and finally, a component called the "ADC - Analog to Digital Converter" samples the electrical signal at a fixed sampling rate (ex. 44.1KHz, 48Khz, 96Khz, etc.), generating a series of amplitude values, i.e. digital audio. To allow this audio to be widely distributed, the digital audio is encoded to a standard digital format (ex. MP3, WAV, AAC, etc.) using well-known algorithms. This allows sampled audio, i.e. music records to be stored on digital hard-drives. The analog equivalent - vinyl records - skip the sampling stage entirely. Instead, a series of concentric grooves is etched into the disc, "encoding" the continuous change in the amplitude of the analog electrical signal. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| For an audio record to be subsequently played back, there must exist a processing system that understands the original encoding scheme. For vinyl records, we have turntables connected to amplifiers. Moving the stylus along the vinyl grooves generates a continuous electrical signal, which is sent to a speaker. The speaker, being an analog device, responds to the continuous electrical signal by moving its membrane, creating air vibrations that we hear as sound. To playback digital records, however, we need the right software. The standardization of audio formats ensures that any piece of software that adheres to certain conventions can "decode" and play a digital record. This is one key principle behind the audio playback feature of production-grade software such as VLC, Windows Media Player, Apple Music, Spotify, and even Mixxx. A second requirement of this playback chain is the accurate conversion of digital audio to the original analog signal, to drive the speaker membrane. Audio playback software communicates with a digital audio interface that contains a **DAC - Digital to Analog Converter**, which reconstructs the analog signal, and supplies it to the speaker. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| The accurate reconstruction of an analog signal from a digital record is mathematically guaranteed under certain conditions. The Fourier theorem - a famous mathematical result - states that any analog signal can be represented by the sum of sinusoidal components of varying frequency and amplitude. The set of frequencies and their amplitudes gives the spectrum of the signal. In this model, the Shannon-Nyquist Sampling theorem states that the frequency at which analog audio is sampled (i.e. sampling rate) is twice its highest-resolvable frequency component -the Nyquist frequency. As a consequence, given that the human hearing range is roughly 20Hz-20kHz, the standard sampling rates 44.1Khz, 48Khz and 96Khz, with Nyquist frequencies 22Khz, 24Khz and 48Khz respectively, ensure that a digital record can represent every audible human frequency. The actual signal reconstruction is carried out in DAC hardware by circuits that implement so-called "digital reconstruction filters". These filters perform mathematical transformations to the discrete sample sequence to recreate the original analog signal. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| #### Sample-Buffers, DAC, ALSA | ||||||
| Remember: Digital audio is represented by a finite sequence of amplitude values, with each amplitude value stored as a fixed-precision floating point number (Mixxx uses 32-bit floats). These samples are more generally treated as logical *frames* for multichannel audio. An audio frame is an array containing channels copies of the current sample value. There are three fundamental components in the audio playback chain: | ||||||
| - Mixxx (user-space audio processing software), | ||||||
| - ALSA (kernel subsystem with userspace API), and the | ||||||
| - DAC (device with transducer to convert digital to analog audio). | ||||||
|
|
||||||
| The Mixxx preferences page allows the user to configure a `Sample Rate (Hz)`, and `Audio Buffer (ms)`. These parameters together influence the quality of output sound. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How? |
||||||
|
|
||||||
| *Mixxx Sound Hardware Preferences* | ||||||
|  | ||||||
|
|
||||||
|
|
||||||
| The audio playback chain has three key buffers: A larger, userspace buffer containing outbound, perhaps processed frames of the digital record, an ALSA-managed, DMA-registered ring buffer of configurable size, and a hardware FIFO buffer in the DAC itself. The DAC has a configurable clock, whose clock rate (a.k.a. DAC sample-rate) specifies the fixed frequency at which frames in its hardware FIFO are consumed by the reconstruction filter. For example, a 44.1Khz DAC expects a frame to be available every 1/44.1K = 22.6us, for a total of 44.1K frames every second. Achieving this condition, however, is determined by upstream subsytems such as ALSA and parameters set by Mixxx. The `Sample Rate (Hz)` setting configures the DAC sample-rate, and the `Audio Buffer (ms)` sets the size in frames of the ALSA ring buffer for the selected `Sample Rate (Hz)`. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| The ALSA ring-buffer is of size `ring_buffer_size = Audio Buffer (ms)/1000 * Sample Rate (Hz)` frames, set via Mixxx preferences. Mixxx writes processed audio frames to the ring-buffer in units of atmost `ring_buffer_size` frames, and the ring-buffer is emptied in units of `period_size` frames (usually = `Audio Buffer/2`)[[1]](#1), negotiated between Mixxx and the audio driver. During playback, a software interrupt is triggered each time `period_size` frames are consumed from the ring buffer and DMA-transferred to the DAC FIFO. The interrupt is handled by a high-priority callback thread in Mixxx, which refills the ring-buffer with more frames. While technically, only `period_size` frames are written to the DAC on every callback, Mixxx prepares `ring_buffer_size` frames in that duration. We can therefore simplify our model by noting that on average, `ring_buffer_size` frames are written to the DAC every callback. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is how other audio applications may work. Mixxx is optimized for low latency and runs directly on a swapped buffer. While one chung kis passed to the DAC, the other is prepared by Mixxx. After a period both buffers are swapped.
Suggested change
|
||||||
|
|
||||||
| From the `period_size` and DAC sample rate, we can calculate `period_time = period_size/Sample Rate` as the hard real-time constraint on the user-space callback function. In other words, for the DAC hardware FIFO to never starve, the userspace callback thread must prepare atleast `period_size` worth of frames in atmost `period_time` time. Whether the constraint is met depends on various factors such as the amount of processing performed in the real-time thread, OS memory pressure, scheduling latency, etc. None of these kernel procedures have strict real-time guarantees, therefore on occasion, if `period_time` is too short, one hears pops during playback. This audio distortion is called a buffer underrun - highly undesireable in any live-DJ software. Since the main aim of Mixxx is to allow DJs to manipulate loaded tracks in creative ways - via mixing, ramping track tempo, digital effects, scratching, etc., the realtime thread must perform far more processing than a standard audio player. This requires low-latency implementations of audio processing workflows without degrading audio quality. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This paragraph is long without many useful information, I would remove it. Maybe rework the paragraph above a bit with these infos. |
||||||
|
|
||||||
| #### Sample Rate Conversion | ||||||
| While buffer underruns are a result of DAC starvation agnostic of the buffer contents, another class of audio distortions is caused by the DAC not having the *right set* of frames in its FIFO. Resampling outbound audio is a procedure that ensures the DAC is receives the set of frames required to generate the analog playback characteristics we desire. The primary use-case is to resolve a sample-rate mismatch between DAC and the digital record. | ||||||
|
|
||||||
| The input sample rate defines how many frames of a digital record represent one second of analog audio. Meanwhile the DAC sample rate specifies how many outbound frames are consumed per second of real-world (wall-clock) time during playback. For example, a digital record sampled from analog at 48 kHz stores 48,000 frames for every second of analog sound. If this record is played back on a DAC operating at 44.1 kHz, assuming no underruns, only 44,100 frames are processed each second—meaning less than a full second of the outbound audio is played back per second. This results in an unintended slowdown. Conversely, if the DAC sample rate exceeds the input sample rate, more than one second of the original recording is heard every second, creating the perception of sped-up playback. | ||||||
|
|
||||||
| Embed audio: [[wav@44.1Khz on 48K]], [[wav@48Khz on 44.1K]], [[wav@48Khz on 48K]] | ||||||
|
|
||||||
| Mixxx offers features besides simple audio playback. Another use-case for resampling is during an intentional tempo ramp, when there is no sample-rate mismatch. For a record sampled at 44.1kHz with a DAC also at 44.1kHz, scaling tempo by a factor of 3 means we want to pass 3x frames to the DAC on each callback than we would during standard playback. Without resampling, attempting to write 3x frames per callback would overfill the Mixxx-ALSA buffer. In the worst case, the excess frames would be dropped. Either way, the DAC would still consume only 44.1k frames per second—nullifying the intended tempo increase. It is clear that we need to create a situation where we can represent a longer amount of track duration using less frames, while ensuring that the resampled frames are still capable of being reconstructed. That is, every second, we need to represent 3 *44.1k frames using 44.1k frames only. This resample is achieved by a procedure called digital decimation, wherein frames are actually removed from a longer sequence before being written to the DAC. Conversely, digital interpolation is used when increasing track tempo, whereby new frames are generated between true samples using various algorithms. `SoundTouch`, `RubberBand`, `libzita` and `libsamplerate` are examples of open-source C++ libraries that implement standard interpolation algorithms to perform time-stretching on streaming data. | ||||||
|
|
||||||
| ### Pull Requests and issues | ||||||
| [mixxx#] libsamplerate with callback API | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| [mixxx#] Support for custom recording samplerates | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
|
|
||||||
| [mixxx#]Support for custom broadcast samplerates | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| ### Future Work | ||||||
| The merge of | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| ### Conclusion | ||||||
| The | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
| ### References | ||||||
| <a id="1">[1]</a> https://0pointer.de/blog/projects/all-about-periods.html | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure if it is a smart move to city this literally. Your project did not solve all this. Instead we now have a recording re-sampler and a working scratching engine draft with libsamplerate without any benchmarks.