Audio loudness normalization

[ghosty-be]

an often read complaint was that talks were too silent and Q&A following that too loud ... and there is a wide variety in volumes over all talks...

[indiscipline]

First of all, equalisation is a specific term for manipulating frequency contents of a sound. The issue described here is called Loudness Normalization. Issue is due for renaming.

While this is a great goal, it's rather hard to enforce live. The speakers have to have a reliable way for measuring their output loudness. The suitable measurement units are Short-Term LUFS or RMS, but not every piece of software provides the meters for that.

[yoe]

So, SReview (the tool that we use for postprocessing and transcoding videos, and which we this year repurposed for also handling upload and preprocessing) actually has builtin support for loudness normalization, using bs1770gain. However, since that software mangles the audio in more ways than just "loudness normalization", which was causing bugs in SReview (apart from it being written by a right-wing extremist nazi), it was disabled for the upload processing for FOSDEM 2021.

I recently implemented loudness normalization using the ffmpeg loudnorm filter, which should allow me to disable the bs1770gain implementation; so if FOSDEM 2022 is still going to be online (in whole or in part), this issue should be fixed.

[indiscipline]

The issue with loudnorm is that it doesn't guarantee linear normalization even in double-pass mode. I've bumped into this with my ffmpeg-loudnorm-helper thingy. For some dynamic content loudnorm doesn't apply compression/limiting to fit peaks into the required range and falls back to dynamic mode which sometimes results in sudden jumps of loudness and overall inferior sound compared to a proper chain of compression and loudness normalization.

[yoe]

Darn, didn't know that (I only just wrote the loudnorm based normalization). In that case I suppose we're not there yet :-/

Do you have any better suggestions for implementing audio loudness normalization?

[indiscipline]

This will probably be a long comment.

The thing is, for the best results you need to control both the dynamic range and the final loudness of the content. Doing it by hand is super easy due to multiple metering and visual clues available and usually takes a bit of back-and-forth passes. Also, standard tools process audio in 32bit floating point which eliminates the issue of clipping at the intermediate stages.

The dumb approach could be something along the following algo which remotely resembles the manual routine:

1. Determine the average loudness and apply gain to normalize it to a baseline.
2. Apply gentle (low-ratio, 1.2:1 to 2:1) compression with slow release, wide knee and the threshold set to catch excessively loud sections of audio. This works in seconds time resolution.
3. Apply more strong (2:1 to 6:1) compression with faster attack and faster release but with the threshold set about 50% higher (closer to 0dB) to catch loud sounds. Time resolution in 50ms to 2s.
4. Apply linear normalization to a given LUFS level.
5. Apply brickwall limiting to prevent clipping. Threshold no higher than -1dB. This is a stage dealing with the smallest time resolution.

Assumptions:

1. No clipping can happen (32bFP).
2. The audio wasn't dynamically processed during recording.

To gracefully deal with the second assumption the tool could track a few moving averages (akin to a short-term LUFS) on the time scales of the points 2 and 3 and if there's no crossing some predetermined levels it means the audio already fits into the required dynamic range corridor and the appropriate processing stage can be skipped.

This routine is based on the approach described in my article which I proposed as go-to reference for voice processing for FOSDEM: https://indiscipline.github.io/post/voice-sound-reference/#strategies-for-applying-processing
The main idea is to deal with macro- and microdynamics sequentially and then setting the final loudness.

Manual processing has a benefit of not necessarily relying on compression for dealing with dynamics on stages 2 and 3, as an engineer can clearly see the portions of the audio which fall below/above the average and adjust the gain accordingly. This is what loudnorm seems to be trying to simulate, but the volume swings are often unnatural and the timing of applying gain adjustments are unreliable. Tracking loudness state ("the average volume shifted" - i.e. distance to a microphone changed, "short volume outlier" - i.e. a loud phrase, exciting moment, etc., "loud sounds happening" - laughter, coughing, dropping things, etc.) and applying simple gain corrections based on it would be preferable to just relying on compression, but this requires a much more complex solution.

UPD: Proper dynamic processing units allow reacting to a loudness measured not only in peak but in RMS, which in a way gets one closer to tracking state which I described above. You can set multiple such units to process audio based on different time resolutions.

Also, I still suggest renaming the issue.

[indiscipline]

Ah, another addition. My previous post is all about post-precessing. This is suboptimal to properly adjusting the sound on the recording end. If the settings were off and the sound was recorded distorted or too noisy or mangled by overeager noise-reduction or abysmal codecs then there's almost nothing you can do about it in post-processing. Preparing some standard protocol of setting things up for the recording can go a long way.

[yoe]

Yeah, okay, afraid that sounds a bit too complex...

My problem is that I'm dealing with audio which can be literally anything:

1. Video that was pre-recorded by the speaker, who doesn't know what they're doing, but at least used a headset or something so sound levels are consistent;
2. Live recordings with terrible mics and people being all over the place;
3. Live recordings with proper mics with an audio tech who knows what they're doing, and that are pretty much already OK when given to SReview

and I want something "reasonable" to roll out. For each case, that would be:

1. Adjust the gain linearly over the entire file so that things end up at the correct level
2. Probably do dynamic loudness normalization
3. Probably do "nothing", or very close to it, so that the hard work of the audio tech is not wasted (at worst, some finetuning so that we end up as close to -23LUFS as we can get)

Without any manual work (because that's the whole design goal of SReview: "do as much automated as possible")

I'm know I'm asking for AI or some magic code that will DWIM without any effort, but I don't need perfection; I just want to get as close as possible to those three results. The alternative is that a (too) small team will have to manually balance 600+ videos in two weeks, and that's just not possible.

I just found ffmpeg also has an "ebur128" filter, which I guess I can look into for more options, but for now I'll stick to loudnorm, accept that it won't be perfect, and put this on the backburner in case I have time left at some undefined point in the future (yeah, right). Alternatively, patches are definitely welcome ;-)

[indiscipline]

The logical simplification of the steps I proposed is sticking one gentle compressor before loudnorm to decrease the number of fallbacks to dynamic normalization. This will hurt case 3 a bit, but unfortunately this is the rarest case and you probably aren't optimizing for it.

[yoe]

It's the rarest at FOSDEM, but it's not always the rarest.

It's true that I'm not optimizing for it, though; and if there's a conference where it is guaranteed to always have happened, it's always just possible to disable the normalization in SReview, so then that shouldn't hurt anymore.

Thanks for your input, you've given me some food for thought. Not sure I'll find the time to implement this any time soon, but at least I know how to improve matters should it be necessary.

[indiscipline]

I'll be glad to help further. Feel free to contact me on matrix with any questions. Not sure I'll be able to contribute any code, though.