Add test cases for STEM

[acolombier]

No description provided.

[daschuer]

Thank you for taking care, unfortunately both files congaing only repeated rectangular noise a receipt to blow your speakers.

This is not suitable to find seeking issues. Can you produce two new files with real audio?

[JoergAtGithub]

@daschuer The waveform in your screenshot looks corrupted. It should look like this:

I agree, that this waveform is not suitable to find seeking issues. This could e.g be achieved by modulating the amplitude of the pulses with a linear ramp. With real music issues are difficult to understand.

[daschuer]

I have just noticed that the files for metadata testing introduced here:
#13044
are also missing. I think we can use the same files right?

• cover-test.stem.mp4
• cover-test.stem.m4a

[acolombier]

I've been struggling to generate useful file . I'll try to give that another go today. FYI, the current file were generated with ffmpeg sine generator

Regarding your comment on Zulip:

We need stem files in the "id3-test-data" folder, ideally with a not repetitive sound. This way they are picked up by the SoundSourceProxyTests

I'm a bit confused as my understanding is that id3-test-data folder is reserved for ID3 tests, whereas these tests strickly follows on decoder capability. Note that by tweaking getFilePaths, all SoundSourceProxyTests will pick up the STEM files

[daschuer]

Ether way is OK.

Regarding the content? Can't you just generate a stem file from a real music? Something without copy right like @Eve00000 release announcements. Other option is to use Audacity fade-in effect on the existing audio.

[acolombier]

I managed to generated a sine signal with modular frequencies as suggested by @JoergAtGithub, I'm just working on my stemgen to support stem creation, as surprisingly, the NI StemCreator doesn't works with AAC anymore... (definitely used it during GSoC so I would assume there is some codec issue, perhaps related to Window update that would have happen since then... This is to wonder why NI pushed for having AAC in the spec 😅 )
I could also demucs a track from there, what do you prefer?

[daschuer]

I think most close to reality is real music. So I vote for that. Other solutions without repeating patterns are also fine.

[acolombier]

I've pushed 4 test files:

• 2 reels, one with ALAC, one with AAC which is failing
• 2 modular sine, generated , also in ALAC and AAC but passing

Not sure what we should do here?

[daschuer]

It looks like all sine based test are to weak. AAC and friends assemble the sound from short sinuides, so it's obvious that this is easy to reproduce.
Let's keep the real sound and fix the decoding issue.
Hi fully a longer lead in will do the trick. Can you also extend the first sound test table with these files?

[daschuer]

Nice pick :-) the real files are good. Can you remove the sine and squash the commit to not clutter the history with outdated binary blobs?

[daschuer]

Interestingly even the [0 -> 10000] seek test fails where both streams are stating with a seek from the beginning which should have the same result. I am preparing a PR with better test output, and a cvs file for visualization in LibreOffice.

[daschuer]

#14658

[daschuer]

It looks like the issues are single outliers as part of a high frequency noise pattern. Here the difference after seeking with internal AAC:

This is probably the expected noise, the level of uncertainty in the lossy AAC codec.
We don't want to be a codec police. We just wan to safe the user from additional artifacts caused by Mixxx and secure the state against later regressions. Maybe we can filter out the noise before the compare?

[JoergAtGithub]

Maybe it's better to integrate the error of each sample, or better the square of the error of each sample and compare with a simple limit at the end?

[acolombier]

Thanks for the investigation @daschuer !

[acolombier]

This is probably the expected noise, the level of uncertainty in the lossy AAC codec.
We don't want to be a codec police. We just wan to safe the user from additional artifacts caused by Mixxx and secure the state against later regressions. Maybe we can filter out the noise before the compare?

Perhaps we could define a epsilon per codec? Currently it is statically defined to 0.01, but looking at your graph and testing locally 0.018 seems to be fine to get AAC to pass, and due to its lossy property, 1.8% doesn't sound alarming. Did you get a chance to test it with fdkaac as well?

[daschuer]

A 0.01 rectangular step is already a hearable click sound while a single outlyer of any height is unhearable. That's why I think we cheat ourself if we just raise the already high ellipson.

The blue dots a outlyers which can be considerd as OK. We need a non visual rule for the test for it.

[github-actions]

This PR is marked as stale because it has been open 90 days with no activity.

[github-actions]

This PR is marked as stale because it has been open 90 days with no activity.

[daschuer]

This PR depends on
#14658
#16396
#16439
#16440

[daschuer]

I have did a whole lot of seeking test and was able to find a seek independent frame that has random samples.

Here some checks I did:

• They are random with the internal aac decoder and with the libfdk_aac decoder. So we can sort out the decoder.
• My next idea was that it is an issue with the Stem sound source. But I could also sort out that, by using only one stream with the original FFMPEG soundsource.
• Next Idea was a buffer overflow. I see these frames in all five streams at different position. These positions do not move relative to the track, when I start decoding at diffrent point of the track.

Conclusion: There is an issue with the encoder. The file "has" random samples.

Here a Diff of the seeked source compared to the continuos reading of the left and right channel.

[daschuer]

LM_aac_5s.stem.mp4

The NI Free example stems work. Here is a 5s snipit. They are freely available for testing purpose.
That's what we do. So there shall be no copyright issue.

[daschuer]

@acolombier can you use that as a replacement?

[daschuer]

This is one of the remaining PRs for the 2.6.0 release.
#14658 and #16396 are still left to merge.
@acolombier Did you consider to use the alternative testing file?

[acolombier]

The used track is available under CC BY-NC which allows us to use this as-is. There is no license issue, and unlike the NI sample, this is a real track,

Equally, I am not sure the NI free sample are copyleft track. AFAIR, they are available for free download for test purposes, but I am not sure we are allow to modify them, and distribute them like so. Please share reference to the license otherwise.

[daschuer]

Any other track that passes the test, has no such random samples will work. Do you have another one?

[acolombier]

I will extract another 5 seconds of the track. Could you please share your script to I can make sure these other 4 seconds don't suffer from the same problem?