FFT data seems strange (vastly different to shadertoy.com)

[markus-wa]

I was having trouble porting some shaders from shadertoy.com and after some investigating it looks like the FFT part of :overtone-audio is either incorrect or just somehow quite different to shaertoy.com

This video contains a short demo of what I found: https://www.youtube.com/watch?v=3HWCf0sXXtg

Some observations:

• no issues with the waveform 👍 -> waveform is from -1 to 1 instead of 0 to 1 (or was it the other way around? - either way, shadertoy.com is one and shadertone is the other)
• many 'missing' frequencies
  • especially high ones
• red bar with resolution at 512, doesn't happen with the default of 4096
• the freqs that are there are often displayed as louder than in shadertoy.com

Unfortunately this makes most shaders I found useless since they don't work that well as is (very 'jumpy'/ jittery).

Maybe this is also an issue with my setup as I didn't see anyone else having this kind of problem here (maybe related to #34 / external SC).

This is my current setup on Ubuntu with Jack and external SC:

Edit: the red bar to the right made me curious, so I tried using 1024 as width:

It's sort of strange that the waveform falls to the bottom on shadertoy.com but just continues straight with shadertone - but I'm not exactly sure what to make of it.

[rogerallen]

We'll want to compare internal vs. external here, also Mac/Win/Linux. I tested on Mac back in the day as Windows & Linux seemed much more difficult to get running. It may be a couple weeks before I can get setup, so anyone else out there who could help--please do.

A test audio pattern with known frequency values (sin 440, or square wave, etc) would make the test easier to judge as to which is truly correct.

[markus-wa]

A test audio pattern with known frequency values (sin 440, or square wave, etc) would make the test easier to judge as to which is truly correct.

Good idea, i will give it a try if I find some time.

[markus-wa]

So it's been a while but I finally had some time to do some more testing!

video: https://youtu.be/TAUCjgb8mQQ
shader: https://www.shadertoy.com/view/Mlj3WV
audio: https://soundcloud.com/markuswa/440-4360-1300-21800a/s-Qh7sVSsmpbS

this is with the following frequencies (as visible in the SuperCollider FreqScope view):

• 440
• 4360
• 13000
• 21800

Screenshots

FreqScope (reference)

shadertoy.com

shadertone

Conclusions

• shadertoy.com seems to cut off around 12000 hz or so.
  • while I think the higher spectrum is nicer, maybe shadertone should be as compatible as possible with the site. but I'm not sure on this one
• in shadertone, everything is very flickery
  • I think this could be because the sample is chunked and not continuous
  • maybe this can be helped by using a ring-buffer for the sample on which the FFT is based?
    • I think as it is right now this buffer is inside SC, so we might not be able to change that, in that case maybe we can calculate the FFT completely in Clojure
    • this should also help with performance

I'm honestly not sure if I'm interpreting things correctly here so if you see something differently please let me know 🙂

[markus-wa]

in shadertone, everything is very flickery

note to self: should try this with internal server. there we might not have this issue

[markus-wa]

I assume the reason for shadertoy.com to cut off at 24000 hz is because they are using AnalyserNode.getByteFrequencyData which says.

The frequencies are spread linearly from 0 to 1/2 of the sample rate. For example, for 48000 sample rate, the last item of the array will represent the decibel value for 24000 Hz.

Edit: nevermind, I forgot it cuts off at 12k, not 24. could still be related though maybe.

[rogerallen]

They may be doing a cutoff at 12k just because most interesting-for-graphics audio data is below 12k. Above 12k is not that nice sounding. The highest keys on the piano are just 4kHz.

I thought this matched shadertone way-back-when. This could be a change by them?

If you scale the lookup into that texture by 1/2, I think you could line them up.

Alternatively, feel free to hack away at the code in here: https://github.com/overtone/shadertone/blob/master/src/shadertone/tone.clj#L74-L92

I think I just got it to look "good enough" and moved on. my lin-mag magnitude calculation on line 91 might not be the right thing.

[markus-wa]

Yeah I tried replicating this here: https://github.com/supercollider/supercollider/blob/f65a1ef7bfb100f4ccb7dcab3a6dff6826be4344/SCClassLibrary/Common/GUI/PlusGUI/Control/FreqScope.sc#L24-L37

but without any luck. whenever I change something it just breaks the whole thing or makes it much much worse. and I guess it doesn't help that I have (for the most part) absolutely no clue about what those calculations do.

[markus-wa]

but I will give it another go at some point 😄

[rogerallen]

If it makes you feel any better, I had to study it line-by-line and lookup every single function when I worked on it. FFT is not a basic concept and the docs seem to be super-basic and expect you to be an expert. If FFT details are not familiar, suggest taking your time...

[markus-wa]

Thanks, I will try to do that 🙂

[markus-wa]

just a note: after some more investigation it seems like the/(a?) issue would be the lack of PV_MagSmear. I noticed there's also a comment that suggests it's intentional, but unfortunately without explanation.

[rogerallen]

PV_MagSmear should be equivalent to what is supposed to be just a faster version of it at: https://github.com/overtone/shadertone/blob/master/src/shadertone/tone.clj#L88-L91

If you find that is not the case, that would be a bug.