Change gst-projectM to F32 audio format

[rootnotez]

ProjectM uses F32 (Float 32) internally - https://github.com/projectM-visualizer/projectm/wiki/Integration-Quickstart-Guide#supplying-audio-data

GST-ProjectM is using S16 (Signed 16) internally, and only S16 shows up in the plugin Caps once compiled -

gst-projectm/src/caps.c

Lines 21 to 27 in 54c481c

	GST_AUDIO_CAPS_MAKE("audio/x-raw, "
	"format = (string) " GST_AUDIO_NE(
	S16) ", "
	"layout = (string) interleaved, "
	"channels = (int) { 2 }, "
	"rate = (int) { 44100 }, "
	"channel-mask = (bitmask) { 0x0003 }");

Switching GST-ProjectM to F32 would align it with ProjectM, and remove the need for ProjectM to do an internal conversion.

Also, note in the gst-projectM Readme, the example given, the second audioconvert element is implicitly converting to S16, because that is the only Cap that gstProjectM has, so this is another unnecessary conversion, albeit in the paradigm use case, not in the code itsel.

gst-projectm/README.md

Lines 201 to 210 in 54c481c

	```shell
	gst-launch-1.0 -e \
	filesrc location=input.mp3 ! \
	decodebin ! tee name=t \
	t. ! queue ! audioconvert ! audioresample ! \
	capsfilter caps="audio/x-raw, format=F32LE, channels=2, rate=44100" ! avenc_aac bitrate=320000 ! queue ! mux. \
	t. ! queue ! audioconvert ! projectm preset=/usr/local/share/projectM/presets texture-dir=/usr/local/share/projectM/textures preset-duration=6 mesh-size=1024,576 ! \
	identity sync=false ! videoconvert ! videorate ! video/x-raw,framerate=60/1,width=3840,height=2160 ! \
	x264enc bitrate=50000 key-int-max=200 speed-preset=veryslow ! video/x-h264,stream-format=avc,alignment=au ! queue ! mux. \
	mp4mux name=mux ! filesink location=output.mp4

Including some Claude.AI output, as potential places to look.

Changing ProjectM GStreamer from S16 to F32

Main Changes Required

1. Change the audio sink caps format

In src/caps.c, you would need to modify the get_audio_sink_cap function:

// Change this:
format = GST_AUDIO_CAPS_MAKE("audio/x-raw, "
                          "format = (string) " GST_AUDIO_NE(S16) ", "
                          "layout = (string) interleaved, "
                          "channels = (int) { 2 }, "
                          "rate = (int) { 44100 }, "
                          "channel-mask = (bitmask) { 0x0003 }");

// To this:
format = GST_AUDIO_CAPS_MAKE("audio/x-raw, "
                          "format = (string) " GST_AUDIO_NE(F32) ", "
                          "layout = (string) interleaved, "
                          "channels = (int) { 2 }, "
                          "rate = (int) { 44100 }, "
                          "channel-mask = (bitmask) { 0x0003 }");

2. Change the audio data processing

In src/plugin.c, you need to modify how the audio data is processed. Find the ProjectM PCM audio function call:

// Change this:
projectm_pcm_add_int16(plugin->priv->handle, (gint16 *)audioMap.data,
                       audioMap.size / 4, PROJECTM_STEREO);

// To this:
projectm_pcm_add_float(plugin->priv->handle, (float *)audioMap.data,
                      audioMap.size / 8, PROJECTM_STEREO);

Note the division factor change from /4 to /8 since F32 samples are 4 bytes each (versus 2 bytes for S16), and the size is in bytes.

3. Buffer size calculations

Check src/gstglbaseaudiovisualizer.c and look for calculations based on audio format. Particularly, you might need to adjust the buffer size calculations like:

// In the setup function:
bscope->req_spf = (bscope->ainfo.channels * bscope->ainfo.rate * 4) / bscope->vinfo.fps_n;

I'm changing the multiplier from 2 to 4 since F32 samples are twice the size of S16 samples.

4. Adjust any memory map operations

Make sure any manual memory pointer calculations handle the increased sample size:

// Before:
gst_buffer_map(audio, &audioMap, GST_MAP_READ);
// ...some code...
// After (check if memory access pattern needs to change):
gst_buffer_map(audio, &audioMap, GST_MAP_READ);
// You might need to adjust how the data is accessed if there are direct pointer operations

[revmischa]

yup!

[hack-s]

almost...under 3 bscope->req_spf is not a buffer size, it's "required samples per frame". So it's not dependent on the sample size, but on the sample rate. However, the calculation is wrong I think and there is a good chance that it's the bug I have been looking for.