[7.1] Crash on AMD Turion II CPU

[ziouf]

The problem

After upgrading Music Assistant from 2.3.6 to 2.4.0, FFMPEG is crashing on my old AMD Turion II CPU.

As said by music-assistant member in this comment, this project embeds static-ffmpeg

Is there any compiling options that could be missing to fix this issue ?

How to reproduce

music-assistant:2.4.0 uses static-ffmpeg:7.1 binaries

$ docker container run --rm -it --entrypoint /bin/sh ghcr.io/music-assistant/server:2.4.0 -c 'ffmpeg -version'
Illegal instruction (core dumped)

But static-ffmpeg:6.1.1 works like a charm on same hardware

$ docker container run --rm -it --entrypoint /bin/sh ghcr.io/music-assistant/server:2.3.6 -c 'ffmpeg -version'
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13.2.1 (Alpine 13.2.1_git20240309) 20240309
configuration: --prefix=/usr --disable-librtmp --disable-lzma --disable-static --disable-stripping --enable-avfilter --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libmp3lame --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librist --enable-libsoxr --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-lto=auto --enable-lv2 --enable-openssl --enable-pic --enable-postproc --enable-pthreads --enable-shared --enable-vaapi --enable-vdpau --enable-version3 --enable-vulkan --optflags=-O3 --enable-libjxl --enable-libsvtav1 --enable-libvpl
libavutil      58. 29.100 / 58. 29.100
libavcodec     60. 31.102 / 60. 31.102
libavformat    60. 16.100 / 60. 16.100
libavdevice    60.  3.100 / 60.  3.100
libavfilter     9. 12.100 /  9. 12.100
libswscale      7.  5.100 /  7.  5.100
libswresample   4. 12.100 /  4. 12.100
libpostproc    57.  3.100 / 57.  3.100

On what type of hardware are you running?

# cat /proc/cpuinfo 
model name	: AMD Turion(tm) II Neo N40L Dual-Core Processor
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_sav

[wader]

Hi, huh print version causes illegal instruction, that is a bit odd. Possible to run in gdb to see what instruction it is? might have to run container as privileged and run in gdb using something like gdb --args ffmpeg --version, then r to run and hopefully crash, then disassemble to show instruction listing

[ziouf]

I don't think the GDB output will help much. The binary does not embed debugging symbols.

see here

Program received signal SIGILL, Illegal instruction

[wader]

Aha maybe something like disassemble $rip,$rip+100 could work without symbols? not sure if eip or rip is the correct register, not a x86 expert :)

[ziouf]

I have absolutely no clue on what I am seeing ...

see here

I hope it's more obvious to you

[wader]

If we can trust the output "pinsrq" seem to be an instruction added in SSE4.1. Guess we might need debug symbols to know what is using it and also if it's the compiler outputs or if some explicit asm code somewhere.

I would probably take Dockerfile, trip away all libs, remove --disable-debug for ffmpeg configure, maybe also remove all steps after building ffmpeg. Then you should have a docker image with a ffmpeg_g binary that has debug symbols

[ziouf]

This cpu is missing sse4.1 instruction set.

# cat /proc/cpuinfo | grep flags | tr " " "\n" | grep -ow 'sse.*' | sort | uniq
sse
sse2
sse4a

But static-ffmpeg 6.1.1 worked like a charm.

It should be possible to have a conditional use of sse4.1 instruction set. (https://stackoverflow.com/a/20223986)

Maybe we can detect if this instruction set is available (https://stackoverflow.com/a/20202694) and use different path in the code base with a cpu dispatcher (https://stackoverflow.com/a/15610628). One optimised with sse4.1 and another compiled without.

I don't know if this is hard work to do but it would keep backward compatibility

[wader]

Hey, don't think i would like to modify or patch and upstream software to fix this. But i think this could be fixed by compiling ffmpeg and all dependencies with -march <some more generic x86_64> etc (not sure about rust builds) but not sure how much work it would be or if it could be done in a more "global" way during build. Also has to be sure that it does not end up disabling various modern cpu features that can be detected at runtime. SSE etc is used a lot to speed up media processing.

My guess why the 6.1.1 build works is that gcc by default detects cpu capabilities from it's build environment and that the environment has changed or possibly that newer gcc has started using sse4.1.

[ziouf]

Hey, don't think i would like to modify or patch and upstream software to fix this.

I completely understand

My guess why the 6.1.1 build works is that gcc by default detects cpu capabilities from it's build environment and that the environment has changed or possibly that newer gcc has started using sse4.1.

It's not very portable to have a build that depends on the build environment. Don't you think?

So there's no quick win other than upgrading hardware
It's sad :'(

[wader]

My guess why the 6.1.1 build works is that gcc by default detects cpu capabilities from it's build environment and that the environment has changed or possibly that newer gcc has started using sse4.1.

It's not very portable to have a build that depends on the build environment. Don't you think?

Yeap agree, that feels a bit shaky :( i do wonder how linux distributions handles this when building packages? relying on packaging scripts and each softwares build system to get it right seems messy.

So there's no quick win other than upgrading hardware It's sad :'(

Not sure what Music Assistant requires from ffmpeg but maybe a workaround could be that they have some build arg to make it possible to use an alternative ffmpeg build somehow?

[wader]

#606 includes this fix fraunhoferhhi/vvenc#493 which sounds suspiciously related this

[wader]

Try latest once it’s done building