So, I have machine with CUDA 12.9 installed driver which prevents me native compilation on the host.
I've decided to reject usage of pre-compiled 3.11 wheel or other stuff that didn't work for me anyway.
After spending some time I've managed to compile it correctly, so I hope it will work for you too if you are stuck in the compilation phase and need a reproducible environment to perform compilation yourself.
https://gist.github.com/dasstyx/2e76de35c4850d87f8a9a4a03353886a