VK: Use surface currentTransform as swapchain pre-transform. Issue #3473

[bkaradzic]

cc @attilaz

[attilaz]

👍 That is pretty similar to what I did. The other part is that the rendering should be setup properly when rendering to swapchain fb. SetViewPort/SetScissor/projection matrix should be rotated too and the dfdx/dfdy functions in the shader. I have handled inside bgfx, but I guess could be handled outside of bgfx. But the user should know about if prerotation is needed.

[bkaradzic]

Yeah, I was thinking of that. While I can rotate viewport, scissor, this shader part seem very non-standard feature.

[attilaz]

This whole prerotation thing is a mess In my opinion. It could have been much better if android(google?) would just require that devices with vulkan have a DPU that can handle this.

The shader dfdx/dfdy part wasn't needed for us so I skipped it, but in general it is needed.
Viewports are already flipped. So the whole thing is a little messy, I have only implemented what we were needed.
And our shaders were not using the full 4x4 projection matrix to render into the swapchain fb. It is the final "combine" step of 3d sceneview postprocessing and ui for us, so a full 4x4 matrix multiplication was not needed, but I guess is not a big vertex shader optimization. So You can require that the user uses the modified projection matrix to prerotate.
My solution was to add a bgfx system uniform vec4 that stores the 2x2 prerotation matrix, and patching the end of vertex shaders with this transform. For normal view the matrix is identity. For us the patching was manual, so for optimization I skipped it for shaders that can't render into swapchain fb.
dfdx/dfdy could also be rotated with this vec4.