HLSL: Added support for tessellation control/evaluation shaders

[xen2]

Adds hull shader (tesc) and domain shader (tese) code generation to the SPIRV-Cross HLSL backend.
This PR is built on top of #2603 which should be first merged in.
I also keep it as draft as I am still testing it in our project and refining it, but so far it worked well on several tessellation shaders our engine generates.
Please give me your thought regarding the general approach, would it be OK to merge it?

Hull shader (tesc) — function splitting

HLSL requires hull shader work split into a per-control-point entry point and a separate patch constant function, but SPIR-V expresses both as a single main(). The compiler splits them using a redirected emission approach: rather than writing to the main output stream, emit_instruction() writes to one of two capture buffers (tesc_per_cp_buffer or tesc_patch_buffer) based on a TescEmitPhase state machine:

• Setup → PerCP: triggered when an OpAccessChain using gl_InvocationID is seen (output writes begin).
• PerCP → Patch: triggered by OpFunctionCall to the patch constant function, or an OpStore to a tess level/patch variable.
• Patch → PostPatch: triggered at the merge block of the if (gl_InvocationID == 0) construct; the closing brace is stripped and subsequent output is discarded.
• OpControlBarrier is suppressed entirely (hull shaders synchronize implicitly).

After emission, the two buffers are assembled into tesc_main() and tesc_main_patch(). A separate patch_constant() wrapper calls tesc_main_patch() and writes tess levels to the SPIRV_Cross_PatchConstant return struct.

Note: The function splitting relies on the GLSL-to-SPIR-V compiler wrapping patch constant work in an if (gl_InvocationID == 0) { ... } guard block. This is a limitation, but it is the pattern consistently emitted by both glslang and DXC when compiling GLSL/HLSL tessellation shaders to SPIR-V, so in practice it covers all real-world inputs.

Domain shader (tese)

Nothing special, entry point signature: [domain(...)] main(SPIRV_Cross_PatchConstant pc, const OutputPatch<SPIRV_Cross_Input, N> patch, float3/float2 domain : SV_DomainLocation).
Patch-decorated inputs copy from pc; per-CP inputs copy from patch[i] in a loop.

Other

• SPIRV_Cross_PatchConstant: shared struct between tesc/tese with patch-decorated vars + tess level builtins (SV_TessFactor / SV_InsideTessFactor), sized by domain (tri: 3/1, quad: 4/2, isoline: 2/0).
• get_tesc_tese_flags(): merges execution mode flags from the other tessellation stage, since SPIR-V allows domain/spacing/winding modes on either entry point.
• input_cp_count: derived from BuiltInPosition input array size, with fallback to scanning other per-CP input arrays, then output_vertices. Maybe there is a better way?
• Tess level builtins routed through SPIRV_Cross_PatchConstant
• gl_TessCoord assigned from SV_DomainLocation at entry.

[xen2]

It seems to work well enough on my use cases (tested with D3D11 on various shaders incl flat & PN, with and without adjacent edge average displacement.
I will come back with additional fixes later if I encounter issues on more complex shaders.

[HansKristian-Work]

I need to get to this at some later point when I actually have time ... 1ksloc PRs of new difficult features takes a ton of energy and time to digest.

[xen2]

No problem, I understand it is a big feature and it might take time to get improved/merged.
Thanks for your help!

[xen2]

FYI I am trying to rewrite it in a better way (getting rid of state machine, working directly at SPIR-V level, all changes local to spirv_hlsl.cpp).
It should be easier to review as it doesn't touch much of the existing emission code anymore.

If I succeed I will close this PR and open a new one.
Marking it as draft for now (to avoid merge) until I am sure we can discard it.

[xen2]

I changed how the TCS split works.

Old approach

• Hook into emit_instruction to track a phase (Setup → PerCP → Patch).
• Redirect output into two text buffers (tesc_per_cp_buffer, tesc_patch_buffer).
• After emit, fix up the text to remove the if (gl_InvocationID == 0) block (brace counting).

New approach

Work at the SPIR-V level instead:

1. Pre-scan the entry function CFG to find:
   • the gate selection header block,
   • the OpControlBarrier index inside that header,
   • the per-CP OpFunctionCall index inside that header.
2. For each of the two output functions (tesc_main and tesc_main_patch):
   • Save the header block state.
   • Truncate its ops list.
   • Change its terminator from Select to Direct.
   • Point next_block at:
     • the merge block for tesc_main (skips the if),
     • the then-block for tesc_main_patch (skips per-CP work).
   • Run the normal emit_block_chain.
   • Restore the header block.

Result

• No hooks in the emit path.
• No capture buffers.
• No phase tracking.
• No text post-processing.

Each pass is just the existing emit_block_chain on a small, local edit of the IR.

As a result, it should be much more straightforward to review.
Of course, it still does some SPIR-V processing to try to find the if (gl_InvocationID == 0) gate but nothing tricky like before.

Also added some simple unit test as bonus.

[xen2]

It seems some tests are failing, I will investigate.

edit: fixed now; however, still reviewing another issue (it works on my shaders but not after I run spirv-opt -Ohlsl on them)
edit2: OK now