HLSL: support SPIR-V library modules (no OpEntryPoint, only Export-decorated functions)

[castano]

spirv-cross --hlsl cannot consume SPIR-V "library" modules, that is, modules with OpCapability Linkage, no OpEntryPoint, and one or more functions decorated with OpDecorate LinkageAttributes "" Export. Such modules are produced by dxc -T lib_6_* and are useful for distributing pre-compiled shader libraries that callers can include from HLSL/GLSL source.

Today, parsing fails immediately:

SPIRV-Cross threw an exception: There is no entry point in the SPIR-V module.

at spirv_parser.cpp:155.

Repro:

// lib.hlsl
static const uint table[4] = { 10u, 20u, 30u, 40u };

export uint lookup(uint i) {
    return table[i];
}

dxc -T lib_6_3 -spirv -fspv-target-env=universal1.5 lib.hlsl -Fo lib.spv
spirv-cross --hlsl lib.spv
# SPIRV-Cross threw an exception: There is no entry point in the SPIR-V module.

(For the constant table to actually carry its values into the SPV, a separate DXC fix is needed tracked at microsoft/DirectXShaderCompiler#8425. The spirv-cross issue here is independent, even with that fix, spirv-cross can't process the module because there's no entry point.)

Desired Behavior:

When the module has no entry point but does have Export-decorated functions, emit each exported function as a top-level free function in the target language (HLSL/GLSL), with no [numthreads]/main() boilerplate. Output should be includable as a header by HLSL/GLSL source code.

Expected output for the repro:

static const uint table[4] = { 10u, 20u, 30u, 40u };

uint lookup(uint i) {
    return table[i];
}

Motivation:

I'd like to distribute the Spark codecs as SPIR-V libraries that users can transform to HLSL/GLSL header-only libraries and include in their shader code.

Proposed approach:

1. Parser (spirv_parser.cpp): collect function IDs decorated with LinkageAttributes ... Export. If no OpEntryPoint exists but exports do, synthesize a stub entry point pointing at the first export (model GLCompute) so the rest of the pipeline (analyses keyed on default_entry_point) keeps working. Mark the IR as a library module via a new is_library_module flag on ParsedIR. Fall back to the linkage name when OpName was stripped (e.g. by spirv-opt --strip-debug).
2. CFG analysis (spirv_cross.cpp): extend build_function_control_flow_graphs_and_analyze to walk each library export's call tree (not just default_entry_point's).
3. HLSL emitter (spirv_hlsl.cpp): in library mode, replace the emit_function(default_entry_point) + emit_hlsl_entry_point() calls with a loop emitting each library export. Skip the main rename. Patch emit_function_prototype so the first export, which happens to be default_entry_point for analysis purposes, is emitted as a normal function, not as the entry point.
4. Reachability filter for declarations: in library mode, build a set of IDs referenced anywhere in the SPIR-V stream (excluding each instruction's own result id). Gate emit_specialization_constants_and_structs and emit_composite_constants on that set so unused constants/undefs/specializations don't leak into the output.

I have a working implementation locally on top of vulkan-sdk-1.4.341.0. The HLSL pipeline produces clean, includable output for all our shaders. I'd be happy to open a PR. Would you be open to reviewing it? Or would you suggest a different approach?

[HansKristian-Work]

This seems useful and the proposed approach makes sense. Thanks for checking in beforehand how it could land upstream. If this can be split up into multiple clean commits or PRs that do things one step at a time that would greatly accelerate the reviewing process I think. You've already split it up into 4 steps in the proposal and doing it like that makes sense.

[castano]

Good call. Splitting up the changes made me realize the reachability filter is not actually necessary, so this keeps things simpler. I'm gonna do some more testing and maybe work on the GLSL output next.

[castano]

One problem with the GLSL output is that we currently emit #version and #extension directives that would be invalid in an include file (unless it's included before anything else). My first approach was to simply skip them when producing libraries. A problem with that is that the produced code may depend on SPIRV_CROSS_ definitions that are now missing and that the user would have to provide. What do you think is the best way to go? For my particular application I would prefer bare headers, but the missing definitions may be surprising to users. Maybe I should emit the code commented out, or under an #ifdef block.

[HansKristian-Work]

The version isn't super important, but extensions could be. Unfortunately GLSL requires that extensions are declared early in the module in many cases, so the most practical option is to emit the library as:

#ifdef SPIRV_CROSS_LIBRARY_HEADER
#version 450
#extension ...
#endif

Consumers of the library header can just ignore it if it works, or it can do simple text parsing to fish out the required extensions and paste them into the amalgamated source.

[castano]

Yeah, it's not an ideal solution, but I can't think of anything better. I'll update the glsl backend to emit it like that.

[castano]

One issue that I'm running into when using global resources is that StructuredBuffers are output as ByteAddressBuffers, so:

struct TablesStruct {
    uint integer_of_trits_q192[256];
    uint integer_of_trits_q48[256];
};
StructuredBuffer<TablesStruct> TablesBuffer : register(SPK_HLSL_ASTC_TABLE_REGISTER);

turns into:

struct TablesStruct
{
    uint integer_of_trits_q192[256];
    uint integer_of_trits_q48[256];
};

ByteAddressBuffer TablesBuffer : register(t3, space0);

Is this by design? I wonder we could infer that this was originally a structured buffer. (Found the solution for this already.)

I also wonder if there's a way to parameterize the bindings. In the original sources the register binding is configurable by the user through a preprocessor definition, but in the compiled spv it is baked to the default value and that is what spirv-cross picks up. This may result in conflicts with the user bindings.

One idea is to remove the DescriptorSet/Binding decorations from the spv library and have spirv-cross emit configurable bindings incrementally in that case:

#ifndef SPIRV_CROSS_TablesBuffer_DescriptorSet
#define SPIRV_CROSS_TablesBuffer_DescriptorSet 0
#endif
#ifndef SPIRV_CROSS_TablesBuffer_Binding
#define SPIRV_CROSS_TablesBuffer_Binding 0
#endif
ByteAddressBuffer TablesBuffer : register(SPIRV_CROSS_TablesBuffer_Binding, SPIRV_CROSS_TablesBuffer_DescriptorSet);

The default value for the descriptor set and the base value for the register binding could be configurable through command line options.

[castano]

Never mind, I just found about the --hlsl-preserve-structured-buffers option that does exactly what I need. One caveat is that dxc only emits UserTypeGOOGLE decorations when using StructuredBuffers, but not when using the old tbuffer syntax. I can live with that, though.

[castano]

So far things are looking pretty good on my end, the generated libraries seem to run correctly!

There are only some minor issues remaining. One is these static variables declared globally:

static uint16_t _433;
static uint16_t _434;
static uint16_t _435;
static uint4 _456;

These seem to correspond to uninitialized variables on branches that I know the shader won't take. Not a problem in standalone shaders, but with shader libraries you may run into conflicting declarations from other included shaders. It doesn't happen on my shaders, but it would be good to prevent that.

Another problem is that in order to emit globals I had to skip this test in Compiler::is_hidden_variable:

	if (ir.get_spirv_version() >= 0x10400 && !ir.is_library_module && var.storage != StorageClassGeneric &&
	    var.storage != StorageClassFunction && !interface_variable_exists_in_entry_point(var.self))
	{
		return true;
	}

I have to dig deeper to understand why this is only in spirv_version 1.4 and what are the interface variables. I'll take a closer look next week.

After that I'll have to make sure there are no performance regressions with respect to the original shaders.

[HansKristian-Work]

In SPIR-V 1.4 global variables must be declared by OpEntryPoint's interface list, despite being non-input, non-output. SPIR-V pre 1.4 only requires inputs/outputs to be declared, and globals are implied through static use. I'm not sure how this is intended to work for libraries since there are no entry points. Most likely, interface_variable_exists_in_entry_point needs to use old style path if export path is used or something.