Update gpurt from commit dee2554b

Export option for persistent launch
Fix validation when using disk mapping
CPS Hit Object refactor
StackCommon: add the ability to discard data
Add intrinsic for atomic conditional sub on LDS
Add HPLOC support
Move EncodeTopLevelCommon to shadersClean
Add missing include to EncodeTopLevelCommon.hlsli
Use an InlineBuffer for the DispatchRaysConstantData to remove one level of indirection
Fix validation with symlinks/mounts
Enable the no-parameter-usage warning in shader library
Fix performance drop in persistent workgroup
StackCommon: fix DECLARE_VALUE_SET_I32 signature
Update the readme
[NFC] Refactor GpuRt related files
Move BuildCommon.hlsl to shadersClean/
FixLaneGroup.hlsli Validation / Add BitOr64 LaneGroup Intrinsic
[NFC] Move DispatchRaysConstBuf and ray pipeline flags codes into shadersClean
New FVM Morton Code Generator
Change a few ternaries over to select()
Refactor EncodeTriangleNode function body
StackCommon: manually declare intrinsics.
[NFC] traversal: Move ray state related structs into shadersClean
Remove PASS_HIT_OBJECT_ARG define
BuildCommonScratch(/Global) and TaskCounter
Refactor RayQuery related files and functions
DGF: Preliminary support for DGF compressed geometry in BVH builds
Add an Unbiased Origin  workaround that can be set from driver
Add a shader to support Vulkan CaptureReplay feature

Author: qiaojbao

README.md

@@ -1,25 +1,44 @@
-### GPU Ray Tracing Library
+# GPU Ray Tracing Library
 
 The GPU Ray Tracing (GPURT) library is a static library (source deliverable) that provides ray tracing related functionalities for AMD drivers supporting DXR (DirectX 12®) and the Vulkan® RT API. The GPURT library is built on top of AMD's Platform Abstraction Library (PAL). Refer to PAL documentation for information regarding interfaces used by GPURT for various operations including command generation, memory allocation etc.
 
-GPURT uses a C++ interface. The public interface is defined in .../gpurt/gpurt, and clients must only include headers from that directory. The interface is divided into multiple header files based on their dependencies and usage.
-
-* gpurt/gpurt.h
-  * Provides definiton of the public GPURT device interface used to perform various ray tracing operations and querying traversal shader code.
-* gpurt/gpurtAccelStruct.h
-  * Provides definition of the GPURT acceleration structure produced by the GPURT acceleration structure builder.
-* gpurt/gpurtCounter.h
-  * Shared header file between C++ and HLSL code that provides the definitions of GPURT counters and related data structures.
+GPURT provides the majority of non-compiler functionality required to support DX12 and Vulkan raytracing APIs.
 
 The primary functionality provided by GPURT includes the following:
 
 * Building acceleration structures.
 * Traversal loop and associated intrinsic functions written in HLSL.
+  * Ray tracing traversal counter and ray history data capture.
 * Acceleration structure capture and decode.
-* Ray tracing traversal counter and ray history data capture.
+
+## Code Style
 
 GPURT follows [PAL Coding Standards](https://github.com/GPUOpen-Drivers/pal/blob/dev/doc/process/palCodingStandards.md) whenever possible.
 
+## Code Organization
+
+We are moving code from the unorganized src/shaders/ directory to the src/shadersClean/ directory. Within src/shadersClean the code is organized as follows:
+
+| Directory | Description |
+| --------- | ----------- |
+| _build_ | Code related to BuildRaytracingAccelerationStructure and vkCmdBuildAccelerationStructuresKHR |
+| _traversal_ | Code related to DispatchRays & vkCmdTraceRaysKHR. |
+| _common_ | Code common to build and traversal. |
+| _debug_ | Code related to implementing GPU_ASSERT. |
+
+Code in the src/shadersClean directory undergoes validation. Before compiling shader entrypoints, each HLSL file in src/shadersClean is compiled separately (results are discarded) to check if it has no implicit dependencies. HLSL files can only include HLSLI files (or C++ Interfaces) and HLSLI files can only include other HLSLI files (or C++ Interfaces).
+
+### C++ Interfaces
+
+GPURT uses a C++ interface. The public interface is defined in .../gpurt/gpurt, and clients must only include headers from that directory. The interface is divided into multiple header files based on their dependencies and usage.
+
+* gpurt/gpurt.h
+  * Provides definition of the public GPURT device interface used to perform various ray tracing operations and querying traversal shader code.
+* gpurt/gpurtAccelStruct.h
+  * Shared header file between C++ and HLSL that provides the definition of the GPURT acceleration structure produced by the GPURT acceleration structure builder.
+* gpurt/gpurtCounter.h
+  * Shared header file between C++ and HLSL code that provides the definitions of GPURT counters and related data structures.
+
 ## Acceleration Structure
 
 GPURT interfaces support various acceleration structure operations including:

cmake/GpuRtGenerateShaders.cmake

@@ -1,7 +1,7 @@
 ##
  #######################################################################################################################
  #
- #  Copyright (c) 2022-2024 Advanced Micro Devices, Inc. All Rights Reserved.
+ #  Copyright (c) 2022-2025 Advanced Micro Devices, Inc. All Rights Reserved.
  #
  #  Permission is hereby granted, free of charge, to any person obtaining a copy
  #  of this software and associated documentation files (the "Software"), to deal
@@ -67,6 +67,13 @@ set(gpurtTraceShadersSpirv "${gpurtOutputDir}/g_GpuRtLibrary_spv.h")
 
 set(gpurtDebugInfoFile "${CMAKE_CURRENT_BINARY_DIR}/g_gpurtDebugInfo.h")
 
+    # Find binaries in PATH
+    find_program(gpurtDxcCompiler dxc REQUIRED)
+    find_program(gpurtSpirvRemap spirv-remap REQUIRED)
+    # Find dxcompiler library.
+    get_filename_component(gpurtDxcCompilerDirectory "${gpurtDxcCompiler}" DIRECTORY)
+    find_library(gpurtDxcompilerLib dxcompiler HINTS ${gpurtDxcCompilerDirectory} /usr/lib/dxc REQUIRED)
+
 set(originalShaderSourceDir "${GPU_RAY_TRACING_SOURCE_DIR}/src/shaders/")
 set(originalShaderSource ${GPURT_SHADER_SOURCE_FILES})
 list(TRANSFORM originalShaderSource PREPEND "${originalShaderSourceDir}")
@@ -136,6 +143,7 @@ if(GPURT_CLIENT_API STREQUAL "VULKAN")
             ${gpurtStripWhitelist}
             ${gpurtDxcCompiler}
             ${gpurtSpirvRemap}
+
         COMMAND ${RT_SHADER_VALIDATION_COMMAND}
 
         COMMAND Python3::Interpreter "${gpurtCompileScript}"

gpurt/gpurt.h

@@ -84,7 +84,7 @@ enum class StaticPipelineFlag : uint32
 {
     SkipTriangles                  = 0x100,        // Always skip triangle node intersections
     SkipProceduralPrims            = 0x200,        // Always skip procedural node intersections
-    Unused                         = (1u << 31),   // Available for use
+    UseUnbiasedOrigin              = (1u << 31),   // Avoid biasing the origin by TMin
     UseTreeRebraid                 = (1u << 30),   // Use Tree Rebraid for TraceRays
     EnableAccelStructTracking      = (1u << 29),   // Enable logging of TLAS addresses using AccelStructTracker
     EnableTraversalCounter         = (1u << 28),   // Enable Traversal counters
@@ -277,9 +277,10 @@ struct NodeMapping
 // Structure containing shader code in various intermediate forms.
 struct PipelineShaderCode
 {
+#if GPURT_CLIENT_INTERFACE_MAJOR_VERSION < 55
     const void* pAmdilCode; // Code in AMD intermediate language form
     size_t      amdilSize;  // Size in bytes of AMDIL code
-
+#endif
     const void* pDxilCode;  // Code in DXIL form
     size_t      dxilSize;   // Size in bytes of DXIL code
 
@@ -354,6 +355,7 @@ enum class InternalRayTracingCsType : uint32
 #endif
     BuildFastAgglomerativeLbvh,
     EncodeQuadNodes,
+    BuildHPLOC,
 #if GPURT_BUILD_RTIP3_1
     BuildTrivialBvh,
     BuildSingleThreadGroup32,
@@ -366,6 +368,8 @@ enum class InternalRayTracingCsType : uint32
     Update3_1,
     RefitInstanceBounds,
 #endif
+    EncodeDGF,
+    PrepareShadowSbtForReplay,
     Count
 };
 
@@ -509,8 +513,10 @@ typedef uint32 GeometryFlags;
 // Type of geometry node
 enum class GeometryType : uint32
 {
-    Triangles = 0,      // Triangle geometry.  Geometry::triangles is valid.
-    Aabbs               // Procedural bounding box geometry.  Geometry::aabbs is valid.
+    Triangles              = 0, // Triangle geometry.  Geometry::triangles is valid.
+    Aabbs                  = 1, // Procedural bounding box geometry.  Geometry::aabbs is valid.
+    CompressedTriangles    = 2, // Compressed triangle geometry.  Geometry::compressedTriangles is valid.
+    CompressedTrianglesOmm = 3, // Compressed triangle geometry with OMM.  Geometry::compressedTriangles is valid.
 };
 
 // Index format for triangle geometry.
@@ -540,6 +546,11 @@ enum class VertexFormat : uint32
     R8G8_Unorm          // 8-bit fixed-point unsigned normalized R8G8 X,Y,0 format
 };
 
+enum class CompressedTriangleFormat : uint32
+{
+    Dgf1, // Dense Geometry Format version 1
+};
+
 // Geometry node triangle data
 struct GeometryTriangles
 {
@@ -561,6 +572,18 @@ struct GeometryAabbs
     gpusize         aabbByteStride; // Stride in bytes between consecutive bounding boxes
 };
 
+// Geometry node compressed triangle data
+struct GeometryCompressedTriangles
+{
+    gpusize                  compressedDataAddr;
+    gpusize                  compressedDataSize;
+    uint32                   numTriangles;
+    uint32                   numVertices;
+    uint32                   maxPrimitiveIndex;
+    uint32                   maxGeometryIndex;
+    CompressedTriangleFormat format;
+};
+
 // Bottom-level geometry node information (analogous to D3D12DDI_RAYTRACING_GEOMETRY_DESC).
 // This API-independent structure does not match D3D12 or Vulkan. The client must convert the API version to the
 // GPURT version through ClientConvertAccelStructBuildGeometry().
@@ -570,8 +593,10 @@ struct Geometry
     GeometryFlags flags;                // Geometry flags
     union
     {
-        GeometryTriangles triangles;    // Triangle geometry.  Valid if type is Triangles.
-        GeometryAabbs     aabbs;        // Procedural AABB geometry.  Valid if type is Aabbs.
+        GeometryTriangles           triangles;           // Triangle geometry.  Valid if type is Triangles.
+        GeometryAabbs               aabbs;               // Procedural AABB geometry.  Valid if type is Aabbs.
+        GeometryCompressedTriangles compressedTriangles; // Compressed triangle geometry.
+                                                         // Valid if type is CompressedTriangles.
     };
 };
 
@@ -600,14 +625,14 @@ enum class InputElementLayout : uint32
 enum class BvhBuildMode : uint32
 {
     Linear   = 0, // Linear BVH builder
-    Reserved = 1, // Formerly agglomerative clustering BVH builder
+    HPLOC    = 1, // Hierarchical PLOC
     PLOC     = 2, // Parallel locally-ordered clustering BVH builder
     Auto     = 4, // Used in override build to fall back to regular build options
     Count
 };
 
 static_assert(uint32(BvhBuildMode::Linear)   == 0, "Enums encoded in the acceleration structure must not change.");
-static_assert(uint32(BvhBuildMode::Reserved) == 1, "Enums encoded in the acceleration structure must not change.");
+static_assert(uint32(BvhBuildMode::HPLOC)    == 1, "Enums encoded in the acceleration structure must not change.");
 static_assert(uint32(BvhBuildMode::PLOC)     == 2, "Enums encoded in the acceleration structure must not change.");
 
 // BVH CPU builder modes
@@ -789,6 +814,7 @@ struct DeviceSettings
 #endif
 
     uint32                      plocRadius;                           // PLOC nearest neighbor search adius
+    uint32                      hplocRadius;                          // HPLOC nearest neighbor search radius
 #if GPURT_CLIENT_INTERFACE_MAJOR_VERSION < 54
     uint32                      maxTopDownBuildInstances;             // Max instances allowed for top down build
 #endif
@@ -863,6 +889,7 @@ struct DeviceSettings
         uint32 disableRdfCompression : 1;                   // Disable compression in RDF chunks
         uint32 enableRebraid : 1;                           // Enable tree rebraid in TLAS
         uint32 cullIllegalInstances : 1;
+        uint32 useUnbiasedOrigin : 1;                       // Avoids offsetting by tMin in traversal
     };
 
     uint64                      accelerationStructureUUID;  // Acceleration Structure UUID
@@ -1742,6 +1769,16 @@ class IDevice
 
     virtual uint32 CalculateBvhPrimitiveCount(const AccelStructBuildInputs& inputs) const = 0;
 
+    // Prepares shadow shader binding table for replay (Vulkan capture replay feature)
+    //
+    // @param cmdBuffer            [in/out] Opaque handle to command buffer where commands will be written
+    // @param userData             [in] Addresses of input/output buffers
+    // @param totalSbtEntryCount   Total number of SBT entries
+    virtual void PrepareShadowSbtForReplay(
+        ClientCmdBufferHandle                    cmdBuffer,
+        const PrepareShadowSbtForReplayUserData& userData,
+        uint32                                   totalSbtEntryCount) = 0;
+
 protected:
 
     /// Client must create objects by explicitly calling CreateDevice method

gpurt/gpurtCounter.h

@@ -432,6 +432,23 @@ GPURT_STATIC_ASSERT(INSTANCE_INPUTS_STRIDE_OFFSET == offsetof(InstanceInputHeade
 GPURT_STATIC_ASSERT(INSTANCE_INPUTS_INSTANCES_OFFSET == offsetof(InstanceInputHeader, instances), "");
 GPURT_STATIC_ASSERT(sizeof(InstanceInputHeader) == INSTANCE_INPUTS_HEADER_SIZE, "Instance inputs header mismatch.");
 
+// ====================================================================================================================
+struct BuildInputDescHeader
+{
+    uint32_t type;  // Triangles, AABBs or Instances
+    uint32_t flags; // Geometry flags
+    union
+    {
+        TriangleInputHeader triangles;
+        AABBInputHeader     aabbs;
+        InstanceInputHeader instances;
+    };
+};
+
+#define BUILD_INPUT_DESC_TYPE_TRIANGLES         0
+#define BUILD_INPUT_DESC_TYPE_AABBS             1
+#define BUILD_INPUT_DESC_TYPE_INSTANCES         2
+
 // ====================================================================================================================
 // 32-bit unique ray identifier calculated as below.
 //      uint32_t id = threadID.x + (threadID.y * dim.x) + (threadID.z * dim.x * dim.y);

gpurt/gpurtDispatch.h

@@ -47,14 +47,6 @@ constexpr uint32 MaxBufferSrdSize = 8;
 constexpr uint32 MaxBufferSrdSize = 4;
 #endif
 
-// Dispatch rays arguments top-level descriptor table (GPU structure)
-struct DispatchRaysTopLevelData
-{
-    uint64 dispatchRaysConstGpuVa;                  // DispatchRays info constant buffer GPU VA
-    uint32 internalUavBufferSrd[MaxBufferSrdSize];  // Internal UAV shader resource descriptor
-    uint32 accelStructTrackerSrd[MaxBufferSrdSize]; // Structured buffer SRD pointing to the accel struct tracker
-};
-
 #define DISPATCHRAYSCONSTANTDATA_STRUCT_OFFSET_DISPATCHID  48
 
 // Dispatch rays constant buffer data (GPU structure). Note, using unaligned uint64_t in HLSL constant buffers requires
@@ -108,11 +100,25 @@ struct DispatchRaysConstantData
 #pragma pack(pop)
 #endif
 
+// Dispatch rays arguments top-level descriptor table (GPU structure)
+struct DispatchRaysTopLevelData
+{
+#if GPURT_CLIENT_INTERFACE_MAJOR_VERSION < 56
+    uint64                   dispatchRaysConstGpuVa;                  // DispatchRays info constant buffer GPU VA
+#else
+    DispatchRaysConstantData constData;                               // Dispatch rays constant buffer data
+#endif
+    uint32                   internalUavBufferSrd[MaxBufferSrdSize];  // Internal UAV shader resource descriptor
+    uint32                   accelStructTrackerSrd[MaxBufferSrdSize]; // Structured buffer SRD pointing to the accel struct tracker
+};
+
 // GPU structure containing all data for DXR/VK ray dispatch command
 struct DispatchRaysConstants
 {
     DispatchRaysTopLevelData descriptorTable;  // Top-level internal dispatch bindings (includes pointer to infoData)
+#if GPURT_CLIENT_INTERFACE_MAJOR_VERSION < 56
     DispatchRaysConstantData constData;        // Dispatch rays constant buffer data
+#endif
 };
 
 #if __cplusplus
@@ -188,7 +194,37 @@ struct InitExecuteIndirectConstants
     uint32 cpsGlobalMemoryAddressHi;    // Separate CPS stack memory base address high 32-bits
 };
 
+// Resource bindings required for PrepareShadowSbtForReplay
+struct PrepareShadowSbtForReplayUserData
+{
+    uint64 constantsVa;                  // PrepareShadowSbtForReplayConstants struct
+    uint64 shadowRayGenerationTableVa;   // Shadow ray generation table
+    uint64 shadowHitGroupTableVa;        // Shadow hit group table
+    uint64 shadowMissTableVa;            // Shadow miss shader table
+    uint64 shadowCallableTableVa;        // Shadow callable shader table
+    uint64 captureReplayMappingBufferVa; // Capture replay mapping buffer
+};
+
+// Constants for PrepareShadowSbtForReplay shader
+struct PrepareShadowSbtForReplayConstants
+{
+    uint32 hitGroupTableStrideInBytes;           // Hit group table record byte stride
+    uint32 hitGroupTableEntryCount;              // Hit group table entry count
+    uint32 missTableStrideInBytes;               // Miss shader table record byte stride
+    uint32 missTableEntryCount;                  // Miss shader table entry count
+    uint32 callableTableStrideInBytes;           // Callable shader table record byte stride
+    uint32 callableTableEntryCount;              // Callable shader table entry count
+    uint32 captureReplayMappingBufferEntryCount; // Capture replay mapping buffer entry count
+};
+
+struct CaptureReplayMappingBufferEntry
+{
+    uint32 capturedVa;
+    uint32 replayVa;
+};
+
 constexpr uint32 InitExecuteIndirectConstantsDw = sizeof(InitExecuteIndirectConstants) / sizeof(uint32);
+constexpr uint32 PrepareShadowSbtForReplayConstantsDw = sizeof(PrepareShadowSbtForReplayConstants) / sizeof(uint32);
 
 #if __cplusplus
 #if GPURT_CLIENT_INTERFACE_MAJOR_VERSION >= 47

gpurt/gpurtLib.h

@@ -42,7 +42,7 @@ namespace GpuRt
 // update their definition of GPURT_CLIENT_INTERFACE_MAJOR_VERSION to indicate that they have made the required changes
 // to support a new version. When the client version is updated, the old interface will be compiled out and only the
 // new one will remain.
-#define GPURT_INTERFACE_MAJOR_VERSION 54
+#define GPURT_INTERFACE_MAJOR_VERSION 56
 
 #if GPURT_CLIENT_INTERFACE_MAJOR_VERSION < 44
 // Minor interface version. This number is incrememnted when a compatible interface change is made. Compatible changes

src/gpurtBvhBatcher.cpp

@@ -461,6 +461,11 @@ void BvhBatcher::BuildMultiDispatch(Util::Span<BvhBuilder> builders)
         Barrier();
         BuildPhase(BuildPhaseFlags::BuildFastAgglomerativeLbvh, builders, &BvhBuilder::BuildFastAgglomerativeLbvh);
     }
+    if (PhaseEnabled(BuildPhaseFlags::BuildHPLOC))
+    {
+        Barrier(BarrierFlagSyncIndirectArg | BarrierFlagSyncDispatch);
+        BuildPhase(BuildPhaseFlags::BuildHPLOC, builders, &BvhBuilder::BuildHPLOC);
+    }
     if (PhaseEnabled(BuildPhaseFlags::BuildPLOC))
     {
         Barrier();