Autodesk: [hdSt] Persistent run-time-populated MaterialX codegen cache #3661

erikaharrison-adsk · 2025-06-03T20:50:46Z

Description of Change(s)

This change extends the existing MaterialX shader registry (an in-memory cache) with a persistent on-disk cache as a performance optimization. If we've generated a MaterialX shader with a unique ID on a previous run of the application and encounter the same ID on a subsequent run, we avoid redoing the expensive codegen process and instead restore the necessary data from cached files.

The cache directory can be set with the HDST_MTLX_CODEGEN_CACHE_DIR_PATH environment variable or the hdstMtlxCodegenCacheDirPath render setting. It's left up to the application to manage the cache, such as:

determine where to place the cache directory;
cleaning the directory if the cached files become invalidated, e.g. due to upgrading the versions of USD and/or MaterialX;
cleaning up cached files over a certain age or once a storage limit has been reached.

For reference, the optimization reduces the duration of HdRenderIndex::SyncAll for https://github.com/usd-wg/assets/tree/main/full_assets/OpenChessSet from about 550ms to about 250ms in our measurements.

Link to proposal (if applicable)

N/A

Fixes Issue(s)

N/A

Checklist

I have created this PR based on the dev branch
I have followed the coding conventions
I have added unit tests that exercise this functionality (Reference:
testing guidelines)
I have verified that all unit tests pass with the proposed changes
I have submitted a signed Contributor License Agreement (Reference:
Contributor License Agreement instructions)

ppenenko · 2025-06-03T21:20:19Z

pxr/imaging/hd/instanceRegistry.h

+using HdInstanceKey = uint64_t;
+using HdInstanceRegistryMutex = std::mutex;
+using HdInstanceRegistryLock = std::unique_lock<HdInstanceRegistryMutex>;


Previously, these were typedefs in the templated class definition of HdInstance but they never depended on template parameters. They've also been referenced in the HdInstanceRegistry implementation below, where they had to be namespaced with HdInstance::. I'm extending the HdInstanceRegistry implementation in order to optionally support persistence, and these verbose typedefs were hurting the readability in that code, so I'm making them standalone.

ppenenko · 2025-06-03T21:22:12Z

pxr/imaging/hd/instanceRegistry.h

+    explicit HdInstance(HdInstanceKey           key,
+                        ValueType const         &value,
+                        HdInstanceRegistryLock  &&registryLock,
+                        REGISTRY                *registry)


Previously, it was enough for the instance to hold a pointer to the hash map container owned by the registry, because it would just directly put values in it. But now the registry can implement a custom persistence mechanism, so we need the instance to hold a pointer to the registry and call its methods instead.

ppenenko · 2025-06-03T21:29:36Z

pxr/imaging/hd/instanceRegistry.h

    /// is used to present a consistent interface to clients in cases
    /// where shared resource registration is disabled.
-    explicit HdInstance(KeyType const &key)
+    explicit HdInstance(HdInstanceKey key)


We know that the key is always just an integer, so it's better to pass it around by value.

ppenenko · 2025-06-03T21:35:00Z

pxr/imaging/hd/instanceRegistry.h

+/// The DERIVED template parameter can be used in a Curiously Recurring
+/// Template Pattern to extend this class with a persistent cache backing the
+/// in-memory dictionary.


BTW, this opens up the possibility to use the same pattern for other types of resources.

ppenenko · 2025-06-03T21:39:24Z

pxr/imaging/hd/instanceRegistry.h

+    /// Copy constructor.  Need as HdInstanceRegistryBase is placed in a map
+    /// and mutex is not copy-constructible, so can't use default


This comment doesn't seem to be correct: these objects are never stored by value in the USD codebase, so this constructor shouldn't be necessary and this class should comply with the rule of zero (right now, it doesn't: the copy operator is deleted below but it's copy-constructible).

ppenenko · 2025-06-03T21:42:34Z

pxr/imaging/hd/instanceRegistry.h

+        return it;
+    }
+
+    VALUE value = static_cast<DERIVED*>(this)->LoadFromDisk(key);


Loading from disk, if implemented by the derived class.

ppenenko · 2025-06-03T21:44:10Z

pxr/imaging/hdSt/CMakeLists.txt

+pxr_register_test(testHdStMaterialXShaderGen_testCacheSerialization
+    COMMAND "${CMAKE_INSTALL_PREFIX}/tests/testHdStMaterialXShaderGen --testCacheSerialization"
+    EXPECTED_RETURN_CODE 0
+    STDOUT_REDIRECT shadergen_testCacheSerialization.out
+    DIFF_COMPARE shadergen_testCacheSerialization.out
+    TESTENV testHdStMaterialXShaderGen
+)


A new test that tests if metadata is serialized correctly by the new MaterialX codegen cache. It reuses an existing test executable testHdStMaterialXShaderGen with a new command-line argument.

ppenenko · 2025-06-03T21:45:35Z

pxr/imaging/hdSt/materialNetwork.cpp

 #ifdef PXR_MATERIALX_SUPPORT_ENABLED
        if (!isVolume) {
-            _materialXGfx = HdSt_ApplyMaterialXFilter(&surfaceNetwork, materialId,
+            _materialXCodegenResult = HdSt_ApplyMaterialXFilter(&surfaceNetwork, materialId,


This used to return a shared pointer to a MaterialX::Shader object. Unfortunately, a MaterialX::Shader is not serializable, and source code is only one part of its state. So, to make caching feasible, I extract all the data that the downstream pipeline needs from MaterialX::Shader, encapsulate it in an object called "codegen result" and cache that to disk.

ppenenko · 2025-06-03T21:47:21Z

pxr/imaging/hdSt/materialXFilter.cpp

+    HdSt_MaterialParamVector const& fallbackParams =
+        mxCodegenResult.GetFallbackParams();


Fallback parameters are one necessary part of the codegen result.

ppenenko · 2025-06-03T21:48:06Z

pxr/imaging/hdSt/materialXFilter.cpp

-        // MaterialX parameter Information
-        const auto* variable = paramsBlock[i];
-        const auto varType = HdStMaterialXHelpers::GetMxTypeDesc(variable);
+    for (HdSt_MaterialParam const& fallbackParam : fallbackParams) {


The old code was populating the fallback parameters from the MaterialX shader.

In the new code, they are retrieved from MaterialX in the codegen result's constructor if codegen takes place at run time, or read from disk if the codegen happened on a previous run and was cached, and stored in the codegen result object.

Awesome! :)

ppenenko · 2025-06-03T21:48:38Z

pxr/imaging/hdSt/materialXFilter.cpp

-        // MaterialX glslfxShader. 
-        else {
-            std::string separator;
-            const auto varValue = variable->getValue();


All of this has moved to the codegen result's constructor.

ppenenko · 2025-06-03T21:49:04Z

pxr/imaging/hdSt/materialXFilter.cpp

-        if (!param.fallbackValue.IsEmpty()) {
-            materialParams->push_back(std::move(param));
-        }
+    for (TfToken textureParamName : mxCodegenResult.GetTextureParams()) {


Similarly, texture parameters are cached in the codegen result.

ppenenko · 2025-06-03T21:49:56Z

pxr/imaging/hdSt/materialXFilter.cpp

+    mx::ShaderPtr mxShader = HdSt_GenMaterialXShader(
        mtlxDoc, stdLibraries, searchPaths, mxHdInfo, apiName);
+
+    return std::make_shared<HdSt_MaterialXCodegenResult>(*mxShader);


Creating a codegen result from the MaterialX Shader, and destroying the latter because we've already extracted everything we will ever need from it.

ppenenko · 2025-06-03T21:51:26Z

pxr/imaging/hdSt/materialXFilter.cpp

+    // TfHashAppend hashes TfTokens not as strings, but as pointers to interned
+    // strings which are not stable from run to run
+    void _AppendPersistentHash(
+        Tf_HashState& hashState, TfToken const& token)
+    {
+        TfHashAppend(hashState, token.GetString());
+    }
+
+    // A VtValue may store a TfToken, in which case we need to convert it to a
+    // string in order to avoid the same pitfall as above
+    void _AppendPersistentHash(Tf_HashState& hashState, VtValue const& vtValue)
+    {
+        if (vtValue.IsHolding<TfToken>()) {
+            TfHashAppend(hashState, vtValue.Get<TfToken>().GetString());
+        } else {
+            TfHashAppend(hashState, vtValue.GetHash());
+        }
+    }


Without these workarounds, the cache always generated new, random hashes, and could never find an existing file in the cache directory.

I want to dig around internally and see if we have a function that does this already. If not, I may try to promote something like this to pxr/base. I think there are definitely some other cases where the TfHash isn't what you want for a fingerprint, though I'm not sure whether you'd hit them in a material network traversal.

ppenenko · 2025-06-03T21:52:10Z

pxr/imaging/hdSt/materialXFilter.cpp

+    HdSt_MaterialXShaderRegistry* const materialXShaderRegistry =
+        resourceRegistry->GetMaterialXShaderRegistry();


This is a specialized instance registry that supports a persistent cache. To minimize include dependencies, I expose it to clients of HdStResourceRegistry by pointer.

ppenenko · 2025-06-03T21:54:06Z

pxr/imaging/hdSt/materialXShaderRegistry.cpp

+TF_DEFINE_ENV_SETTING(HDST_MTLX_CODEGEN_CACHE_DIR_PATH, "",
+    "Path to the directory of the persistent MaterialX codegen cache");


Environment variable which can be used to set the path to the directory where the cached files are written. It serves as the default value for an HdSt render setting.

So as my guiding light for this kind of feature, I'm using the NVidia GL shader disk cache, which I think gets configured by environment variable or the magic nvidia control panel.

From Autodesk's perspective, what kind of control interface would you like for the cache? Env var? Something in C++ so you can put it in a settings panel? Plugin-based so people can configure weird site-specific multilevel caches or something? Curious to hear what the long term plans for something like this would be.

ppenenko · 2025-06-03T21:56:24Z

pxr/imaging/hdSt/materialXShaderRegistry.cpp

+                    bool val = false;
+                    issValue >> val;
+                    return VtValue(val);


These implementations mostly moved here from materialXFilter.cpp

ppenenko · 2025-06-03T21:57:47Z

pxr/imaging/hdSt/materialXShaderRegistry.cpp

+            { _tokens->name,    JsValue(param.name) },
+            { _tokens->type,    JsValue(param.fallbackValue.GetTypeName()) },
+            { _tokens->value,   JsValue(osValue.str()) } };


Serializing parameters to the JSON file.

ppenenko · 2025-06-03T21:59:22Z

pxr/imaging/hdSt/renderDelegate.cpp

+        HdRenderSettingDescriptor{
+            "Path to the directory of the persistent MaterialX codegen cache",
+            HdStRenderSettingsTokens->hdstMtlxCodegenCacheDirPath,
+            VtValue(HdSt_MaterialXShaderRegistry::GetCacheDirPathEnvSetting()) }


The cache directory exposed as a Storm render setting. The env var supplies the default value.

ppenenko · 2025-06-03T21:59:57Z

pxr/imaging/hdSt/renderDelegate.cpp

+        _HgiToResourceRegistryMap::GetInstance().GetOrCreateRegistry(
+            _hgi
+#ifdef PXR_MATERIALX_SUPPORT_ENABLED
+            , cacheDirPath.c_str()


Passing the render setting to the registry.

ppenenko · 2025-06-03T22:00:41Z

pxr/imaging/hdSt/resourceRegistry.cpp

+    , _materialXShaderRegistry(
+        std::make_unique<HdSt_MaterialXShaderRegistry>(mtlxCacheDirPath))


The MaterialX shader registry is now owned by unique pointer to simplify include dependencies.

ppenenko · 2025-06-03T22:01:25Z

pxr/imaging/hdSt/testenv/testHdStMaterialXShaderGen.cpp

+            TfToken(name), value);
+    };
+
+    addFallbackParam("BoolParamTrue", VtValue(true));


Testing the serialization/deserialization of all supported data types.

ppenenko · 2025-06-03T22:01:53Z

pxr/imaging/hdSt/testenv/testHdStMaterialXShaderGen.cpp

+        std::move(textureParams));
+
+    std::stringstream ss;
+    codegenResult0.SaveMetadata(ss);


Serializing the codegen result we've just populated to a string.

ppenenko · 2025-06-03T22:02:16Z

pxr/imaging/hdSt/testenv/testHdStMaterialXShaderGen.cpp

+
+    JsValue jsMetadata = JsParseStream(ss);
+
+    HdSt_MaterialXCodegenResult codegenResult1(


Deserializing into a new codegen result object.

ppenenko · 2025-06-03T22:02:41Z

pxr/imaging/hdSt/tokens.h

    ((stormMsaaSampleCount, "storm:msaaSampleCount"))

+#ifdef PXR_MATERIALX_SUPPORT_ENABLED
+#define HDST_MTLX_CODEGEN_CACHE_DIR_PATH_TOKEN (hdstMtlxCodegenCacheDirPath)


A token for the render setting's name.

jesschimein · 2025-06-04T17:03:21Z

Filed as internal issue #USD-11070

(This is an automated message. See here for more information.)

When testing [parallel MaterialX codegen in Storm](PixarAnimationStudios/OpenUSD#3567), and comparing the generated code between test runs using a [persistent MaterialX cache](PixarAnimationStudios/OpenUSD#3661), I noticed that the results were varying around `float` literal formatting. It turned out to be due to a data race on `static` variables controlling the formatting settings. This is similar to #2378 but limited to multithreaded codegen.

tcauchois

Good stuff! I think the MaterialXCodeGenResult refactor is landable now, but left some comments about the disk-backed instance registry.

tcauchois · 2025-12-04T00:20:48Z

pxr/imaging/hd/instanceRegistry.h

+public:
+    friend class HdInstanceRegistryBase<VALUE, HdInstanceRegistry<VALUE>>;
+
+    void SaveToDisk(


We're anticipating the need for a more sophisticated management layer for the cache, and think that should live outside of hdSt. We have a big requirements list for it (including stuff like location on disk, size limit, eviction policy, security concerns, fingerprinting, hooks for testing); we don't expect you to hit all of that, but we also don't think the instance registry is a good place to park that complexity, since it's just a bit of memoization code.

If the control flow works out (which it looks like it does) I think the idea of the instance registry calling out to the persistent cache seems neat, though! But I wonder if there's a way to get that extensibility without all of the recurrent templates.

tcauchois · 2025-12-04T00:23:04Z

pxr/imaging/hdSt/materialXFilter.cpp

-        // MaterialX parameter Information
-        const auto* variable = paramsBlock[i];
-        const auto varType = HdStMaterialXHelpers::GetMxTypeDesc(variable);
+    for (HdSt_MaterialParam const& fallbackParam : fallbackParams) {


Awesome! :)

tcauchois · 2025-12-04T00:25:24Z

pxr/imaging/hdSt/materialXFilter.cpp

+    // TfHashAppend hashes TfTokens not as strings, but as pointers to interned
+    // strings which are not stable from run to run
+    void _AppendPersistentHash(
+        Tf_HashState& hashState, TfToken const& token)
+    {
+        TfHashAppend(hashState, token.GetString());
+    }
+
+    // A VtValue may store a TfToken, in which case we need to convert it to a
+    // string in order to avoid the same pitfall as above
+    void _AppendPersistentHash(Tf_HashState& hashState, VtValue const& vtValue)
+    {
+        if (vtValue.IsHolding<TfToken>()) {
+            TfHashAppend(hashState, vtValue.Get<TfToken>().GetString());
+        } else {
+            TfHashAppend(hashState, vtValue.GetHash());
+        }
+    }


I want to dig around internally and see if we have a function that does this already. If not, I may try to promote something like this to pxr/base. I think there are definitely some other cases where the TfHash isn't what you want for a fingerprint, though I'm not sure whether you'd hit them in a material network traversal.

tcauchois · 2025-12-04T00:29:29Z

pxr/imaging/hdSt/materialXShaderRegistry.cpp

+TF_DEFINE_ENV_SETTING(HDST_MTLX_CODEGEN_CACHE_DIR_PATH, "",
+    "Path to the directory of the persistent MaterialX codegen cache");


So as my guiding light for this kind of feature, I'm using the NVidia GL shader disk cache, which I think gets configured by environment variable or the magic nvidia control panel.

From Autodesk's perspective, what kind of control interface would you like for the cache? Env var? Something in C++ so you can put it in a settings panel? Plugin-based so people can configure weird site-specific multilevel caches or something? Curious to hear what the long term plans for something like this would be.

Persistent run-time-populated MaterialX codegen cache

a9b4f1f

ppenenko reviewed Jun 3, 2025

View reviewed changes

erikaharrison-adsk marked this pull request as ready for review June 4, 2025 15:41

This was referenced Jun 11, 2025

Ensure determinism of ShaderGen with color and unit transforms AcademySoftwareFoundation/MaterialX#2378

Merged

Fix race in format settings for multithreaded codegen AcademySoftwareFoundation/MaterialX#2454

Merged

tcauchois reviewed Dec 4, 2025

View reviewed changes

		/// Copy constructor. Need as HdInstanceRegistryBase is placed in a map
		/// and mutex is not copy-constructible, so can't use default

		HdSt_MaterialParamVector const& fallbackParams =
		mxCodegenResult.GetFallbackParams();

		HdSt_MaterialXShaderRegistry* const materialXShaderRegistry =
		resourceRegistry->GetMaterialXShaderRegistry();

		TF_DEFINE_ENV_SETTING(HDST_MTLX_CODEGEN_CACHE_DIR_PATH, "",
		"Path to the directory of the persistent MaterialX codegen cache");

		, _materialXShaderRegistry(
		std::make_unique<HdSt_MaterialXShaderRegistry>(mtlxCacheDirPath))


		JsValue jsMetadata = JsParseStream(ss);

		HdSt_MaterialXCodegenResult codegenResult1(

Autodesk: [hdSt] Persistent run-time-populated MaterialX codegen cache #3661

Are you sure you want to change the base?

Autodesk: [hdSt] Persistent run-time-populated MaterialX codegen cache #3661

Uh oh!

Conversation

erikaharrison-adsk commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Change(s)

Link to proposal (if applicable)

Fixes Issue(s)

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ppenenko Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jesschimein commented Jun 4, 2025

Uh oh!

tcauchois left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

erikaharrison-adsk commented Jun 3, 2025 •

edited

Loading

ppenenko Jun 3, 2025 •

edited

Loading