Deadlock in `_PyResolverScopedCache::Enter()`

[johnhaddon]

We're encountering a deadlock in _PyResolverScopedCache::Enter(), where it calls _GetResolver(). I believe this happens when the following sequence occurs :

1. Thread A running Python calls _PyResolverScopedCache::Enter(), and then enters ArGetResolver() without releasing the GIL.
2. At the same time on thread B, a direct call to ArGetResolver() is made from C++. This one gets to construct the _static _DispatchingResolver, holding a lock while it does so.
3. Creation of the dispatcher on thread B calls through to PlugPlugin::Load(), which in turn calls TfScriptModuleLoader() to load associated Python bindings. This attempts to acquire the GIL, but fails because it is held by Thread A.
4. Meanwhile, thread A is blocked on the construction of the _static _DispatchingResolver.
5. Deadlock.

If this analysis is correct, then I think the problem should be solvable by releasing the GIL temporarily in _PyResolverScopedCache::Enter(). Does that make sense?

Stack traces for two real-world threads matching the above description can be found below :

Thread A

Thread 25 (Thread 0x7f6c9cff9640 (LWP 346385) "python"):
#0  0x00007f6ceb90773d in syscall () from /lib64/libc.so.6
#1  0x00007f6cec25281e in __cxxabiv1::__cxa_guard_acquire (g=0x7f6cc404c968 <guard variable for pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_GetResolver()::resolver>) at ../../../../libstdc++-v3/libsupc++/guard.cc:319
#2  0x00007f6cc4037997 in pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_GetResolver() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#3  0x00007f6cc4038329 in pxrInternal_v0_25_5__pxrReserved__::ArGetResolver() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#4  0x00007f6cc403ceb7 in pxrInternal_v0_25_5__pxrReserved__::ArResolverScopedCache::ArResolverScopedCache() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#5  0x00007f6cc3fe37ec in (anonymous namespace)::_PyResolverScopedCache::Enter() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/python/pxr/Ar/_ar.so
#6  0x00007f6cc3fe3b63 in pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::objects::caller_py_function_impl<pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::caller<void ((anonymous namespace)::_PyResolverScopedCache::*)(), pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::default_call_policies, pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::type_list<void, (anonymous namespace)::_PyResolverScopedCache&> > >::operator()(_object*, _object*) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/python/pxr/Ar/_ar.so
#7  0x00007f6cc7cf9e9d in pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::objects::function::call(_object*, _object*) const () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#8  0x00007f6cc7cfa078 in std::_Function_handler<void (), std::reference_wrapper<pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::objects::(anonymous namespace)::bind_return> >::_M_invoke(std::_Any_data const&) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#9  0x00007f6cc7cf2705 in pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::exception_handler::operator()(std::function<void ()> const&) const () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#10 0x00007f6cc725840b in std::_Function_handler<bool (pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::exception_handler const&, std::function<void ()> const&), std::_Bind_result<bool, pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::translate_exception<pxrInternal_v0_25_5__pxrReserved__::TfBaseException, void (*)(pxrInternal_v0_25_5__pxrReserved__::TfBaseException const&)> (std::_Placeholder<1>, std::_Placeholder<2>, void (*)(pxrInternal_v0_25_5__pxrReserved__::TfBaseException const&))> >::_M_invoke(std::_Any_data const&, pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::exception_handler const&, std::function<void ()> const&) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/python/pxr/Tf/_tf.so
#11 0x00007f6cc7cf26a3 in pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::handle_exception_impl(std::function<void ()>) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#12 0x00007f6cc7cf7dca in function_call () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#13 0x00007f6cebcef764 in _PyObject_Call (kwargs=0x0, args=0x7f6c5847c340, callable=0x7f6ccc546a60, tstate=0x7f6c9b455180) at Objects/call.c:343
#14 _PyObject_Call (tstate=0x7f6c9b455180, callable=0x7f6ccc546a60, args=0x7f6c5847c340, kwargs=0x0) at Objects/call.c:313
#15 0x00007f6cc65b2de2 in pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::objects::full_py_function_impl<pxrInternal_v0_25_5__pxrReserved__::Tf_ModuleProcessor::_InvokeWithErrorHandling, pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::type_list<_object*> >::operator()(_object*, _object*) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_tf.so
#16 0x00007f6cc7cf9e9d in pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::objects::function::call(_object*, _object*) const () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#17 0x00007f6cc7cfa078 in std::_Function_handler<void (), std::reference_wrapper<pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::objects::(anonymous namespace)::bind_return> >::_M_invoke(std::_Any_data const&) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#18 0x00007f6cc7cf2705 in pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::exception_handler::operator()(std::function<void ()> const&) const () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#19 0x00007f6cc725840b in std::_Function_handler<bool (pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::exception_handler const&, std::function<void ()> const&), std::_Bind_result<bool, pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::translate_exception<pxrInternal_v0_25_5__pxrReserved__::TfBaseException, void (*)(pxrInternal_v0_25_5__pxrReserved__::TfBaseException const&)> (std::_Placeholder<1>, std::_Placeholder<2>, void (*)(pxrInternal_v0_25_5__pxrReserved__::TfBaseException const&))> >::_M_invoke(std::_Any_data const&, pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::detail::exception_handler const&, std::function<void ()> const&) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/python/pxr/Tf/_tf.so
#20 0x00007f6cc7cf26a3 in pxrInternal_v0_25_5__pxrReserved__::pxr_boost::python::handle_exception_impl(std::function<void ()>) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
#21 0x00007f6cc7cf7dca in function_call () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_python.so
...
#59 0x00007f6cebcf0c6d in _PyObject_CallFunctionVa (is_size_t=0, va=0x7f6c9cfdd258, format=<optimized out>, callable=0x7f6c587c7400, tstate=0x7f6c9b455180) at Objects/call.c:508
#60 PyObject_CallFunction (callable=0x7f6c587c7400, format=<optimized out>) at Objects/call.c:558

Thread B

Thread 1 (Thread 0x7f6ceba7b780 (LWP 346265) "gaffer"):
#0  0x00007f6ceb88722a in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f6ceb88996f in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2  0x00007f6cebde397f in PyCOND_TIMEDWAIT (us=<optimized out>, mut=0x7f6cec0e6eb8 <_PyRuntime+440>, cond=0x7f6cec0e6e88 <_PyRuntime+392>) at Python/condvar.h:73
#3  take_gil (tstate=tstate@entry=0x7f6cec10f6b8 <_PyRuntime+166328>) at Python/ceval_gil.h:231
#4  0x00007f6cebde3e7e in PyEval_RestoreThread (tstate=tstate@entry=0x7f6cec10f6b8 <_PyRuntime+166328>) at Python/ceval.c:521
#5  0x00007f6cebe65682 in posix_do_stat (module=module@entry=0x7f6cdd9ad710, path=path@entry=0x7ffd89f03050, dir_fd=<optimized out>, follow_symlinks=<optimized out>, function_name=0x7f6cebf261bf "stat") at ./Modules/posixmodule.c:2591
#6  0x00007f6cebe65931 in os_stat_impl (follow_symlinks=<optimized out>, dir_fd=<optimized out>, path=0x7ffd89f03050, module=0x7f6cdd9ad710) at ./Modules/posixmodule.c:2933
#7  os_stat (module=0x7f6cdd9ad710, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at ./Modules/clinic/posixmodule.c.h:71
#8  0x00007f6cebc9629a in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at ./Include/cpython/methodobject.h:52
#9  0x00007f6cebde4abf in _PyEval_EvalFrame (throwflag=0, frame=0x7f6cebb8c000, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#10 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=0x7f6c587c9180, func=0x7f6c588711c0, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at Python/ceval.c:6434
#11 PyEval_EvalCode (co=co@entry=0x7f6c58a6e4c0, globals=globals@entry=0x7f6c587c9180, locals=locals@entry=0x7f6c587c9180) at Python/ceval.c:1148
#12 0x00007f6cebde0be0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7f6c587c9180, globals=0x7f6c587c9180, source=0x7f6c58a6e4c0) at Python/bltinmodule.c:1077
#13 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#14 0x00007f6cebd3eea2 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f6cdd988f90, args=0x7f6c587a6ed8, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:443
#15 0x00007f6cebc97e32 in do_call_core (use_tracing=<optimized out>, kwdict=0x7f6c587c9100, callargs=0x7f6c587a6ec0, func=0x7f6cdd988f90, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at Python/ceval.c:7349
#16 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#17 0x00007f6cebde4c21 in _PyEval_EvalFrame (throwflag=0, frame=0x7f6cebb8bd08, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#18 _PyEval_Vector (tstate=0x7f6cec10f6b8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#19 0x00007f6cebcf0274 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ffd89f03560, callable=0x7f6cdd94fce0, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#20 object_vacall (tstate=tstate@entry=0x7f6cec10f6b8 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7f6cdd94fce0, vargs=vargs@entry=0x7ffd89f035e8) at Objects/call.c:819
#21 0x00007f6cebcf0458 in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:879
#22 0x00007f6cebe12961 in import_find_and_load (abs_name=0x7f6bcf3aefb0, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at Python/import.c:1748
#23 PyImport_ImportModuleLevelObject (name=name@entry=0x7f6bcf3aefb0, globals=globals@entry=0x7f6c8921f800, locals=locals@entry=0x7f6c8921f800, fromlist=fromlist@entry=0x7f6c587a5580, level=0) at Python/import.c:1847
#24 0x00007f6cebdde874 in builtin___import___impl (level=<optimized out>, fromlist=0x7f6c587a5580, locals=0x7f6c8921f800, globals=0x7f6c8921f800, name=0x7f6bcf3aefb0, module=<optimized out>) at Python/bltinmodule.c:277
#25 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:78
#26 0x00007f6cebd3eea2 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f6cdd988b30, args=0x7f6bcf370198, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:443
#27 0x00007f6c9f084cee in ?? () from //packages/vendor/industrial/kiln.platform/2024.0.gaffer/AlmaLinux_9.5/x86_64/pyside/6.5.6/lib/libshiboken6.abi3.so.6.5
#28 0x00007f6cebd3e873 in cfunction_call (func=0x7f6cca005580, args=<optimized out>, kwargs=<optimized out>) at Objects/methodobject.c:542
#29 0x00007f6cebcefa13 in _PyObject_MakeTpCall (tstate=0x7f6cec10f6b8 <_PyRuntime+166328>, callable=0x7f6cca005580, args=<optimized out>, nargs=5, keywords=0x0) at Objects/call.c:214
#30 0x00007f6cebcf0849 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffd89f038c0, callable=0x7f6cca005580, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#31 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffd89f038c0, callable=0x7f6cca005580, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:77
#32 _PyObject_CallFunctionVa (tstate=tstate@entry=0x7f6cec10f6b8 <_PyRuntime+166328>, callable=callable@entry=0x7f6cca005580, format=format@entry=0x7f6cebf1d4cd "OOOOi", va=va@entry=0x7ffd89f03938, is_size_t=is_size_t@entry=0) at Objects/call.c:536
#33 0x00007f6cebcf0c6d in _PyObject_CallFunctionVa (is_size_t=0, va=0x7ffd89f03938, format=0x7f6cebf1d4cd "OOOOi", callable=0x7f6cca005580, tstate=0x7f6cec10f6b8 <_PyRuntime+166328>) at Objects/call.c:508
#34 PyObject_CallFunction (callable=callable@entry=0x7f6cca005580, format=format@entry=0x7f6cebf1d4cd "OOOOi") at Objects/call.c:558
#35 0x00007f6cebe13077 in PyImport_Import (module_name=module_name@entry=0x7f6bcf3aefb0) at Python/import.c:2036
#36 0x00007f6cebe1325a in PyImport_ImportModule (name=<optimized out>) at Python/import.c:1465
#37 0x00007f6cc65be50e in pxrInternal_v0_25_5__pxrReserved__::TfScriptModuleLoader::_LoadLibModules(std::vector<std::pair<pxrInternal_v0_25_5__pxrReserved__::TfToken const, pxrInternal_v0_25_5__pxrReserved__::TfScriptModuleLoader::_LibInfo> const*, std::allocator<std::pair<pxrInternal_v0_25_5__pxrReserved__::TfToken const, pxrInternal_v0_25_5__pxrReserved__::TfScriptModuleLoader::_LibInfo> const*> > const&) const () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_tf.so
#38 0x00007f6cc65bf139 in pxrInternal_v0_25_5__pxrReserved__::TfScriptModuleLoader::LoadModules() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_tf.so
#39 0x00007f6cc6540c0b in pxrInternal_v0_25_5__pxrReserved__::TfDlopen(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_tf.so
#40 0x00007f6cc7cb2f8b in pxrInternal_v0_25_5__pxrReserved__::PlugPlugin::_Load() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_plug.so
#41 0x00007f6cc7cb69d6 in pxrInternal_v0_25_5__pxrReserved__::PlugPlugin::_LoadWithDependents(pxrInternal_v0_25_5__pxrReserved__::PlugPlugin::_SeenPlugins*) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_plug.so
#42 0x00007f6cc7cb6ee6 in pxrInternal_v0_25_5__pxrReserved__::PlugPlugin::Load() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_plug.so
#43 0x00007f6cc4033697 in pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_PluginResolver<pxrInternal_v0_25_5__pxrReserved__::ArResolver, pxrInternal_v0_25_5__pxrReserved__::Ar_ResolverFactoryBase>::Create() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#44 0x00007f6cc4036d73 in pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_CreateResolver(pxrInternal_v0_25_5__pxrReserved__::TfType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#45 0x00007f6cc40370e7 in pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_DispatchingResolver::_InitializePrimaryResolver(std::vector<pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_ResolverInfo, std::allocator<pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_ResolverInfo> > const&)::{lambda(pxrInternal_v0_25_5__pxrReserved__::TfType const&)#1}::operator()(pxrInternal_v0_25_5__pxrReserved__::TfType const&) const () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#46 0x00007f6cc40374d3 in pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_DispatchingResolver::_InitializePrimaryResolver(std::vector<pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_ResolverInfo, std::allocator<pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_ResolverInfo> > const&) [clone .constprop.0] () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#47 0x00007f6cc4037b7e in pxrInternal_v0_25_5__pxrReserved__::(anonymous namespace)::_GetResolver() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#48 0x00007f6cc4038329 in pxrInternal_v0_25_5__pxrReserved__::ArGetResolver() () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_ar.so
#49 0x00007f6cc44456cf in pxrInternal_v0_25_5__pxrReserved__::SdfLayer::_ComputeInfoToFindOrOpenLayer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, pxrInternal_v0_25_5__pxrReserved__::SdfLayer::_FindOrOpenLayerInfo*, bool) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_sdf.so
#50 0x00007f6cc4445ae8 in pxrInternal_v0_25_5__pxrReserved__::SdfLayer::FindOrOpen(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_sdf.so
#51 0x00007f6cc34c6550 in pxrInternal_v0_25_5__pxrReserved__::_OpenLayer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pxrInternal_v0_25_5__pxrReserved__::ArResolverContext const&) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_usd.so
#52 0x00007f6cc34fda38 in pxrInternal_v0_25_5__pxrReserved__::UsdStage::Open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pxrInternal_v0_25_5__pxrReserved__::UsdStage::InitialLoadSet) () from /packages/vendor/ImageEngine/gaffer/linux_64/python3/1.6.6.1/lib/libusd_usd.so

[jesschimein]

Filed as internal issue #USD-11896

(This is an automated message. See here for more information.)

[spiffmon]

Exactly so, @johnhaddon . This is an expected and common pattern in OpenUSD's python integration, and we provide a mechanism for it (TF_PY_ALLOW_THREADS_IN_SCOPE()) that you will see sprinkled throughout our code in places where we are launching parallel work, where we expect plugins may be loaded, or when python interaction may subsequently be instigated (as with our TfRefPtr 's interaction with python.

Feel free to close this if TF_PY_ALLOW_THREADS_IN_SCOPE takes care of it for you, which I expect it should - thanks!

[johnhaddon]

Sorry, I'm not sure I've understood completely. Are you saying that it is our responsibility to put TF_PY_ALLOW_THREADS_IN_SCOPE somewhere in our code? I was assuming the bug was that _PyResolverScopedCache::Enter() didn't have TF_PY_ALLOW_THREADS_IN_SCOPE, and the fix would be to add it there. I'm happy to make a PR for the latter.

For a bit more context, this deadlock is happening in Gaffer, where Thread A is running third-party Python extension code we don't control, and thread B is running a native C++ node which has no knowledge of or dependency on Python.

[spiffmon]

Ah, so sorry, @johnhaddon - I was too hasty. Yep, that's our code and yep, we will fix it!

[johnhaddon]

No worries - I've been responsible for enough GIL bugs that it might well have been one of mine! Thanks for fixing. In the meantime we seem to be having some luck with calling GetResolver() during startup, so everything is initialised before we hit the threaded code.