Remote registry load problem #1450
Description
Describe the bug
API servers crash with ModuleNotFoundError when trying to load UI defaults for custom registry actions that are not installed locally. This prevents the API from serving action schemas and default values in multi-replica Kubernetes deployments where custom packages are only available on executors.
Provide logs
ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/app/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 177, in __call__
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/app/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
| raise exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
| await self.app(scope, receive, _send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
| with recv_stream, send_stream, collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/app/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
| response = await self.dispatch_func(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/tracecat/middleware/security.py", line 9, in dispatch
| response = await call_next(request)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
| raise app_exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
| with recv_stream, send_stream, collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/app/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
| response = await self.dispatch_func(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/tracecat/middleware/request.py", line 26, in dispatch
| return await call_next(request)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
| raise app_exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
| with recv_stream, send_stream, collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/app/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
| response = await self.dispatch_func(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/tracecat/middleware/auth.py", line 53, in dispatch
| response = await call_next(request)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
| raise app_exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/app/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/app/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/app/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/app/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/app/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
| response = await f(request)
| ^^^^^^^^^^^^^^^^
| File "/app/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
| raw_response = await run_endpoint_function(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
| return await dependant.call(**values)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/tracecat/workflow/actions/router.py", line 131, in get_action
| reg_action = await ra_service.load_action_impl(action.type)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/tracecat/registry/actions/service.py", line 332, in load_action_impl
| bound_action = get_bound_action_impl(action, mode=mode)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/app/tracecat/registry/loaders.py", line 42, in get_bound_action_impl
| fn = load_udf_impl(impl)
| ^^^^^^^^^^^^^^^^^^^
| File "/app/tracecat/registry/loaders.py", line 117, in load_udf_impl
| mod = importlib.import_module(module_path)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
| return _bootstrap._gcd_import(name[level:], package, level)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
| File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
| File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
| File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
| File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
| File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
| File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
| ModuleNotFoundError: No module named 'custom_registry'
+------------------------------------
During handling of the above exception, another exception occurred:
To reproduce
- Deploy Tracecat in with Kubernetes
- Install a custom registry action via git repository
- Restart the API container
- Try to access the action in the UI or call the API endpoint
GET /actions/{action_id} - See ModuleNotFoundError in API server logs
Expected behavior
- API servers should be able to serve action schemas and UI defaults without requiring custom packages to be installed to local.
- Only executors should require the actual packages for execution
- UI should display proper field defaults and validation
Environment:
- Tracecat version: Latest main branch
- OS: Kubernetes (any)
- Deployment: Multi-replica API servers with custom registry actions
- Issue occurs in any containerized environment where packages aren't universally installed
Additional context
The root cause is that API servers attempt to import Python modules when loading action implementations for UI purposes, but these modules are only needed for actual execution. The database already contains the action interface schema that could be used for UI defaults instead.
This is an architectural issue where validation mode (for UI) and execution mode (for running actions) both try to import modules, when only execution mode should require the actual packages.