Bug summary

Setup: self-hosted, HA, multiple API instances, single “server services” instance, Redis + Postgres

Symptom: prefect.automation.triggered appears (UI/DB/Redis), entry lands in Redis stream actions and is consumed (pending=0), but:

• no prefect.automation.action.executed / ...failed
• deployment not started
• message ends up in dlq:*
• no error in pod logs

Root cause: actions service uses prefect.server.api.clients.OrchestrationClient, which calls create_app() → create_ui_static_subpath() and tries to create:
/usr/local/lib/python3.12/site-packages/prefect/server/ui_build
→ in read-only directory (rootless container) this raises PermissionError
→ exception is swallowed, message goes to DLQ.

Workaround: set on the services pod:

PREFECT_UI_STATIC_DIRECTORY=/writable/dir

Ask: log the actual exception before DLQ; skip/relax UI creation for background/actions services.

Version info

Version:              3.4.25
API version:          0.8.4
Python version:       3.12.9
Git commit:           8a37e7b1
Built:                Thu, Oct 23, 2025 07:58 PM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          server
Pydantic version:     2.12.3
Server:
  Database:           postgresql
  PostgreSQL version: 14.18
Integrations:
  prefect-docker:     0.6.6
  prefect-redis:      0.2.5

Additional context

No response