fix: use create_api_app in OrchestrationClient to prevent read-only container failures

[Br1an67]

Closes #19317

Background services (e.g. actions service) use OrchestrationClient, which previously called create_app(). This triggers UI static directory creation via create_ui_static_subpath(), which fails with PermissionError in read-only containers (common in rootless/secure deployments). The exception is swallowed, causing automation actions to silently fail and messages to go to DLQ with no error logs.

Changes

1. Use create_api_app for OrchestrationClient (src/prefect/server/api/clients.py)

OrchestrationClient only needs API routes for in-process ASGI calls. Using create_api_app() instead of create_app() avoids UI creation and background services startup entirely.

2. Add error logging for unexpected action failures (src/prefect/server/events/actions.py)

Catch and log unexpected exceptions in the action consumer before re-raising. This provides visibility when actions fail for reasons other than ActionFailed, instead of messages silently going to DLQ.

3. Regression test (tests/server/api/test_clients.py)

Verifies that OrchestrationClient uses create_api_app.

Checklist

• This pull request references any related issue by including "closes Automations 'run-deployment' action silently fails in HA when UI static dir is read-only #19317"
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

Merging this PR will not alter performance

✅ 2 untouched benchmarks

---

_{Comparing Br1an67:fix/19317-ha-readonly-automation (79ce72e) with main (62b415b)}

[desertaxle]

I think a better solution to this issue would be to use create_api_app instead of create_app. That will fix the UI issue and also avoid adding a lifecycle that starts background services when we don't want them.

[Br1an67]

Thanks @desertaxle, that's a much cleaner approach — create_api_app avoids both the UI issue and the unnecessary lifespan/background services. I've updated the PR to use it instead.

[Br1an67]

Reverted back to create_app(ephemeral=True) — the switch to create_api_app() caused 394 test failures for two reasons:

1. 404 errors: create_api_app() returns the API sub-app with routes at the root (/work_pools/filter), but BaseClient uses base_url='/api', so requests go to /api/work_pools/filter which doesn't match any routes on the bare sub-app.

2. AttributeError on PrefectRouter.routes: create_api_app() with different kwargs (no exception handlers) creates a separate cached instance. When create_app(final=True) has already run, del router.routes makes the global routers unusable for any new include_router() calls.

create_app(ephemeral=True) is the correct fix because:

• It returns the full app with /api mount prefix, so URL routing works correctly
• It shares the same cached API app instance (APP_CACHE) as the running server
• ephemeral=True already skips UI static file creation (line 529 of server.py: and not ephemeral), which was the original issue Automations 'run-deployment' action silently fails in HA when UI static dir is read-only #19317

[Br1an67]

Updated to use create_api_app() directly as suggested. It provides exactly what OrchestrationClient needs — just the API routes for in-process ASGI calls — without UI creation or background services.