feat: surface real root-cause errors on model launch failure

[OliverBryant]

Problem

When a model launch fails, the Web UI rarely shows the real cause. Users see a short, unhelpful line — or a toast that vanishes before it can be read.

Notably, this is not a case of the backend replacing errors with a generic message. str(e) is forwarded faithfully at every layer, and xoscar preserves the exception type, message and traceback across actor boundaries. The information is lost at six specific points:

1. The launch dialog discarded the error object. handleLaunch's .catch() took no argument, so the component never saw detail and relied entirely on a global toast.
2. That toast could not hold a traceback. sonner's ~4s auto-dismiss at a 356px width makes a Python traceback unreadable and impossible to copy in time.
3. The traceback was dropped at the wire. launch_model sent only str(e). xoscar had already made the full remote traceback available on e.__traceback__, and str(e) is often just the outermost wrapper's short message — with the root cause buried in __cause__ behind an [address=..., pid=...] prefix.
4. The ERROR replica status was overwritten. The supervisor recorded {status: ERROR, error_message: ...}, then the cleanup that follows a failed launch (terminate_model → worker._update_model_state("stopping")) reset it to TERMINATING/TERMINATED, orphaning error_message on a row indistinguishable from a normal shutdown. The frontend never rendered that field at all.
5. Multi-replica launches reported only the first failure. The remaining N-1 were dropped, so a benign error (e.g. "already in the model list") could mask the real CUDA error.
6. With wait_ready=False the error vanished completely. callback_for_async_launch never called task.exception(), so the failure was garbage collected with an "exception was never retrieved" warning and reported nowhere.

Changes

Backend

• New xinference/core/error_utils.py — walks the __cause__/__context__ chain to the root cause, strips xoscar's actor prefix, and formats tracebacks. Opt out with XINFERENCE_DISABLE_ERROR_TRACEBACK for hardened deployments (enabled by default: the operator hitting the failure is exactly who needs it, and these paths are already exposed via the log center).
• DetailedHTTPException carries the traceback in a separate traceback field. detail stays a plain string, so the Python client, the Web UI interceptor, and every existing consumer are unaffected — a dict-valued detail would have broken them all. Scoped to launch_model and /progress; the ~90 other str(e) sites are untouched.
• ERROR is now a sticky replica status — only an explicit READY (a genuine relaunch) or remove_replica_status clears it. Non-status keys such as model_state still apply, so cleanup stays observable.
• All replica failures are aggregated via AggregatedLaunchError. When exactly one replica fails the original exception object is re-raised, preserving its type so the existing 400/503/499 status mapping does not degrade to 500.
• /progress now fails outright once a launch has errored, instead of reporting a dead launch as 0% — this is what lets a polling client tell "failed" from "not started".
• wait_for_load records the root cause too. This is the path where engines that initialise on a background thread (vLLM, SGLang) actually fail, and it previously set no error state at all.

Frontend

• The dialog stays open on failure and shows the cause inline: a destructive alert with the root-cause message, an expandable traceback, and a copy button.
• Each replica card renders its own error_message (the field existed in the types but had zero usages).
• Toasts carrying a traceback stay up until dismissed, with a copy action.
• FastAPI's array-shaped 422 detail is normalized, so it no longer renders as [object Object].
• New i18n keys added to all four locales (en/zh/ja/ko).

Verification

Tested end to end against a local server:

Scenario	Result
Bogus model_path	detail = ValueError: Invalid input. model_path: /nonexistent/... does not exist., plus a traceback whose deepest frame is in the model loader, not the supervisor
Replica status after failure	status: ERROR with error_message populated — TERMINATED before this change
/progress after failure	HTTP 500 with the real cause, no longer a silent 0%
3 replicas failing	All three listed in detail; all three cards show ERROR
wait_ready=false	Returns 200 immediately, then /replicas reports ERROR with the cause; no "exception was never retrieved" warning, and the traceback is in the log
Engine unavailable	Reports AssertionError: Torch not compiled with CUDA enabled — the true origin, from xoscar/virtualenv/platform.py
422 malformed payload	Renders readable text instead of [object Object]

Automated: 45 new tests across test_error_utils.py, test_launch_error_reporting.py, and test_launch_error_response.py; 163 tests pass in the affected suites. pre-commit clean (one pre-existing E231 in an unrelated docstring remains), tsc --noEmit clean, eslint at its existing 57-warning baseline with no new warnings.

Not verified here: the successful-launch regression path. This machine has no CUDA, so a real launch cannot complete — it fails inside the virtualenv/torch probe. The failure paths above are fully covered; a GPU environment should confirm that a successful launch still returns {"model_uid": ...} and redirects to /running-model.

Compatibility

All API changes are additive: traceback on error bodies, model_state on /replicas, and error_message on InstanceInfo. detail remains a string and no existing field changes type or disappears.

[gemini-code-assist]

Code Review

This pull request enhances error reporting and handling during model deployment in Xinference by capturing, formatting, and propagating detailed backend tracebacks to the frontend. On the backend, a new DetailedHTTPException and error-formatting utilities are introduced to extract root causes and preserve them across actor boundaries. On the frontend, the launch dialog and request contexts are updated to display these detailed errors and tracebacks with collapsible sections and copy-to-clipboard functionality. The review feedback highlights three areas for improvement: adding a try-catch block around JSON.stringify in the error detail extraction to prevent potential crashes from circular references or BigInts, and checking the availability of navigator.clipboard before attempting copy operations in both the launch dialog and request context to avoid showing false success toasts.

[qinxuye]

Two issues need to be addressed before approval.

[OliverBryant]

Pushed 25cc2e6 addressing all review feedback, and rebased onto latest main — the conflict is resolved and the PR is mergeable again.

@qinxuye — permission boundary. You were right, and my original reasoning was wrong: I justified the default-on traceback by saying these paths are already exposed through the log center, without checking that the log center requires a different permission. logs:list and models:write are independent scopes, so a model operator was receiving diagnostics they cannot otherwise reach.

The traceback field is now gated on logs:list (admin always qualifies; an unauthenticated cluster returns it to everyone, since there is no boundary to protect). The normalized root-cause detail is unchanged for models:write, so the UI stays useful. The check live-reads DB permissions to match the auth service's own policy, and fails closed. API keys are capped at query/inference scopes and never qualify.

Caller	detail	traceback
No auth configured	yes	yes
admin	yes	yes
models:write + logs:list	yes	yes
models:write only	yes	no
API key	yes	no

@qinxuye — rebase. Both done-callbacks are kept at both call sites, including _launch_builtin_sharded_model.

Gemini — all three comments addressed. The two clipboard ones now reuse the existing copyToClipboard helper, which already reports real success/failure and falls back to execCommand on non-HTTPS origins; that also caught a bad i18n key of mine (common.copy, which does not exist and would have rendered as a literal key path). JSON.stringify is guarded in both branches.

Validation: 129 tests pass (11 new for the permission gate, including the central case that models:write alone is denied); pre-commit clean; tsc --noEmit clean; eslint and prettier at baseline. The permission matrix above was verified against the real RESTfulAPI methods, and a bogus-model_path launch still returns the correct root cause end to end.

Still unverified on my side: the successful-launch path, since this machine has no CUDA and a real launch cannot complete.