Server stops accepting /v1/audio/speech after unload, returns 503 indefinitely with no logged request, requires container restart to recover

[rubella]

Summary

On three separate occasions, after the model was unloaded via POST /api/unload
(either once, or several times in quick succession), a subsequent POST to
/v1/audio/speech (OpenAI-compatible endpoint) returns 503 Service Unavailable
on every attempt, indefinitely, with no corresponding log line on the server
side at all, no "OpenAI speech: processing...", no error, nothing. A working
request always logs "OpenAI speech: processing N chunk(s)..."; a wedged one
logs nothing, as if the request never reached the route handler. The only
recovery found is restarting the container (docker restart); neither a
further wait nor additional requests clear it on their own within the
windows observed (up to ~60s of continuous retries).

Environment

• Chatterbox-TTS-Server image/tag: chatterbox-tts-server:cu128
• Model: chatterbox-turbo (ChatterboxTurboTTS)
• GPU: NVIDIA RTX 5060 Ti 16GB, driver 580.142, CUDA 13.0
• Deployed via Docker Compose, fronted by a custom reverse proxy that
  transparently retries 503 responses on this endpoint (the proxy is not
  calling /api/unload itself in this incident; a separate consumer is)
• Python 3.10, diffusers with the LoRACompatibleLinear deprecation warning
  visible at startup (unrelated, just for version context)

Reproduction (partial — not yet 100% reliable)

Not consistently reproducible on demand yet. Observed across 3 incidents:

1. A model unload storm (~24 POST /api/unload calls over ~11 minutes, no
   reload attempts logged in between), followed by a synthesis attempt
   through the legacy /tts endpoint failing immediately and cleanly with
   [ERROR] server: TTS request failed: Model not loaded. Recovered only
   via the web UI's Save Settings, which triggers a hot-swap
   (/restart_server) reload.

2. The same unload-storm pattern, followed by 30 consecutive POST
   /v1/audio/speech attempts (3 client retries x 10 proxy retries each,
   2s apart) all returning 503 over roughly 60 seconds, with zero log
   lines on the server side for any of them. nvidia-smi during this window
   showed ~10.5GB of 16GB VRAM in use by an unrelated process, no VRAM
   pressure. Recovered via docker restart.

3. A SINGLE clean /api/unload call (logged normally, completed in <1s),
   followed ~3 minutes later by 20+ consecutive POST /v1/audio/speech
   attempts (2 client retries x 10 proxy retries, 2s apart) over the
   following ~40 seconds, all 503, again zero log lines on the server
   side. This is the case that concerns me most: it did not take a storm
   of unloads to reach this state, a single normal unload was enough.

Representative log excerpt (incident 2/3 shape)

[server log, last entry before the wedge:]
2026-08-27 09:12:43 [INFO] server: Request received for /api/unload (Model Unload).
2026-08-27 09:12:43 [INFO] engine: Initiating model unload sequence...
2026-08-27 09:12:44 [INFO] engine: Model unloaded and GPU memory released.
2026-08-27 09:12:44 [INFO] server: Model successfully unloaded and GPU memory released.
[... nothing else logged for the container, ever, until restart ...]

[reverse proxy log, ~3 minutes later:]
POST http://chatterbox-tts-server:8004/v1/audio/speech "HTTP/1.1 503 Service Unavailable"
[repeats 20+ times over ~40s, all 503, before the proxy gives up and
surfaces the failure to its own caller]

Additional finding

While the server was in this state, GET /api/model-info returned 200 with:

{"loaded":false,"type":null,"class_name":null,"device":null,"sample_rate":null,
"supports_paralinguistic_tags":false,"available_paralinguistic_tags":[],
"turbo_available_in_package":true,"multilingual_available_in_package":true,
"supports_multilingual":false,"supported_languages":{"en":"English"}}

This rules out a global deadlock, the process is responsive and correctly
reports no model loaded, not stuck mid-load. This means the bug is isolated
to whatever is supposed to trigger an on-demand reload when an inference
request arrives while unloaded: it's not firing, or it's failing before
logging anything or updating any observable state.

(Note: GET /health returned 404, that path doesn't appear to exist on this
server at all, unrelated to the issue above.)

Questions

• What triggers the documented on-demand reload for /v1/audio/speech when
  the model is unloaded? Given /api/model-info responds normally and
  reports loaded:false accurately, is there a scenario where that trigger
  could silently fail to fire, or fail before any logging occurs?
• Is there a way to force/inspect this from outside besides a full
  container restart, e.g. does /restart_server (used by the web UI's
  hot-swap) work reliably as a lighter-weight recovery than a full
  container restart?