Voice Generation server eventually starts producing "1005" error and requires reboot #159
Description
Backend impacted
The Rust implementation
Operating system
Linux
Hardware
GPU with CUDA
Description
Hardware: 1xA100
We have been successfully using the rust server to stream voice generation for a while now (excellent job on your end!) which will eventually be used in a production environment. We haven't thrown anything at it like 30+ generations at the same time (per the documentation referencing the number of concurrent streams) but we do sometimes hit this error. Once we do it seems to be unrecoverable.
For now, we are just rebooting the server every day at 5am but this is obviously not a solution. Any help would be greatly appreciated!
Extra information
^[[2m2025-11-10T13:00:11.045421Z^[[0m ^[[31mERROR^[[0m ^[[2mmoshi_server::py_module^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/py_module.rs^[[0m^[[2m:^[[0m^[[2m545:^[[0m no free channels
^[[2m2025-11-10T13:00:11.045490Z^[[0m ^[[31mERROR^[[0m ^[[2mmoshi_server^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/main.rs^[[0m^[[2m:^[[0m^[[2m873:^[[0m py ^[[3merr^[[0m^[[2m=^[[0mno free channels
^[[2m2025-11-10T13:03:44.554482Z^[[0m ^[[32m INFO^[[0m ^[[2mmoshi_server^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/main.rs^[[0m^[[2m:^[[0m^[[2m903:^[[0m handling py streaming query
^[[2m2025-11-10T13:03:44.554574Z^[[0m ^[[32m INFO^[[0m ^[[2mmoshi_server::py_module^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/py_module.rs^[[0m^[[2m:^[[0m^[[2m538:^[[0m py query ^[[3mquery^[[0m^[[2m=^[[0mPyStreamingQuery { auth_id: None, format: PcmMessagePack, voice: Some("expresso/ex03-ex01_happy_001_channel1_334s.wav") }
^[[2m2025-11-10T13:03:44.554586Z^[[0m ^[[31mERROR^[[0m ^[[2mmoshi_server::py_module^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/py_module.rs^[[0m^[[2m:^[[0m^[[2m545:^[[0m no free channels
^[[2m2025-11-10T13:03:44.554647Z^[[0m ^[[31mERROR^[[0m ^[[2mmoshi_server^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/main.rs^[[0m^[[2m:^[[0m^[[2m873:^[[0m py ^[[3merr^[[0m^[[2m=^[[0mno free channels
^[[2m2025-11-10T13:15:05.250423Z^[[0m ^[[31mERROR^[[0m ^[[2mmoshi_server::py_module^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/py_module.rs^[[0m^[[2m:^[[0m^[[2m545:^[[0m no free channels
^[[2m2025-11-10T13:15:05.250474Z^[[0m ^[[31mERROR^[[0m ^[[2mmoshi_server^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/main.rs^[[0m^[[2m:^[[0m^[[2m873:^[[0m py ^[[3merr^[[0m^[[2m=^[[0mno free channels
^[[2m2025-11-10T13:15:33.449117Z^[[0m ^[[31mERROR^[[0m ^[[2mmoshi_server::py_module^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/py_module.rs^[[0m^[[2m:^[[0m^[[2m651:^[[0m recv loop err ^[[3merr^[[0m^[[2m=^[[0mWebSocket protocol error: Connection reset without closing handshake
Caused by:
0: WebSocket protocol error: Connection reset without closing handshake
1: Connection reset without closing handshake
^[[2m2025-11-10T13:15:33.449264Z^[[0m ^[[32m INFO^[[0m ^[[2mmoshi_server::py_module^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/py_module.rs^[[0m^[[2m:^[[0m^[[2m645:^[[0m send loop exited
^[[2m2025-11-10T13:15:33.449297Z^[[0m ^[[31mERROR^[[0m ^[[2mmoshi_server::py_module^[[0m^[[2m:^[[0m ^[[2m/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/moshi-server-0.6.3/src/py_module.rs^[[0m^[[2m:^[[0m^[[2m651:^[[0m recv loop err ^[[3merr^[[0m^[[2m=^[[0mWebSocket protocol error: Connection reset without closing handshake
Caused by:
0: WebSocket protocol error: Connection reset without closing handshake
1: Connection reset without closing handshake
Environment
Fill in the following information on your system.
- Operating system version:
If the backend impacted is PyTorch:
- Python version:
- PyTorch version:
- CUDA version (run
python -c 'import torch; print(torch.version.cuda)'): - GPU model and memory:
If the backend is MLX:
- Mac model: