[Qwen3-Omni] Fix concurrent request CUDA crash in speech pipeline

[ischencheng]

Motivation

Closes #229. The Qwen3-Omni speech pipeline crashes with CUDA illegal memory access when handling concurrent TTS requests. Even 2 simultaneous requests corrupt GPU state, causing all subsequent requests to fail with HTTP 500 until server restart.

Two independent bugs contribute to this:

1. GPU inference race in code_predictor / code2wav: Both executors call run_in_executor on shared model instances without synchronization, allowing concurrent CUDA operations within the same process.
2. Tree-cache prefix vs input_embeds mismatch in talker prefill: When consecutive requests share the same prompt tokens, sglang's radix tree cache matches a prefix and reduces extend_input_len (e.g. 23 → 16). However, input_embeds is not sliced accordingly, causing a shape mismatch in the TVM store_cache kernel ("expected 23 but got 16") that corrupts GPU state.

Modifications

• code_predictor_executor.py: Add asyncio.Lock to serialize GPU inference calls, preventing concurrent CUDA operations on the shared model.
• code2wav_executor.py: Same asyncio.Lock serialization for the codec decoder.
• sglang_ar.py: Slice input_embeds by prefix_indices in _rebuild_prefill_input_embeds so the embed count matches extend_input_len. Prefer the correctly-sliced version over sglang core's unsliced forward_batch.input_embeds for projected prefill.
• tests/test_concurrent_tts.py: Add regression tests covering both fixes (CPU-only mocks, no GPU required).

Related Issues

• Closes Qwen3 Omni speech pipeline crashes with concurrent requests (CUDA illegal memory access) #229

Testing Notes

• Unit tests: pytest tests/test_concurrent_tts.py — 5 tests pass
  • code_predictor GPU lock serialization
  • code2wav GPU lock serialization
  • prefill input_embeds prefix slicing (with/without cache hit, multi-request)
• Integration: 4 concurrent TTS requests on live server — all return 200 with valid audio
• Cache correctness: 3 sequential identical requests verified — cache-hit requests produce audio with same RMS quality as first (no-cache) request
• Accuracy: No model-side logic changes; projected embeddings are deterministic for identical prompts, so tree-cache prefix reuse is semantically correct