[BUG] Poor single-server concurrency behavior in Qwen 3.5 under Ollama despite strong token speed

[charlesdrakon-cmyk]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

Summary
We are seeing effectively serialized query handling with Qwen 3.5 models in Ollama, even on high-memory Apple Silicon systems where single-stream performance is very strong. The result is that Qwen delivers excellent per-request speed, but poor true multi-user concurrency in a single server process.

Environment

Model family: Qwen 3.5
Models tested: 35B and 122B variants
Runtime: Ollama
Backend: MLX / Apple Silicon
Deployment type: private multi-user serving environment
Front end: Open WebUI

Observed behavior

Individual query speed is excellent.
Under multiple simultaneous user requests, generation appears to serialize within a single Ollama server instance.
This creates a “pseudo-concurrency” effect where the system still feels responsive because Qwen is fast, but requests are not actually being served in parallel in the way needed for efficient multi-user production use.
Ollama developers have indicated that the affected models have architectures that prevent parallel queries within a single Ollama server process, and that the workaround is to run multiple Ollama servers behind a reverse proxy.
That workaround duplicates model weights in memory, reducing memory available for context and making it much less efficient operationally.

Expected behavior

Better support for true concurrent inference within a single server instance, or model architectural characteristics that allow runtime schedulers to serve multiple requests more efficiently without duplicating weights.

Why this matters
Qwen’s token speed and response quality are excellent, and that makes the model highly attractive for production use. However, in a shared multi-user environment, concurrency characteristics are just as important as raw quality and speed. Requiring multiple independent server instances to achieve concurrency is memory-expensive and reduces deployability.

Impact

Limits efficient multi-user serving
Forces a tradeoff between concurrency and context capacity
Reduces the operational advantage of high-memory systems
Makes production scaling less elegant than it could be

Request
Are there plans to improve concurrency characteristics in future Qwen architectures, especially for shared-weight, single-server, multi-request inference scenarios?

If there are recommended model settings, architectural notes, or future roadmap items related to concurrency-friendly serving, that information would be very helpful.

Additional note
This is not a complaint about output quality or speed. Qwen performs very well on both. This is specifically feedback about deployment behavior in real multi-user serving environments.

期望行为 | Expected Behavior

No response

复现方法 | Steps To Reproduce

No response

运行环境 | Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):

备注 | Anything else?

No response

[GuilhermeGors]

Hi @charlesdrakon-cmyk and team.

To help determine whether the reported serialization is strictly an Apple Silicon (MLX) backend limitation or a broader architectural constraint, I conducted a controlled investigation focusing on Ollama's parallelism scheduler behavior on CUDA hardware.

Since Qwen 3.5 (35B/122B) heavily exceeds my local VRAM, I established a control group using standard Transformer models (GQA/SWA) to audit how the llama.cpp/Ollama slot system manages concurrency under severe memory pressure. I utilized an asynchronous Python script firing 6 simultaneous requests against a server configured with OLLAMA_NUM_PARALLEL=4.

---

Control Group Results (CUDA / RTX 3050 4GB)

Model	Architecture	VRAM Utilized	Wall Clock	Temporal Overlap	Verdict
Qwen 2.5 3B	Transformer (GQA)	~2.7 GiB	14.5s	79.55%	PARALLEL
LLaMA 3.2 3B	Transformer (GQA)	~2.5 GiB	27.5s	78.90%	PARALLEL
Mistral 7B	Transformer (GQA+SWA)	~2.4 GiB (Heavy CPU Offload)	67.7s	85.64%	PARALLEL

---

Scheduler Model / Architecture Diagram

graph TD
    A[Incoming Concurrent Requests] --> B(Ollama / llama.cpp Scheduler)
    B --> C{Model Architecture?}

    C -- "Standard Transformer (GQA / SWA)" --> D[KV Cache Slots Partitioned]
    D --> E["Parallel Decoding (Overlap > 75%)"]
    E --> F((SUCCESS))

    C -- "Hybrid Architecture (Qwen 3.5 GDN)" --> G[Rolling Recurrent State]
    G --> H[State Cannot Be Partitioned]
    H --> I[Requests Serialized / Queued]
    I --> J((BOTTLENECK))

    style E fill:#d4edda,stroke:#28a745,color:#155724
    style I fill:#f8d7da,stroke:#dc3545,color:#721c24

Loading

---

Key Findings

1. Ollama's parallelism functions correctly: For standard Transformer architectures, the scheduler successfully partitions the KV Cache into independent slots. We observed up to ~7x speedups with true concurrency.

2. VRAM pressure does not induce serialization: Even when Mistral 7B forced severe offloading to the CPU (leaving primarily the KV Cache in VRAM), the scheduler maintained parallel execution.

3. The Qwen 3.5 architectural bottleneck: The data supports that the serialization reported in this issue is specific to hybrid models. Qwen 3.5's Gated Delta Networks (GDN) / Linear Attention rely on a continuous rolling recurrent state. This state is fundamentally incompatible with the static, partitioned KV Cache slot management currently implemented in the inference engine.

---

Conclusion

The root cause is not Ollama's network scheduling, but rather the underlying C++ engine's current inability to rapidly checkpoint and restore the GDN-specific recurrent state tensor for multiple requests sharing the same model weights in memory.

I have submitted a Pull Request (#2185) containing the asynchronous benchmark tool used for this analysis. (Note: The script has been iteratively refined through CI review to ensure strict token counting via eval_count and robust overlap math, making it a production-ready diagnostic tool.) This will allow the team and community to reproduce these exact parallelism metrics across larger instances and validate future backend fixes for the GDN architecture.