find_possible_tp() may return invalid TP values for vLLM

[anfredette]

Summary

find_possible_tp() in src/planner/capacity_planner.py:795 returns all integer divisors of num_attention_heads as valid tensor parallelism values. For example, Qwen2.5-14B has 40 attention heads, so the function returns {1, 2, 4, 5, 8, 10, 20, 40}.

However, vLLM only supports TP values that are powers of 2 (1, 2, 4, 8). This means TP=5, TP=10, etc. are returned as valid but would fail at deployment time.

Observed behavior

A recommendation was generated with "5x H200" (TP=5) for Qwen2.5-14B-Instruct, which is not a valid vLLM configuration.

Questions

• Should find_possible_tp() filter to powers of 2 only?
• Are there other inference frameworks where non-power-of-2 TP is valid?
• If filtering to powers of 2, should this be done in find_possible_tp() itself or at the call sites?

[anfredette]

@jgchn @namasl what do you think about this?

[jgchn]

Hi @anfredette, I thought I looked at this a while back. The vLLM AI says that vLLM does not explicitly claim TP must be powers of 2. It also referred to these issues: vllm-project/vllm#13188 and vllm-project/vllm#4232. However, from your empirical experiments, I guess you have found that sizes like 5, 20, and 40 don't work, right?

There's also the source code: https://docs.vllm.ai/en/latest/api/vllm/config/model/#__codelineno-0-1116 which doesn't check for powers of 2. But I think this is just verifying the validity of the TP parameter. In practice when it comes to partitioning weights, we might see something different. Lmk if your experience tells you otherwise.

[anfredette]

I haven't actually tried TP with non-powers of 2 on real vLLM. It was my understanding (apparently incorrect) that TP needed to be a power of 2.

However, I've been thinking about how to handle TP variants, and this got me thinking more. All of the possible TP options probably don't make sense.

Here's a description of an idea: TP and Solution Search.