Do you plan to consider the "--enforce-eager" and "--no-enforce-eager" options in vllm?

[giocafe]

I'm currently comparing aggregation vs. disaggregation performance in vllm.

When configuring disaggregation, the "--enfonce-eager" option makes a significant difference in performance.
In my testing environment, the result of aiconfigurator claims that disaggregation performance improves by up to 1.09x compared to aggregation.
When I ran the aiperf benchmark, the performance comparison was as
--enforce-eager: 1.07x improvement.
--no-enforce-eager: No improvement.

I'd like to know if the aiconfigurator currently considers these options.

[tianhaox]

Hi, thanks for providing this info. I'm afraid we didn't consider this option. We're glad to investigate into this. Could you provide more details about your case if possible,

1. which gpu, how many gpus
2. which model, what quantization, what parallel mapping (both agg and disagg)
3. isl osl
4. SLO: TTFT and TPOT
5. what's the absolute perf you get with this option and without, how about aiconfigurator's result
6. vllm version you're using

We can try to triage your case. Note that vllm support is just at the beginning, we're just getting it funcationally working but with minimum alignment. If the info is not public, we can also discuss over email, tianhaox@nvidia.com
Thank you

[giocafe]

OK, I'll share my configuration.

1. which gpu, how many gpus H200 x 8
2. which model, what quantization, what parallel mapping (both agg and disagg) Qwen3-30B-A3B
3. isl osl ISL 3000, OSL 150
4. SLO: TTFT and TPOT Not setup
5. what's the absolute perf you get with this option and without, how about aiconfigurator's result as below
6. vllm version you're using 0.10.2

1. AI Configurator

$ aiconfigurator cli default --model QWEN3_32B --total_gpus 8 --system h200_sxm --isl 3000 --osl 150

********************************************************************************
*                     Dynamo aiconfigurator Final Results                      *
********************************************************************************
  ----------------------------------------------------------------------------
  Input Configuration & SLA Target:
    Model: QWEN3_32B (is_moe: False)
    Total GPUs: 8
    Best Experiment Chosen: disagg at 454.21 tokens/s/gpu (disagg 1.09x better)
  ----------------------------------------------------------------------------
  Overall Best Configuration:
    - Best Throughput: 454.21 tokens/s/gpu
    - User Throughput: 56.48 tokens/s/user
    - TTFT: 401.25ms
    - TPOT: 17.71ms
  ----------------------------------------------------------------------------
  Pareto Frontier:
              QWEN3_32B Pareto Frontier: tokens/s/gpu vs tokens/s/user
     ┌─────────────────────────────────────────────────────────────────────────┐
800.0┤ •• disagg                                                               │
     │ ff agg                                                                  │
     │ xx disagg best                                                          │
     │                                                                         │
666.7┤ ff                                                                      │
     │  ff                                                                     │
     │    ff                                                                   │
     │      f                                                                  │
533.3┤      fff   •                                                            │
     │         ffff•••••••                                                     │
     │             f   x  ••••••••                                             │
     │              ff            •••                                          │
400.0┤               fffff           •••                                       │
     │                    ff            ••••                                   │
     │                      ffffff          •••                                │
     │                            fffff        ••••                            │
266.7┤                                 ff          •                           │
     │                                   fffff      •                          │
     │                                        fffffff•                         │
     │                                              ffff                       │
133.3┤                                                 •ffffffff               │
     │                                                         ff              │
     │                                                           fff           │
     │                                                             f           │
  0.0┤                                                                         │
     └┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬┘
      0                60                120               180              240
tokens/s/gpu                        tokens/s/user

  ----------------------------------------------------------------------------
  Deployment Details:
    (p) stands for prefill, (d) stands for decode, bs stands for batch size, a replica stands for the smallest scalable unit xPyD of the disagg system
    Some math: total gpus used = replicas * gpus/replica
               gpus/replica = (p)gpus/worker * (p)workers + (d)gpus/worker * (d)workers; for Agg, gpus/replica = gpus/worker
               gpus/worker = tp * pp * dp = etp * ep * pp for MoE models; tp * pp for dense models (underlined numbers are the actual values in math)

disagg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+------------+----------------+-------------+-------+------------+----------------+-------------+
| Rank | tokens/s/gpu | tokens/s/user |  TTFT  | concurrency | total_gpus(used) | replicas | gpus/replica | (p)workers | (p)gpus/worker | (p)parallel | (p)bs | (d)workers | (d)gpus/worker | (d)paral)bs |
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+------------+----------------+-------------+-------+------------+----------------+-------------+
|  1   |    454.21    |     56.48     | 401.25 |  112(=56x2) |    8 (8=2x4)     |    2     | 4 (=3x1+1x1) |     3      |    1 (=1x1)    |    tp1pp1   |   1   |     1      |    1 (=1x1)    |    tp1pp56  |
|  2   |    454.21    |     91.32     | 401.25 |  48(=48x1)  |    8 (8=1x8)     |    1     | 8 (=6x1+1x2) |     6      |    1 (=1x1)    |    tp1pp1   |   1   |     1      |    2 (=2x1)    |    tp2pp48  |
|  3   |    302.81    |     146.21    | 401.25 |  20(=20x1)  |    8 (8=1x8)     |    1     | 8 (=4x1+1x4) |     4      |    1 (=1x1)    |    tp1pp1   |   1   |     1      |    4 (=4x1)    |    tp4pp20  |
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+------------+----------------+-------------+-------+------------+----------------+-------------+

agg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+-------------+----------+----+
| Rank | tokens/s/gpu | tokens/s/user |  TTFT  | concurrency | total_gpus(used) | replicas | gpus/replica | gpus/worker | parallel | bs |
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+-------------+----------+----+
|  1   |    416.19    |     51.04     | 548.36 |  80(=20x4)  |    8 (8=4x2)     |    4     |      2       |   2 (=2x1)  |  tp2pp1  | 20 |
|  2   |    401.89    |     57.95     | 819.62 |  80(=10x8)  |    8 (8=8x1)     |    8     |      1       |   1 (=1x1)  |  tp1pp1  | 10 |
|  3   |    352.46    |     50.13     | 397.08 |  64(=32x2)  |    8 (8=2x4)     |    2     |      4       |   4 (=4x1)  |  tp4pp1  | 32 |
|  4   |    251.55    |     55.60     | 301.69 |  40(=40x1)  |    8 (8=1x8)     |    1     |      8       |   8 (=8x1)  |  tp8pp1  | 40 |
+------+--------------+---------------+--------+-------------+------------------+----------+--------------+-------------+----------+----+
********************************************************************************
INFO 2025-11-13 15:47:34,342 main.py:314] All experiments completed in 19.10 seconds

2. The Aggregated Configuration of Dynamo.VLLM
Replica 4 x TP2 = 8

#!/bin/bash

# Frontend
python -m dynamo.frontend \
    --http-port=8000 &

# Replica 1
CUDA_VISIBLE_DEVICES=0,1 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 2 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --connector none &

# Replica 2
CUDA_VISIBLE_DEVICES=2,3 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 2 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --connector none &

# Replica 3
CUDA_VISIBLE_DEVICES=4,5 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 2 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --connector none &

# Replica 4
CUDA_VISIBLE_DEVICES=6,7 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 2 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --connector none

3. The Disaggregated Configuration of Dynamo.VLLM
Replica 2 x (DP3+TP1) = 8

#!/bin/bash

# Frontend
python -m dynamo.frontend \
    --http-port=8000 &

# Replica 1
# Decode 1
CUDA_VISIBLE_DEVICES=0 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 1 \
    --enforce-eager \ (← or omitted)
    --block-size 64 &
# Prefill 3
CUDA_VISIBLE_DEVICES=1 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 1 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --is-prefill-worker &
CUDA_VISIBLE_DEVICES=2 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 1 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --is-prefill-worker &
CUDA_VISIBLE_DEVICES=3 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 1 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --is-prefill-worker &

# Replica 2
# Decode 1
CUDA_VISIBLE_DEVICES=4 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 1 \
    --enforce-eager \ (← or omitted)
    --block-size 64 &
# Prefill 3
CUDA_VISIBLE_DEVICES=5 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 1 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --is-prefill-worker &
CUDA_VISIBLE_DEVICES=6 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 1 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --is-prefill-worker &
CUDA_VISIBLE_DEVICES=7 \
    python3 -m dynamo.vllm \
    --model "/msr/Qwen3-30B-A3B" \
    --served-model-name qwen3-30b-a3b \
    --tensor-parallel-size 1 \
    --block-size 64 \
    --enforce-eager \ (← or omitted)
    --is-prefill-worker

4. AIPerf Benchmark

#!/bin/bash
aiperf profile \
    --model "qwen3-30b-a3b" \
    --tokenizer "/msr/Qwen3-30B-A3B" \
    --endpoint-type chat \
    --endpoint v1/chat/completions \
    --url 127.0.0.1:8000 \
    --streaming --concurrency 16 \ (← 32, 64, 128, 256)
    --num-dataset-entries 1024 \
    --warmup-request-count 16 \ (← 32, 64, 128, 256)
    --request-count 160 \
    --synthetic-input-tokens-mean 3000 \
    --synthetic-input-tokens-stddev 0 \
    --output-tokens-mean 150 \
    --output-tokens-stddev 0 \
    --extra-inputs min_new_tokens:150 \
    --extra-inputs max_new_tokens:150 \
    --extra-inputs ignore_eos:true \
    --random-seed 0 \
    --artifact-dir batch_16 (← 32, 64, 128, 256)

Evaluation - CUDA Graph OFF
Add '--enforce-eager' Option

Evaluation - CUDA Graph ON
Omit '--enforce-eager' Option

[jasonqinzhou]

From your result, it shows that --enforce-eager=False provides more throughput and less latency. In most cases, --enforce-eager=False has better performance, thus it is safe to use this as default.

[giocafe]

Yes, it's obvious that enabling the CUDA Graph option (--enforce-eager=False) will improve performance.
However, my question isn't about absolute performance improvements.

I'm looking for a scenario where throughput performance is 'disagg > agg',
and I wanted to use the aiconfigurator results to support that.

Aiconfigurator claims that applying the 'disagg > agg' configuration results in a 1.09x performance improvement,
but when I actually configured it and tested the performance via aiperf benchmark, I got 'disagg < agg' results.

[tianhaox]

We will start looking into the perf alignment of vllm. I will try your test case and keep you posted