[Benchmark] Add Video-MME for Qwen3-Omni

[Jayon02]

Motivation

Modifications

Related Issues

Fixes #253

Accuracy Test

Benchmark & Profiling

Checklist

• Format your code according with pre-commit.
• Add unit tests.
• Update documentation / docstrings / example tutorials as needed.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.

[zhaochenyang20]

Hey @Jayon02, thanks for pushing this through. A few thoughts:

On thinker_max_seq_len: You diagnosed it right — hardcoded 8192 with no assert means video inputs silently crash the CUDA kernel instead of failing at the request boundary. But please don't unify the parameter-passing story here. #318 is actively reworking the override layer (apply_server_args_overrides primitive, CLI precedence, schema layering); landing a second parameter through that mechanism while it's in flight will conflict. Keep the temporary path for now, add an explicit TODO comment noting it should migrate onto the #318 primitive afterward (same for video_fps and similar runtime params).

Please split this into three PRs:

1. Bug fix — the prompt-length assert that turns the CUDA crash into a clean request-level error. Ship this fast, it's the user-visible fix.
2. Benchmark — benchmark_omni_videomme on the full set (~2h), following the MMMU / MMSU pattern with accuracy + speed numbers in the PR description.
3. CI — the ~10min subset imported from the benchmark script, same pattern as MMSU / MMMU.

Decouples bug fix from full-set runs, and CI from benchmark review.

On the dataset: YouTube link-rot is real. "First N reachable" is fine for getting it running, but CI needs a frozen snapshot — please push what's currently reachable to an HF dataset. Already-delisted videos are unrecoverable, don't worry about those.

On code quality: The existing video code paths were written early and it shows. Don't rescue all of it here — stand up the benchmark + CI, mark rough edges with TODOs, refactor incrementally. Priority: bug fix → benchmark → CI → cleanup.

Concrete next step: split out the prompt-length assert as PR 1 and get it merged fast.

[Jayon02]

Discussion

The full benchmark is complete, but there are some regressions. After splitting the PR into three phases, many CI data points that previously passed are now failing. Interestingly, these failures seem to be state-related: a failed data point runs successfully on a fresh server, but in the pipeline, everything starts failing after a certain number of processed items. I suspect there’s a state-leak or an issue in the preprocessing stage that's causing this cumulative failure.

Previous Results

====================================================
  Video-MME Accuracy — qwen3-omni
====================================================
  Total samples:               50
  Correct:                     29
  Accuracy:                    0.5800 (58.0%)
  Failed requests:             0
  MC parse fallback:           1
====================================================

============================================================
                      Video-MME Speed                       
============================================================
  Model:                         qwen3-omni
  Concurrency:                   1
  Completed requests:            50
  Failed requests:               0
------------------------------------------------------------
  Latency mean (s):              19.719
  Latency median (s):            18.762
  Latency p95 (s):               28.401
  Latency p99 (s):               31.016
  Tok/s (per-req mean):          6.0
  Tok/s (per-req median):        5.7
  Tok/s (aggregate):             6.0
  Gen tokens (mean):             119
  Gen tokens (total):            5964
  Prompt tokens (mean):          10724
  Prompt tokens (total):         536220
  Throughput (req/s):            0.051
============================================================

Current Results

====================================================
  Video-MME Accuracy — qwen3-omni
====================================================
  Total samples:               50
  Correct:                     14
  Accuracy:                    0.2800 (28.0%)
  Failed requests:             26
  MC parse fallback:           0
====================================================

============================================================
                      Video-MME Speed                       
============================================================
  Model:                         qwen3-omni
  Concurrency:                   1
  Completed requests:            24
  Failed requests:               26
------------------------------------------------------------
  Latency mean (s):              24.702
  Latency median (s):            19.479
  Latency p95 (s):               54.725
  Latency p99 (s):               67.266
  Tok/s (per-req mean):          4.9
  Tok/s (per-req median):        4.3
  Tok/s (aggregate):             4.3
  Gen tokens (mean):             106
  Gen tokens (total):            2555
  Prompt tokens (mean):          10442
  Prompt tokens (total):         250612
  Throughput (req/s):            0.032
============================================================

How to run

python -m sglang_omni.cli.cli serve \
  --model-path Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --text-only \
  --thinker-max-seq-len 18000 \
  --port 8000

python -m benchmarks.eval.benchmark_omni_videomme \
  --model qwen3-omni \
  --port 8000 \
  --repo-id zhaochenyang20/Video_MME_ci

[Ratish1]

Discussion

The full benchmark is complete, but there are some regressions. After splitting the PR into three phases, many CI data points that previously passed are now failing. Interestingly, these failures seem to be state-related: a failed data point runs successfully on a fresh server, but in the pipeline, everything starts failing after a certain number of processed items. I suspect there’s a state-leak or an issue in the preprocessing stage that's causing this cumulative failure.

Previous Results

====================================================
  Video-MME Accuracy — qwen3-omni
====================================================
  Total samples:               50
  Correct:                     29
  Accuracy:                    0.5800 (58.0%)
  Failed requests:             0
  MC parse fallback:           1
====================================================

============================================================
                      Video-MME Speed                       
============================================================
  Model:                         qwen3-omni
  Concurrency:                   1
  Completed requests:            50
  Failed requests:               0
------------------------------------------------------------
  Latency mean (s):              19.719
  Latency median (s):            18.762
  Latency p95 (s):               28.401
  Latency p99 (s):               31.016
  Tok/s (per-req mean):          6.0
  Tok/s (per-req median):        5.7
  Tok/s (aggregate):             6.0
  Gen tokens (mean):             119
  Gen tokens (total):            5964
  Prompt tokens (mean):          10724
  Prompt tokens (total):         536220
  Throughput (req/s):            0.051
============================================================

Current Results

====================================================
  Video-MME Accuracy — qwen3-omni
====================================================
  Total samples:               50
  Correct:                     14
  Accuracy:                    0.2800 (28.0%)
  Failed requests:             26
  MC parse fallback:           0
====================================================

============================================================
                      Video-MME Speed                       
============================================================
  Model:                         qwen3-omni
  Concurrency:                   1
  Completed requests:            24
  Failed requests:               26
------------------------------------------------------------
  Latency mean (s):              24.702
  Latency median (s):            19.479
  Latency p95 (s):               54.725
  Latency p99 (s):               67.266
  Tok/s (per-req mean):          4.9
  Tok/s (per-req median):        4.3
  Tok/s (aggregate):             4.3
  Gen tokens (mean):             106
  Gen tokens (total):            2555
  Prompt tokens (mean):          10442
  Prompt tokens (total):         250612
  Throughput (req/s):            0.032
============================================================

How to run

python -m sglang_omni.cli.cli serve \
  --model-path Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --text-only \
  --thinker-max-seq-len 18000 \
  --port 8000

python -m benchmarks.eval.benchmark_omni_videomme \
  --model qwen3-omni \
  --port 8000 \
  --repo-id zhaochenyang20/Video_MME_ci

Hey @Jayon02 , did you find any reason that this is happening?. Could you look into this in detail if you are able to find out why this is happening. Thanks

[zhaochenyang20]

This is my only concern. If CI pass, we can let this be and keep this for Video use case.