Skip to content

[Test] Add e2e tests for Qwen3-TTS speech endpoint#1206

Open
linyueqian wants to merge 9 commits intovllm-project:mainfrom
linyueqian:test/qwen3-tts-e2e
Open

[Test] Add e2e tests for Qwen3-TTS speech endpoint#1206
linyueqian wants to merge 9 commits intovllm-project:mainfrom
linyueqian:test/qwen3-tts-e2e

Conversation

@linyueqian
Copy link
Contributor

@linyueqian linyueqian commented Feb 4, 2026

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Add e2e tests for Qwen3-TTS /v1/audio/speech endpoint. The existing unit tests used mocks that didn't match real behavior, allowing bugs like #1159 to slip through undetected.

Also add doc for tts.

Test Plan

pytest tests/e2e/online_serving/test_qwen3_tts.py -v -s

Test Result

Verified the test correctly catches the multimodal_output bug:

  • Without fix (main branch):
    FAILED tests/e2e/online_serving/test_qwen3_tts.py::TestQwen3TTSCustomVoice::test_speech_english_basic
    Server returns {"error":{"message":"TTS model did not produce audio output."}} instead of WAV audio.

  • With fix:
    PASSED - Valid WAV audio returned (87KB, 24000Hz mono)


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: linyueqian <linyueqian@outlook.com>
@hsliuustc0106
Copy link
Collaborator

could you please also add the api server endpoint in doc?

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds comprehensive e2e tests for the Qwen3-TTS model's /v1/audio/speech endpoint. The tests were created in response to bug #1159, where unit tests with mocks didn't catch real behavior issues. The new tests verify actual model inference without mocks.

Changes:

  • Adds e2e tests for Qwen3-TTS CustomVoice and VoiceDesign models
  • Tests /v1/audio/speech, /v1/audio/voices, and /v1/models endpoints
  • Includes regression test for multimodal_output bug that caused "TTS model did not produce audio output" error

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: linyueqian <linyueqian@outlook.com>
@linyueqian
Copy link
Contributor Author

could you please also add the api server endpoint in doc?

OK. I have added that doc.

linyueqian and others added 3 commits February 5, 2026 08:14
Signed-off-by: linyueqian <linyueqian@outlook.com>
Signed-off-by: linyueqian <linyueqian@outlook.com>
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
@hsliuustc0106 hsliuustc0106 added the ready label to trigger buildkite CI label Feb 5, 2026
Signed-off-by: linyueqian <linyueqian@outlook.com>
Copy link
Contributor

@congw729 congw729 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add markers as suggested; more usage could be checked at #577

Signed-off-by: linyueqian <linyueqian@outlook.com>
@linyueqian
Copy link
Contributor Author

Please add markers as suggested; more usage could be checked at #577

Thank you! I have added them.

@hsliuustc0106
Copy link
Collaborator

fix ci please

@linyueqian
Copy link
Contributor Author

fix ci please

Need to merge #1203 first. The test catches that bug so it's expected to fail without it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants