Skip to content

README_EN.md

Latest commit

 

History

History
174 lines (134 loc) · 14.4 KB

README_EN.md

File metadata and controls

174 lines (134 loc) · 14.4 KB
Raw

Alibaba Cloud Bailian - Tongyi Speech Large Models - Development Examples

English | 简体中文

This repository demonstrates how to call Speech Large Models (including Qwen-Audio-3.0-TTS, Qwen-Audio-3.0-ASR, CosyVoice, Fun-ASR, etc.) via Alibaba Cloud Bailian, enabling functions like speech recognition (speech-to-text), speech synthesis (text-to-speech), and advanced AI capabilities such as voice chat, audio analysis, and real-time translation when integrated with large language models (e.g., Qwen-OMNI, Qwen, Baichuan, Moonshot, 01.AI, MiniMax). Developers can test examples using Bailian's free quota and integrate these samples into their projects for further development. For technical support, join our DingTalk group.

⭐ Latest Updates ⭐

2026/08/14

  • Added Qwen-Audio-3.0-TTS speech synthesis and voice cloning examples, with voice style, dialect and emotion controllable by instruction.
  • Added the Qwen-Audio-3.0-ASR speech recognition examples, covering non-streaming recognition, streaming recognition (with hotwords and dialog context) and recorded-file transcription.

2026/07/14

  • Added Qwen-Audio-3.0-Realtime end-to-end real-time voice conversation example (companion chat, tool calling, and reasoning routing scenarios).

👉 Getting Started

  • Clone the repository via git clone or download the ZIP file.
  • Prerequisites: Create an Alibaba Cloud account, activate Bailian Model Service, generate an API_KEY, configure the environment, and install the DashScope SDK. Detailed steps are in PREREQUISITES.md. Check individual README.md files in example directories for specific dependencies.

👉 Use Cases & Examples

🚀 Recommended Scenarios

Use Case Description Example
Qwen-Audio-3.0-ASR Streaming Recognition Unlimited-duration real-time streaming recognition with hotwords and dialog context Streaming Recognition
Qwen-Audio-3.0-ASR Non-Streaming Recognition One-shot recognition of an audio URL, with dialog context, up to 5 min / 2 GB Non-Streaming Recognition
Qwen-Audio-3.0-ASR File Transcription Asynchronous transcription of recordings up to 12 hours, with speaker diarization File Transcription
Qwen-Audio-3.0-TTS Synthesis Control style, dialect and emotion by instruction, with real-time streaming playback Instruction-Controlled Synthesis
Qwen-Audio-3.0-TTS Voice Cloning Clone a voice from one audio sample and synthesize any text with it Voice Cloning Synthesis
Qwen-Audio-3.0-Realtime Voice Chat End-to-end real-time voice conversation over WebSocket (companion chat, tool calling, reasoning routing) qwen-audio-3.0-realtime

More Agent scenarios: qwen-audio-agent provides production-grade Agent examples built on Qwen-Audio-3.0-Realtime, including multi-turn memory, tool orchestration, and multimodal input.

✨ Advanced Usage

Use Case Description Example
Streaming + Dialog Context Pass dialog history or domain terms to improve accuracy, with limit-aware trimming Dialog Context
Streaming + Precompiled Vocabulary Create a vocabulary once and reuse by ID; covers the full create/query/recognize/delete lifecycle Precompiled Vocabulary
TTS Instruction Guide Style, dialect, emotion tags and combined usage with a full tag reference table Instruction Guide
TTS Text Normalization (TN) Synthesis of hard-to-read text: quantities, units, percent signs, ranges, hotline numbers and polyphones Text Normalization Showcase
TTS All-in-One Multilingual One single voice longanhuan_mtlv7 speaks 16 languages while keeping the same timbre Multilingual Synthesis

🎵 Music Generation

Use Case Description Example
Generate Music from Prompt Input music style description, auto-generate lyrics and sing Generate from Prompt
Generate Music from Lyrics Provide custom lyrics, AI composes and sings Generate from Lyrics

☎️ Call Center & Dialogue Analysis

Use Case Description Example
Real-Time Call Recognition Real-time speech recognition for call systems Microphone Real-Time Recognition
Real-Time Agent Response Synthesis Synthesize agent responses Streaming Synthesis
Custom Voice Synthesis Voice cloning for personalized TTS Voice Cloning Example
Custom Voice Synthesis by Qwen-Audio-3.0-TTS Voice cloning with the Qwen-Audio-3.0-TTS model Voice Cloning Example
Batch Call Recording Recognition Batch processing of call recordings Batch Mode Recognition
Qwen-Audio-3.0-ASR Real-Time Call Recognition Improve accuracy on product names and business terms with hotwords and dialog context Qwen-Audio-3.0-ASR Streaming Recognition
Qwen-Audio-3.0-ASR Call Recording Review Asynchronously transcribe calls and separate agent from customer via speaker diarization Qwen-Audio-3.0-ASR File Transcription

📢 Voice Broadcasting & Dubbing

Use Case Description Example
Information Broadcasting Convert text to speech for announcements Streaming Synthesis
Real-Time LLM Output Broadcasting Stream LLM outputs as speech LLM Streaming Synthesis
Dialect & Emotional Dubbing Set dialect and speaking style by instruction, or control emotion with inline tags Instruction-Controlled Synthesis
Signature Voice Broadcasting Clone a host or brand voice for consistent content broadcasting Voice Cloning Synthesis
Multilingual Broadcasting Switch among 16 languages with one All-in-One voice for internationalized content Multilingual Synthesis
Numeric & Professional Text Text Normalization auto-handles quantities, units and symbols for natural reading Text Normalization Showcase

🙋 Meeting Analysis

Use Case Description Example
Real-Time Meeting Recognition Real-time speech-to-text for meetings Microphone Real-Time Recognition
Real-Time Meeting Translation Real-time multilingual translation Real-Time Translation
Batch Meeting Recognition Batch processing of meeting recordings Batch Mode Recognition
Batch Meeting Translation Batch translation of meeting recordings Batch Mode Translation
Qwen-Audio-3.0-ASR Real-Time Meeting Captions Transcribe unlimited-duration meeting audio streams over a WebSocket connection Qwen-Audio-3.0-ASR Streaming Recognition
Qwen-Audio-3.0-ASR All-Day Meeting Transcription Asynchronously transcribe recordings up to 12 hours and tell speakers apart via diarization Qwen-Audio-3.0-ASR File Transcription

🎞️ Audio/Video Analysis

Use Case Description Example
Batch Audio/Video Recognition Batch speech recognition Batch Recognition
Rich Info Recognition Extract text/emotion/events from audio/video Rich Info Recognition
Summary & Q&A Summarize and answer questions from audio/video Transcribe & QA
Live & Classroom Captions Continuously caption long-running live streams and lectures with low latency Qwen-Audio-3.0-ASR Streaming Recognition

💬 Voice Chat

Use Case Description Example
Sentence-Level Recognition & Translation Real-time sentence recognition and translation Sentence-Level Example
Voice Chat with LLM Voice interaction with large language models Alibaba Cloud Documentation
Video Chat with LLM Video chat via multimodal large models Omni Example
End-to-End Real-Time Voice Chat WebSocket-based real-time voice conversation SDK and examples (companion chat, tool calling, reasoning routing) qwen-audio-3.0-realtime

👉 High-Concurrency Usage

For Java-based services, refer to the high-concurrency guides:

🎮 Gallery

✨ Community Projects

Project Description Platform
Cosight Contributed by community developer @MrGood4U. A realtime multimodal AI assistant desktop client: talk to it while it watches your screen, and let it draw annotations on the desktop to help explain Windows 10 or later

The Gallery showcases creative applications built with Bailian AI capabilities. Contribute your projects to inspire others!

👉 FAQ

👉 License

This project is licensed under the MIT License.

👉 Changelog

2026/08/14

  • Added Qwen-Audio-3.0-TTS speech synthesis and voice cloning examples, with voice style, dialect and emotion controllable by instruction.
  • Added the Qwen-Audio-3.0-ASR speech recognition examples, covering non-streaming recognition, streaming recognition (with hotwords and dialog context) and recorded-file transcription.

2026/07/14

  • Added Qwen-Audio-3.0-Realtime end-to-end real-time voice conversation example (companion chat, tool calling, and reasoning routing scenarios).

2026/06/16

  • Added Fun-Music (Bailing Music Generation) examples, supporting music generation from prompts and lyrics.
  • Improved graceful session finish in the Omni demo.
  • Added Omni function calling example.

2025/12/19

  • update default asr model to fun-asr
  • update default tts model to cosyvoice-v3-flash

2025/07/21

  • Added qwen-omni demo.
  • Added qwen-tts-realtime demo.

2025/06/17

  • Added English readme docs
  • TTS model update to cosyvoice-v2

2025/02/14

  • Added JavaScript examples for CosyVoice and Paraformer.

2025/01/24

  • Added local file recognition example with Opus format conversion.
  • Updated HTML streaming audio player for mobile/Safari compatibility.

2025/01/17

  • Unified Java examples as Maven projects.
  • Added voice cloning and Gummy translation examples.
  • Introduced Gallery section.

2024/08/14

2024/07/24

  • Added parallel speech recognition/synthesis examples.
  • Added story narration with real-time subtitles (Python/Java).
  • Added web-based AI Assistant example.

2024/07/19

  • Updated documentation structure with speech quality inspection examples.

2024/07/05

  • Added QA documentation.

2024/06/25

  • Initial release with Python/Java examples for microphone/file input and speech synthesis.