Welcome to Full-Duplex-Bench, with v1.0, v1.5, v2.0, and v3.0, a comprehensive framework designed to evaluate the conversational and turn-taking capabilities of spoken language models.
- (2026/5/20) Full-Duplex-Bench v3 Code & Data Release: The v3 codebase and benchmark data are now publicly available! Check out the
v3/folder for the full inference and evaluation pipeline. Download the benchmark data here. - (2026/5/10) Codebase Update for New Models and Bug Fixes: Add Gemini 3.1 Flash Live Preview to v1/v1.5, and update the codebase.
- (2026/4/15) Full-Duplex-Bench v3 Paper Released: The FDB-v3 paper is now on arXiv with a demo website.
- (2026/2/23) Full-Duplex-Bench v2 Framework Release: Introduced the V2 architecture with a real-time WebRTC orchestrator and automated AI examiner in the
v2/folder! - (2026/2/21) Codebase Update for New Models and Bug Fixes: Add Gemini 2.5 Native Audio & PersonaPlex, and update the codebase.
- (2025/8/22) v1.5 Server-client Model inference Code Release: Added server-client inference scripts under
v1_v1.5/model_inference/. - (2025/8/15) v1.5 Data Release: Added v1.5 dataset with overlap scenarios and metadata annotations under
v1_v1.5/dataset/. - (2025/8/14) v1.5 Evaluation Code Release: Added support for overlap handling with new metrics in Full-Duplex-Bench v1.5 under
v1_v1.5/evaluation/. - (2025/6/05) Paper & ASR Model Update: Replaced the ASR model with nvidia/parakeet-tdt-0.6b-v2, which offers more reliable time-aligned transcriptions for evaluation purposes.
Due to the evolution of evaluation paradigms—from static dataset evaluation to dynamic real-time interaction—this repository is organized into distinct architectures:
The legacy v1 and v1.5 pipelines evaluate models based on pre-recorded static datasets in an offline, server-client inference manner.
- Highlights (v1.0): Systematically assesses 4 dimensions: Pause Handling, Backchanneling, Smooth Turn-Taking, and User Interruption. (FDB v1.0 paper)
- Highlights (v1.5): Extends the benchmark with overlap scenarios including listener backchannel, side conversation, and ambient speech. (FDB v1.5 paper)
FDB-v2 is our actively evolving, state-of-the-art framework. It orchestrates real-time audio conversations (via WebRTC or WebSocket) between your target model (the Examinee) and an automated AI evaluator (the Examiner).
- Highlights: Dynamic multi-turn tasks, WebRTC Node.js orchestrator, conversational constraints, LLM-as-a-judge automated scoring. (FDB v2.0 paper)
- Use Case: Best for evaluating how well a model converses reactively in a live environment.
👉 Dive into v3 | Demo Website | Download Data
FDB-v3 (Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency) combines real human disfluent speech with multi-step tool use to evaluate voice agents under realistic conditions.
- What we built: Real human recordings annotated across 5 disfluency types (fillers, pauses, hesitations, false starts, self-corrections), paired with chained API calls across 4 task domains.
Depending on your goal, please navigate to the respective folder:
-
To run offline static evaluations or reproduce results from our v1.0/v1.5 papers:
Navigate to thev1_v1.5/directory to view datasets, setup offline inference, and compute static metrics. -
To benchmark a model using the latest real-time automated AI examiner (v2):
Navigate to thev2/directory and follow the combined Node.js and Python setup instructions. -
To evaluate voice agents on multi-step tool calling with real human disfluent speech (v3):
Navigate to thev3/directory and follow the setup instructions. Download the benchmark data from Google Drive.
If you found this research helpful, please consider citing our work:
@article{lin2025fdb_v1,
title={Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities},
author={Lin, Guan-Ting and Lian, Jiachen and Li, Tingle and Wang, Qirui and Anumanchipalli, Gopala and Liu, Alexander H and Lee, Hung-yi},
journal={arXiv preprint arXiv:2503.04721},
year={2025}
}
@article{lin2025fdb_v15,
title={Full-Duplex-Bench v1. 5: Evaluating Overlap Handling for Full-Duplex Speech Models},
author={Lin, Guan-Ting and Kuan, Shih-Yun Shan and Wang, Qirui and Lian, Jiachen and Li, Tingle and Lee, Hung-yi},
journal={arXiv preprint arXiv:2507.23159},
year={2025}
}
@article{lin2026fdb_v2,
title={Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner},
author={Lin, Guan-Ting and Kuan, Shih-Yun Shan and Shi, Jiatong and Chang, Kai-Wei and Arora, Siddhant and Watanabe, Shinji and Lee, Hung-yi},
journal={arXiv preprint arXiv:2510.07838},
year={2026}
}
@article{lin2026fdb_v3,
title={Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency},
author={Lin, Guan-Ting and Chen, Chen and Chen, Zhehuai and Lee, Hung-yi},
journal={arXiv preprint arXiv:2604.04847},
year={2026}
}
For questions, please feel free to submit an issue or contact Guan-Ting Lin (daniel094144@gmail.com).