Target-Speaker Voice Activity Detection with Chunk-Level Speaker Queries

This repository provides an implementation of a Target-Speaker Voice Activity Detection (TS-VAD) system that leverages chunk-level speaker queries. The system is designed to accurately detect the presence of a target speaker in audio streams, even under challenging acoustic conditions.

This study was inspired by prior work on sequence-to-sequence neural diarization with automatic speaker detection and representation (SSND).

This public repository provides the implementation code only. Trained model weights and pretrained checkpoints are not publicly released.

Related Work

This study was inspired by the following work:

Cheng, Ming, Yuke Lin, and Ming Li. "Sequence-to-sequence neural diarization with automatic speaker detection and representation." IEEE Transactions on Audio, Speech and Language Processing (2025).

Citation

If you find this work useful in your research, please cite the following paper:

@inproceedings{Tawara:26,
  title={Target-Speaker Voice Activity Detection with Chunk-Level Speaker Queries},
  author={Naohiro Tawara and Shota Horiguchi},
  booktitle={Proc. ICASSP},
  pages={22202--22206},
  year={2026}
}

Environment Setup (conda)

You can create a conda environment named chunkwise_tsvad and install Python 3.12.9 with the following commands:

conda create -n chunkwise_tsvad python=3.12.9
conda activate chunkwise_tsvad

To install the required packages, use the provided requirements.txt file:

pip install -r requirements.txt

Usage

Prepare your data

Before training or inference, you need to prepare your dataset in the expected format. This typically involves organizing your audio files and creating a YAML configuration file that describes the dataset structure, including paths to audio files, speaker labels, and any relevant metadata.

Example dataset configuration files can be found in the egs/dihard3/v0/databases/ directory. Make sure to customize these files according to your dataset and experimental setup.

For Simulated VoxCeleb dataset, you can set up the data by running the following command from the egs/sim_voxceleb/v0 directory:

Please note that the actual audio files are not included in this repository due to licensing restrictions. You will need to prepare the audio data yourself and place it in the appropriate directory structure as expected by the setup script.

cd egs/dihard3/v0/databases/sim_vox36k/
bash setup.sh

For the DIHARD III dataset, you can set up the data by running the following command from the egs/dihard3/v0 directory:

cd egs/dihard3/v0/databases/dihard3
bash setup.sh

Training

To train the TS-VAD model, run the following command from the egs/dihard3/v0 directory:

python train_ssnd.py --config-path conf --config-name config.yaml

You can customize the training configuration by editing the YAML files in conf/.

Inference

To perform inference (diarization) using a trained model, use the provided script:

python diar/infer.py --model_path <MODEL_CKPT_PATH> --database <DATABASE_YAML> --protocol <PROTOCOL_NAME> --subset <SUBSET>

Replace <MODEL_CKPT_PATH>, <DATABASE_YAML>, <PROTOCOL_NAME>, and <SUBSET> with your actual checkpoint, database config, protocol, and subset (e.g., test or dev).

Extracting Embeddings

To extract speaker embeddings, run:

python diar/infer_get_embedding.py --model_paths <MODEL_CKPT_PATH> --database <DATABASE_YAML> --protocols <PROTOCOL_NAME> --subset <SUBSET> --output <OUTPUT_DIR>

Example (using SLURM scripts)

You can also use the shell scripts in experiments/ for batch training, inference, and embedding extraction. For example:

bash experiments/ssnd_dynamic.sh
bash experiments/ssnd_infer.sh
bash experiments/ssnd_get_embeddings.sh

Please edit the scripts to match your environment and requirements.

License

Please refer to the LICENSE file for details.