🎧 Dissecting Temporal Understanding in Text-to-Audio Retrieval [ACM Multimedia 2024]

---

🧭 Table of Contents

• 🧩 Environment Instructions
  • 🛠️ Environment Installation
• 📦 Setup: Download Pre-trained Models
  • 1️⃣ HTSAT Model
  • 2️⃣ WavCaps Model
• 🧪 Running the Model on AudioCaps
  • 🔹 Training
  • 🔹 Evaluation
• 🎵 Running the Model on Clotho
  • 🔹 Training
  • 🔹 Evaluation
• 🔁 Running the Model on SynCaps
  • 🔹 Training
  • 🔹 Evaluation
• 🧬 Generating SynCaps
• 📖 Citation
• ✉️ Contact
• 🙏 Acknowledgements
• ⚖️ License

🧩 Environment Instructions

🛠️ Environment Installation

conda env create -f environment.yml
conda activate pt39

---

📦 Setup: Download Pre-trained Models

1️⃣ HTSAT Model

Download the HTSAT.ckp model from this Google Drive link
and place it under:

retrieval/pretrained_models/audio_encoders/

More info can be found in the WavCaps repo.

Alternatively, use the following commands:

cd retrieval
mkdir -p pretrained_models/audio_encoder
gdown --output pretrained_models/audio_encoder/HTSAT.ckpt "https://drive.google.com/uc?id=11XiCDsW3nYJ6uM87pvP3wI3pDAGhsBC1"

---

2️⃣ WavCaps Model

Download the pre-trained WavCaps model:

mkdir pretrained
gdown --output pretrained/HTSAT-BERT-PT.pt "https://drive.google.com/uc?id=1il6X1EiUPlbyysM9hn2CYr-YRSCuSy2m"

Then, update the pretrain_path in:

retrieval/settings/train_ac_trans.yaml

to point to the full path of your downloaded model.

---

🧪 Running the Model on AudioCaps

🔹 Training

From the retrieval folder, run:

python train.py   --config settings/train_ac_trans.yaml   --exp_name <experiment_name>   --lambda_new_loss 0   --val_filename val_audioset_5_orig   --train_filename train_audioset_orig   --test_filename test_full   --seed 18

Notes:

• Replace the pretrain_path in settings/train_ac_trans.yaml with your pretrained model path.
• Set lambda_new_loss to:
  • 10 → reproduce experiments using the additional loss.
  • 0 → use the standard text–audio contrastive loss.
• You can specify dataset versions:
  • Original AudioCaps:

    --train_filename train_audioset_orig     --val_filename val_audioset_5_orig     --test_filename test_full

  • Uniform version:

    --train_filename train_audioset_rearranged     --val_filename val_audioset_5_rearranged     --test_filename test_audioset_5_rearranged

Training logs and final results are saved under:

retrieval/outputs/<exp_name>/logging

---

🔹 Evaluation

To evaluate on AudioCaps (original version):

python test_improved.py   --config settings/inference_ac_orig.yaml   --exp_name <experiment_name>   --lambda_new_loss 0   --val_filename val_audioset_5_orig   --train_filename train_audioset_orig   --test_filename test_full   --seed 19

• Update the checkpoint path in:

  retrieval/settings/inference_ac_orig.yaml
  

• Pre-trained checkpoints used in the paper are available here.

---

🎵 Running the Model on Clotho

🔹 Training

Run the following from the retrieval folder:

python train.py   --config settings/train_clotho_trans.yaml   --exp_name clotho_train   --seed 18   --lambda_new_loss 10   --train_filename train_individual

Update the path of the pretrained model in:

retrieval/settings/train_clotho_trans.yaml

---

🔹 Evaluation

python test_improved.py   --config settings/inference_clotho.yaml   --exp_name clotho_eval   --lambda_new_loss 0   --train_filename train_individual   --seed 18

• Update the checkpoint path in:

  retrieval/settings/inference_clotho.yaml
  

• Paper checkpoints: Google Drive link.

---

🔁 Running the Model on SynCaps

🔹 Training

python train.py   --config settings/train_syncaps.yaml   --exp_name syncaps_train   --lambda_new_loss 0   --seed 19

---

🔹 Evaluation

python test_improved.py   --config settings/inference_syncaps.yaml   --exp_name syncaps   --lambda_new_loss 0   --seed 19

• Update the checkpoint path in:

  retrieval/settings/inference_syncaps.yaml
  

• Checkpoints used in the paper can be found here.

---

🧬 Generating SynCaps

To generate SynCaps, run:

python retrieval/tools/syncaps_gen.py

---

📖 Citation

If you find our work useful, please cite the following:

@InProceedings{Oncescu24,
  author       = {Andreea-Maria Oncescu and Joao~F. Henriques and A. Sophia Koepke},
  title        = {Dissecting Temporal Understanding in Text-to-Audio Retrieval},
  booktitle    = {ACM International Conference on Multimedia},
  year         = {2024},
  doi          = {https://doi.org/10.1145/3664647.3681690},
}

@article{mei2023wavcaps,
  title={WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research},
  author={Mei, Xinhao and Meng, Chutong and Liu, Haohe and Kong, Qiuqiang and Ko, Tom and Zhao, Chengqi and Plumbley, Mark D and Zou, Yuexian and Wang, Wenwu},
  journal={arXiv:2303.17395},
  year={2023}
}

@inproceedings{piczak2015dataset,
  title = {{ESC}: {Dataset} for {Environmental Sound Classification}},
  author = {Piczak, Karol J.},
  booktitle = {{Association for Computing Machinery (ACM) Conference} on {Multimedia}},
  year={2015},
}

---

✉️ Contact

Maintained by Andreea.
For questions and discussions, contact:
📩 oncescuandreea@yahoo.com

---

🙏 Acknowledgements

This codebase is based on WavCaps.

Supported by:

• EPSRC DTA Studentship
• Royal Academy of Engineering (RF\201819\18\163)
• DFG: SFB 1233, Project 276693517
• DFG EXC 2064/1 – Project 390727645

Special thanks to Samuel Albanie and Bruno Korbar for helpful feedback and suggestions.

---

⚖️ License

This work is licensed under a
Creative Commons Attribution–NonCommercial International 4.0 License.