Our implementation is based on FairSeq. Dataset used in the experiments are Multi30k and IKEA.
Install the dependencies using pip
pip install -r requirements.txt
The method is designed to be plug-and-play, making it applicable to various low-resource corpora. A multimodal knowledge graph compatible with the specific corpus can be obtained through the execution of either crawl_direct.py
or crawl_indirect.py
:
python crawl_direct.py
python crawl_indirect.py
Subsequently, data enhancement can be achieved by utilizing the generate_pseudo_data.py
:
python generate_pseudo_data.py
For reference purposes, the Multi30k and IKEA datasets enhanced through our methodology have been made available here (password: yUxF).
If you find our paper and code useful in your research, please consider giving a star ⭐ and citation 📖.
@inproceedings{mmkg_mmt,
author = {Xu, Tianjiao and Liu, Xuebo and Wong, Derek F. and Zhang, Yue and Chao, Lidia S. and Zhang, Min and Gan, Tian},
title = {Exploiting Multimodal Knowledge Graph for Multimodal Machine Translation},
journal = {IEEE Transactions on Multimedia},
year = {2025},
}
Code released under the Apache-2.0 License. Dataset released under the CC BY-NC-SA 4.0.