This is a GitHub reposiutory of the Epoch Spring Camp Final Project of Multi modal emotion recognition from Audio and Transcript.
We are given a RAVDESS data set containing audio clips and we need to train an audio RNN by connverting audios into mel spectrograms and then convert the audio clips into text transcripts and them simulating them with labelled sentences and finally training then with a text TNN like GRU. Then we need to fuse both these using early fusion or late fusion. Finally I have trained an AST Transformer model for Audio spectrograms.
Note : Please refer to the Report pdf file for more details regarding accuracy and evaluation metrics, outputs , Limitations and further improvements of the project. Also there are some instructions for running the poetry file and requirements.txt file.
Note : Transformer model like DistilBERT and BERT can be used for text RNN . This can be a good improvement in future.
The below is the link for the GitHub repository for the further improvements of the model (Aiming to use transformer based models instead of CNN and RNN) :
https://github.com/PathriVidyaPraveen/Multimodal-Emotion-recognition-from-Audio-and-Transcript-Further-Improvements