Self-Supervised Speech Recognition with Transformer-Based Feature Fusion SSF-FT

Prerequisites

1. created your Python environment(python 3.7 and above)

2. Clone our repo from a specific branch named noval_fusion_s3prl

   git clone -b noval_fusion_s3prl https://github.com/ussenuk/my_phd_fusion.git

3. Install the requirement and speechbrain cd speechbrain

pip install -r requirements.txt

pip install --editable .

Basic Structure of the code

1. Experimental code is located at : /recipes/LibriSpeech/ASR/CTC

2. Adjustment between the baseline setup and the elf-supervised Feature fusion Transformer can be found at : /speechbrain/lobes/models/transformer/

3. Our code experiments are within the LibriSpeech recipes under: https://github.com/ussenuk/my_phd_fusion/tree/noval_fusion_s3prl/recipes/LibriSpeech/ASR/CTC

4. Please before running the below scripts ensure that the right Transformer method is used under https://github.com/ussenuk/my_phd_fusion/tree/noval_fusion_s3prl/speechbrain/lobes/models/transformer

use TransformerASR_original.py and Transformer_original.py for baselines experiments

use TranformerASR_modified.py and Transformer_modified.py for SSF-FT

Baselines experiments scripts

1. Please before running the baselines script ensure that the right Transformer method is used https://github.com/ussenuk/my_phd_fusion/tree/noval_fusion_s3prl/speechbrain/lobes/models/transformer -> TransformerASR_original.py and Transformer_original.py.

2. Ensure you remane the name TransformerASR_original.py and Transformer_original.py to TranformerASR.py and Transformer.py

3. python ensemble.py hparams/train_with_wav2vec2_hubert_original.yaml

Self-supervised feature fusion scripts

1. Please before running the baselines script ensure that the right Transformer method is used https://github.com/ussenuk/my_phd_fusion/tree/noval_fusion_s3prl/speechbrain/lobes/models/transformer -> TransformerASR_modified.py and Transformer_modified.py.

2. Ensure you remane the TransformerASR_modified.py and Transformer_modified.py to TranformerASR.py and Transformer.py

3. python ensemble_fusion.py hparams/train_with_wav2vec2_hubert.yaml

Low-resource language

1. Please before running the baselines script ensure that the right Transformer method is used https://github.com/ussenuk/my_phd_fusion/tree/noval_fusion_s3prl/speechbrain/lobes/models/transformer -> TransformerASR_modified.py and Transformer_modified.py.

2. Ensure you remane the TransformerASR_modified.py and Transformer_modified.py to TranformerASR.py and Transformer.py

3. python ensemble_ln_fusion.py hparams/train_with_cdwav2vec_TERA_ln_Hubert_ln.yaml