WER for conformer update

[gandroz]

Hi,
I've just ended a training of a conformer using the sentencepiece featurizer on LibriSpeech over 50 epochs.
Here are the results if you want to update your readme:

dataset_config:
    train_paths:
      - /data/datasets/LibriSpeech/train-clean-100/transcripts.tsv
      - /data/datasets/LibriSpeech/train-clean-360/transcripts.tsv
      - /data/datasets/LibriSpeech/train-other-500/transcripts.tsv
    eval_paths:
      - /data/datasets/LibriSpeech/dev-clean/transcripts.tsv
      - /data/datasets/LibriSpeech/dev-other/transcripts.tsv
    test_paths:
      - /data/datasets/LibriSpeech/test-clean/transcripts.tsv

Test results:
G_WER = 5.22291565
G_CER = 1.9693377
B_WER = 5.19438553
B_CER = 1.95449066
BLM_WER = 100
BLM_CER = 100

The strange part is that I dot the same metrics on test-other dataset hmmm...

[nglehuy]

@gandroz Wow cool, if you got the same result for test-other then you should check the transcript file to see if it points to test-other files. And you should check the test-clean transcripts file too.
Anyway, I'm thinking that maybe the authors have some tricks that reduce the result to 2.7% that we didn't see.

[nglehuy]

And one more thing is that there's a very small difference between greedy and beam search at this kind of WER percent, so we can ignore the difference and test only on greedy to see if it reduces to near 2.7-3%, for getting faster results

[gandroz]

I'll try to continue training for several epochs, training seems not to have ended. I'll read the paper again to look for any clue on how to reduce WER even more.
But I dont have anything special in my transcripts, both test-clean and test-other are well segregated.

[nglehuy]

@gandroz You should check or generate the transcript file again, may be when creating test-other transcript file, you point to the test-clean directory.
If everything is right, then it's so weird haha 😆

[gandroz]

I checked both files, my config file too and got the same results. So weird. I'll try to debug to find any mistake Le sam. 23 janv. 2021 13:03, Nguyễn Lê Huy <notifications@github.com> a écrit :

…

@gandroz <https://github.com/gandroz> You should check or generate the transcript file again, may be when creating test-other transcript file, you point to the test-clean directory. If everything is right, then it's so weird haha 😆 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJCXOANR2CFFQ2EDBTOUSDDS3MFP3ANCNFSM4WOP6C2A> .

[gandroz]

I found why I always got the same test metrics.... I tested on the test-clean dataset and it saved a test.tsv file, but each time I performed another test, as there was already an existing file, only the metrics were computed and no inference was done. I've cleaned this file and have launched another test with the test-other dataset to continue the update.

[ncilfone]

@gandroz Can you post your full config file you are using to generate the ~5% WER results?

Thanks!!!

[gandroz]

@ncilfone sure !

speech_config:
  sample_rate: 16000
  frame_ms: 25
  stride_ms: 10
  num_feature_bins: 80
  feature_type: log_mel_spectrogram
  preemphasis: 0.97
  normalize_signal: True
  normalize_feature: True
  normalize_per_feature: False

decoder_config:
  output_path_prefix: /data/models/asr/conformer_sentencepiece_subword
  model_type: unigram
  target_vocab_size: 1024
  blank_at_zero: True
  beam_width: 5
  norm_score: True
  corpus_files:
    - /data/datasets/LibriSpeech/train-clean-100/transcripts.tsv
    - /data/datasets/LibriSpeech/train-clean-360/transcripts.tsv
    - /data/datasets/LibriSpeech/train-other-500/transcripts.tsv

model_config:
  name: conformer
  encoder_subsampling:
    type: conv2d
    filters: 144
    kernel_size: 3
    strides: 2
  encoder_positional_encoding: sinusoid_concat
  encoder_dmodel: 144
  encoder_num_blocks: 16
  encoder_head_size: 36
  encoder_num_heads: 4
  encoder_mha_type: relmha
  encoder_kernel_size: 32
  encoder_fc_factor: 0.5
  encoder_dropout: 0.1
  prediction_embed_dim: 320
  prediction_embed_dropout: 0.1
  prediction_num_rnns: 1
  prediction_rnn_units: 320
  prediction_rnn_type: lstm
  prediction_rnn_implementation: 1
  prediction_layer_norm: True
  prediction_projection_units: 0
  joint_dim: 320
  joint_activation: tanh

learning_config:
  augmentations:
    after:
      time_masking:
        num_masks: 10
        mask_factor: 100
        p_upperbound: 0.05
      freq_masking:
        num_masks: 1
        mask_factor: 27

  dataset_config:
    train_paths:
      - /data/datasets/LibriSpeech/train-clean-100/transcripts.tsv
      - /data/datasets/LibriSpeech/train-clean-360/transcripts.tsv
      - /data/datasets/LibriSpeech/train-other-500/transcripts.tsv
    eval_paths:
      - /data/datasets/LibriSpeech/dev-clean/transcripts.tsv
      - /data/datasets/LibriSpeech/dev-other/transcripts.tsv
    test_paths:
      - /data/datasets/LibriSpeech/test-clean/transcripts.tsv
      - /data/datasets/LibriSpeech/test-other/transcripts.tsv
    tfrecords_dir: null

  optimizer_config:
    warmup_steps: 10000
    beta1: 0.9
    beta2: 0.98
    epsilon: 1e-9

  running_config:
    batch_size: 2
    accumulation_steps: 4
    num_epochs: 50
    outdir: /data/models/asr/conformer_sentencepiece_subword
    log_interval_steps: 300
    eval_interval_steps: 500
    save_interval_steps: 1000
    checkpoint:
      filepath: /data/models/asr/conformer_sentencepiece_subword/checkpoints/{epoch:02d}.h5
      save_best_only: True
      save_weights_only: False
      save_freq: epoch
    states_dir: /data/models/asr/conformer_sentencepiece_subword/states
    tensorboard:
      log_dir: /data/models/asr/conformer_sentencepiece_subword/tensorboard
      histogram_freq: 1
      write_graph: True
      write_images: True
      update_freq: 'epoch'
      profile_batch: 2

I used a sentencepiece (unigram) model as vocab, currently trying with the BPE version

[ncilfone]

Thanks @gandroz!

Is that the vocab here: vocabularies/librispeech_train_4_1030.subwords

Edit: Based on the config it seems like you might generate one before training?

Also is this just single GPU training?

[gandroz]

no it's not that vocab. However, you can train yours with script\generate_vocab_sentencepiece.py giving your config file.
And I'm training on two GTX 1080Ti. It took soooo long to train, I'm looking for a way to pre-compute the fbanks as they are computed on the fly which might take some time.

[ncilfone]

Yeah just realized that you generate it based on the config options. Thanks for letting me know!

I'm assuming you are doing the featurization of the WAV files in TF as the stft etc. should be a bit faster on the GPU. DALI might be another place to look too although I've never used it...

[ncilfone]

Final question I promise... It looks like you are using and tokens in SentencePiece but I'm guessing the text featurizer for the LibriSpeech transcripts doesn't have those? Or do you pad them onto each one?

[gandroz]

I think the best way to accelerate processing is to pre-process fbank just as it done on fairseq.
For your information, featurization is done by the class tensorflow_asr\featurizers\speech_featurizers.py::TFSpeechFeaturizer.

I'm guessing the text featurizer for the LibriSpeech transcripts doesn't have those? Or do you pad them onto each one?

I'm not sure to understand well your question. Sentencepiece is an unsupervised text tokenizer and detokenizer so you have to train a model on the transcripts from LibriSpeech. Tokenized transcripts are padded to the biggest sentence during training for each batch.