Documentation: ELECTRA Pretraining

[stefan-it]

Dear TF Models Garden Maintainer,

today I recognized that TensorFlow 1.15 is no longer available as supported TPU version. That means, it is no longer possible to pretrain BERT and ELECTRA models with the official code bases.

As I was able to sucessfully pretrain Token Dropping BERT and TEAMS models (many thanks for open sourcing the implementation), I would like to know if it's already possible to pretrain "normal" ELECTRA models with TF Models Garden and if the necessary pretraining commands can be provided here and in the documentation.

Many thanks in advance!

[stefan-it]

Update: I could find BERT pretraining commands here:

https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrain.md

So only ELECTRA would be missing I think 🤗

[laxmareddyp]

Hi @Chen-Zhe,

Could you please check this in your free time.

Thanks