Right now the model is being finetuned on top of the pretraining explained in the t5 paper. We should try it out without pretraining.