mT0-xxl finetuning

[sh0tcall3r]

Hello!
Thanks a lot for your job! I'm using mT0-xxl for question answering task, however it performs with not so high quality I expected it to do. So I'm trying to finetune the model a little bit. If I understood correctly, first of all I should get checkpoint and gin file for the model I want to finetune. Could you please share with these?
And is it possible to finetune it with torch or tf is the only way?

[Muennighoff]

Hey there are some more details on mT0 fine-tuning here: #12
The config is here: #6 (comment)

[sh0tcall3r]

Thanks for reply! Will try mentioned config.

[sh0tcall3r]

Hey @Muennighoff , It's seems that I still can't get a couple of things. Would be very appreciate If you could give me a hand here.
Well, I need to finetune your model mT0-xxl (not the initial T5X-xxl), so accordingly to the manual https://github.com/google-research/t5x/blob/main/docs/usage/finetune.md I need 3 components (excluded SeqIO Task, which is clear as for now) to proceed:

1. Checkpoint -- Could you please share with mT0-xxl checkpoint? In the manual all used checkpoints are the TensorFlow weights etc, but on the HuggingFace there are only PyTorch weights. So I do need either mT0-xxl checkpoint in TensorFlow or finetune the model in PyTorch (is it even possible?)
2. Gin file for the model to finetune (mT0-xxl in the case) -- Could I use the default one like https://github.com/google-research/t5x/blob/main/t5x/examples/t5/mt5/xxl.gin?
3. Gin file configuring finetuning process -- I write it by my own based on https://github.com/google-research/t5x/blob/main/t5x/configs/runs/finetune.gin with some overrides, right?
   Please, correct me if I wrong in some points.

[Muennighoff]

There's a t5x ckpt here: https://huggingface.co/bigscience/mt0-t5x
I don't remember which size that model is though; I don't have the other ones anymore, maybe @adarob does

For 2. & 3., yes I think so

[adarob]

This does appear to be XXL

…

On Thu, May 18, 2023 at 5:02 AM Niklas Muennighoff ***@***.***> wrote: There's a t5x ckpt here: https://huggingface.co/bigscience/mt0-t5x I don't remember which size that model is though; I don't have the other ones anymore, maybe @adarob <https://github.com/adarob> does For 2. & 3., yes I think so — Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIJV2APKVDRBUQIFQ6JKQ3XGXQQZANCNFSM6AAAAAAYCE4Z34> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sh0tcall3r]

Thanks a lot, guys!