Mistral Support

[fakerybakery]

Hi,
Great work on this! Is Mistral supported? Right now I only see GPT-J and Llama 2.
Thank you!

[dkmisra]

We currently dont support Mistral but please check again in mid-Jan for the new release. I would assume that it should be straightforward to replace the LLM calls with Mistral AI as we use the standard huggingface interface. So, one would replace the LLM call from Llama here and here to the Mistral code:

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

which are mentioned here: https://huggingface.co/docs/transformers/main/en/model_doc/mistral

That said, sometimes the code requires some small changes to be made elsewhere. E.g., the gold answer for GPTJ often uses space at the beginning (i.e., the correct token is " London" instead of "London"), whereas for the Llama2 it doesn't.

The refactoring that we are working on will take care of all these small changes and we aim to release it as soon as possible. If your need is more urgent, then let us know how to provide more specific guidance. And we would welcome any pull request that you want to provide.

[fakerybakery]

Thank you!

[Mihaiii]

And we would welcome any pull request that you want to provide.

I could provide pull requests because there are small typos (ex: lname: {best_lnum}), but I thought it's best to first wait for the refactoring. :)

BTW, yesterday I applied LASER to a 34b Yi model. To my knowledge, this is the first model on huggingface that is changed with your method. My intention is to further target other layers (basically apply LASER on multiple layers).

https://huggingface.co/Mihaiii/Pallas-0.5-LASER-0.1

[dkmisra]

@Mihaiii , this is exciting and we are looking forward to your PR. It maybe a good idea to wait for the refactoring and we are working to release it very quickly. I would hope to get it done by Jan 10th-15th.

[dkmisra]

We also plan to release a leaderboard style table on the website. We can add the numbers from your study and cite your HF release, once it is up, and if you want. Thanks again.

[fakerybakery]

And we would welcome any pull request that you want to provide.

I could provide pull requests because there are small typos (ex: lname: {best_lnum}), but I thought it's best to first wait for the refactoring. :)

BTW, yesterday I applied LASER to a 34b Yi model. To my knowledge, this is the first model on huggingface that is changed with your method. My intention is to further target other layers (basically apply LASER on multiple layers).

huggingface.co/Mihaiii/Pallas-0.5-LASER-0.1

Nice! Quick question - how much compute did you need to create the Yi version?

[Mihaiii]

Nice! Quick question - how much compute did you need to create the Yi version?

It's proportional with how many interventions you are willing to try out before settling on best result. I used an A100 for more than 6 hours of experiments. I'd assume it takes somewhere a little less than half an hour per experiment, part of that being the Yi model loading extremely slow and me loading it on each experiment due to memory constraints.

[Mihaiii]

We can add the numbers from your study and cite your HF release, once it is up, and if you want.

Of course I'd want. Thank you! :)
But that model is intermediate work - I'll apply LASER on multiple layers of the same single model instance.

[dkmisra]

Sounds good. @Mihaiii , one thing which we tried in many datasets, is to only focus on the last 20-30% of the layers, and MLP parameters, and try very aggressive reductions. We often used rate (in our code) as 9.9 and 9.95. In ρ term, this would be keeping only 0.01 and 0.005 fraction of the max rank (max{m, n}). I'd be curious to see what kind of gains these reduction do.

E.g., see the for loop here: https://github.com/pratyushasharma/laser/blob/main/src/intervention_llama2_bbh.py#L326

[Mihaiii]

I applied 6 interventions on 6 different layers and the end result of that experiment is not very good, but I messed up with some attn layers and I think that was a mistake (I should have target only mlp layers always).

I'm waiting for your refactoring and your changes mentioned here and then I'll retry.

[dkmisra]

Sounds good. Yes, I would not target attention at all and to begin with, I would start by only targetting a single layer. In our code, --lname fc_in means the first layer of MLP and --lname fc_out means the second layer of MLP. We do have an experiment with stacking in our paper and we do it in a greedy selection way.

If you want, I can take a quick look at your python code, but no pressure at all. I am trying to get refactoring done soon.