Update trained model

We are evalueting whether to add xLSTM to our internally hosted LLMs.

Can you provide a newer version than the huggingface https://huggingface.co/NX-AI/xLSTM-7b from April 2024?

@nejch

[lukfischer]

Hi Tobias!
Currently there is no newer xLSTM7b model available. But if you let us know what you are planning to do, we might have something fitting in our internal development.
(Spoiler: We are waiting for the acceptance of a submitted research, which you for sure will find interesting ;))
Feel free to get in touch e.g. via email: lukas.fischer@nx-ai.com
Thx for your interest!

[stefan-it]

Hey @lukfischer , I have one question about the xlstm-jax repo: do you accidentally have an (probably internally) version that is running with the latest version of jax? I wanted to do experiments with a Blackwell card, and for that the cuda version is too old (xlstm-jax used 12.6 for the pinned 0.4.34 version). Updating to latest jax version results in errors (several triton-jax errors from the mlstm_kernels repo). I have just started to get it running.

Update: I debugged it. The problem is that triton==3.1.0 is not working with newer Blackwell cards. When I update to triton==3.2.0 the code won't compile anymore.

[lukfischer]

Hi @stefan-it , right now we focus on the pytorch version of xlstm. Unfortunately there is no newer JAX version available at the moment. Hopefully in the near future we will be able to put some resources on xlstm JAX again.

Moin @lukfischer, our teams are hosting LLMs for internal consumption in Siemens and we'd be interested to add an xLSTM model to complement the transformer based LLMs we already have running.
The LLM sector is evolving so fast, that there is hesitancy because of the age of the 7B-xLSTM on huggingface to invest effort into adding it to our pallet right now.