[FAQ] Pretrained Model in the sense of TABPFN

[42elenz]

Dear Team,
I just have a qurstion regarding your model and what it could achieve.
Did you ever try a pretraining schema like the TABPFN team did to pretrain your model and ship a foundation model that could be finetuned or used out of the Box? I am not sure how your model specificly attends to rows and columns in comparison with the TABPFN but maybe it could be possible to copy the approach and use it with MAMBA. Also due to the different scaling in compute you could easily incooperate more data points, what seemes to be a limitiation at the moment.
Would this be something your planning on doing?

Best

[AnFreTh]

Thanks for reaching out. I have converted it to a discussion, since it is an interesting question where we do not have a definitive answer.
There is some work regarding MambaPFN already, i.e.: https://openreview.net/forum?id=4dOJ0PRY7R#discussion
However, the reported results have been mixed.

Since TabPFN v2 is now out and can handle 10k observations we are not yet committed in testing out autoregressive models further, but this might change in the future :)