[FEATURE REQUEST]: TabPFN2 Wrapper

[jduerholt]

Motivation

TabPFN2 shows impressive results, also for Regression (https://github.com/PriorLabs/TabPFN). The model is fully differentiable and can also quantify its uncertainty, so it should be also suitable for BO (there is also already one publication from 2023 on this: https://proceedings.mlr.press/v202/muller23a/muller23a.pdf). I propose to implement a botorch wrapper for TabPFN (and finetuned versions of it). Is there already somebody working on it? If not, is there interest in it?

Describe the solution you'd like to see implemented in BoTorch.

Implement a TabPFN wrapper, so that a TabPFN model can be used in botorch as native as a SingleTaskGP.

Describe any alternatives you've considered to the above solution.

No response

Is this related to an existing issue in BoTorch or another repository? If so please include links to those Issues here.

No response

Pull Request

None

Code of Conduct

• I agree to follow BoTorch's Code of Conduct

[saitcakmak]

cc @jelena-markovic

[Balandat]

@yucenli and @sdaulton have worked on this in the past.

[jduerholt]

Nice, is there are already something available open source? How was your impression of it? Do you see value there?

[Balandat]

We don't have anything in OSS yet but shouldn't be too hard to put something together for the community contributions folder or into a PR. @sdaulton, @yucenli do you know what the state of the code is and what would be needed for this?

I'll let Sam+Lily speak about the details, but my tl;dr was that it worked quite well in terms of optimization performance (though not substantially better than GP models), but was quite slow at least for how botorch acquisition function optimization is set up by default b/c we have to backprop through the full Transformer model for each acquisition function evaluation. This can definitely be sped up though.

[jduerholt]

We don't have anything in OSS yet but shouldn't be too hard to put something together for the community contributions folder or into a PR.

Just a branch or PR would be sufficient, then we can start playing with it based on this version and we do not need to start from scratch. If we find it useful for our usecases we would then finish the PR and try to fully integrate it.

[sdaulton]

@yucenli had worked on implementing this. I can clean that up and put up a PR

[jduerholt]

Sounds great. Thanks!