RoPE vs RPB

[Bezdarnost]

Hello,

First and foremost, thank you for your excellent work on this project.

I've noticed that support for Relative Position Bias (RPB) was removed starting with version 0.20.0. I would like to kindly request if you would consider re-enabling it.

The reason for this request is that models using RPB demonstrate significantly better performance when trained on a single resolution and then used for inference across a wide variety of different resolutions. With RPB, the relative positions within the attention window are always consistent, meaning these relationships are always well-trained.

In contrast, when changing resolutions at inference time, a model trained with RoPE tends to show a degradation in performance if it hasn't been exposed to those resolutions during training.

Thank you for your time and consideration.

[alihassanijr]

Thanks for your interest and for sharing your use case.
Bringing RPB support into the fused kernels, especially for backward pass, is not easily achievable without making performance compromises (in terms of runtime/thruput).

We do have plans to bring back some form of unfused implementations again (which were removed in 0.20.0), and the issue where that is being tracked is #262.

If you're specifically interested in the fused form, we might be able to bring RPB support for the Ampere FNA forward pass back at some point as well, at least as a one-off.

[Bezdarnost]

Thanks-RPB support in the Ampere FNA forward pass would already unblock me; a backward version would be even better. I’m also planning to run on Blackwell GPUs, so I probably won’t stop at Ampere-only support.

I’m preparing a paper on applying Neighborhood Attention (or a small modification) to low-level vision. I . This is a side project outside my day job, so my time is tight this month, but I expect to have more time soon. If you’re interested, I’d love to collaborate and include you as a co-author. I’m targeting ECCV 2026; if progress slows for more than ~4 months, I may switch to another venue.

Right now I’m blocked by the lack of a solid RPB path for both forward and backward. I’m not entirely sure which route I’ll take, but at the moment I’m leaning toward implementing everything in Gluon to keep the implementation simpler and make kernel installation easier.

[Bezdarnost]

btw, CONGRATS! With a new position in Nvidia

[alihassanijr]

Ampere FNA forward did include RPB in 0.17.5, so unless you need RPB + stride or even-sized kernels, that should still work?

RPB implementation for forward pass isn't too difficult itself.
One of the reasons we got rid of it was that we didn't have a good definition for RPB under even-sized kernels, stride, and causal masking.

Flex could also implement RPB, and we do have a Flex backend that you can modify.
The only thing I'd be careful about is verification -- in some cases the backward pass doesn't allclose, and unfortunately I haven't had a chance to follow up with PyTorch about this, especially since the implementation is tied to so many things in NATTEN.

[alihassanijr]

Also, I'd love to help push this -- I just don't think I can commit to spending a lot of time on this in the near future.
As-is I'm a little behind on pushing out new releases for NATTEN.

[Bezdarnost]

Thanks for the clarification!

I don’t expect to need even-sized kernels or causal masking (with a small possibility of revisiting causal). But most likely I will do some work on RPB + strides.(70% chance)

I’m also pretty busy right now but still very interested. I’ll do my best to send occasional updates, and before the paper is finished I’d prefer to keep the conversation over email or a similar private channel.