trainable radial attention?

[toyot-li]

@Radioheading @lmxyy Thanks for your awesome work!

I wonder could this function be directly used during training? https://github.com/mit-han-lab/radial-attention/blob/main/radial_attn/attn_mask.py#L150-L206

If not, would you have a schedule to release a trainable one (as indicated in the paper)?

Regards,
Toyota

[Radioheading]

Hi! Trainable Radial Attention is definitely on our roadmap, but currently its priority is somehow lower than supporting training-free acceleration methods like few-step LoRA and quantized attention.

You can look up BlockSparseAttention to implement Radial Attention with backward. This should be easy to implement : )

[toyot-li]

Thanks for your prompt response! I understand your priority consideration, and may I know your estimated schedule for trainable radial attention. Before that, I would attempt to develop it based on your advice.