Comparison to FlexAttention

[michaelfeil]

Cool repo, fyi there is a new torch api https://github.com/pytorch/pytorch/blob/main/torch/nn/attention/flex_attention.py

[alexzhang13]

I'll check it out and run a comparison. Thanks for sharing!

[michaelfeil]

Above implementation might be just interesting, it uses block sparse / attention. Might do something different in the end.