Skip to content

Support Grouped Query Attention (GQA) and update flash attention options#41

Merged
janpfeifer merged 2 commits into
mainfrom
add-gqa-fusedattention
Jul 11, 2026
Merged

Support Grouped Query Attention (GQA) and update flash attention options#41
janpfeifer merged 2 commits into
mainfrom
add-gqa-fusedattention

Conversation

@janpfeifer

Copy link
Copy Markdown
Contributor

This PR implements Grouped Query Attention (GQA) in the XLA backend and updates the custom-call signature to align with the new compute interface:

  • Removed direct mask parameter and moved it into ScaledDotProductAttentionConfig.
  • Allowed numHeads to be a divisible multiple of numKVHeads (GQA/MQA).
  • Implemented GQA key/value head broadcasting prior to invoking the cuDNN flash-attention custom call.
  • Implemented GQA gradient reduction (reduce-sum) for dKey and dValue back to numKVHeads in the VJP backward pass.
  • Updated dispatcher and integration tests to match the new signature.

- Moved optional paramaters to `ScaledDotProductAttentionConfig`.
- Removed redundant `numHeads` and `numKVHeads` parameters.
… implementation by projecting the key/value heads.

Also adjusted the VJP implementation to reduce-sum the resulting VJP back to the original key/value heads.
@janpfeifer
janpfeifer merged commit 353c4f6 into main Jul 11, 2026
0 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant