[WebNN] Support more features for GQA#27234
Conversation
Add support for GroupQueryAttention with: - do_rotary=true (cos_cache/sin_cache inputs) - Packed QKV (optional key/value inputs) - Optional past_key/past_value for prefill mode - Remove fp16->fp32 casting workaround Add ApplyRotaryEmbedding helper function. Fix decode stage by using qkv_sequence_length instead of has_past_key to distinguish prefill vs decode, and use runtime seqlens_k instead of static past_sequence_length for rotary position calculation.
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
run 'lintrunner -a' to make the CI happy |
|
Hmm, linter issues. I can't tell what it's complaining about though (why can't this linter be clearer? 🤨): - emscripten::val input,// Shape: [batch_size, sequence_length, num_heads, head_size]
- emscripten::val cos_cache,// Shape: [max_sequence_length, head_size / 2]
- emscripten::val sin_cache,// Shape: [max_sequence_length, head_size / 2]
- emscripten::val position_ids,// Shape: [batch_size, sequence_length] or [1]
+ emscripten::val input,// Shape: [batch_size, sequence_length, num_heads, head_size]
+ emscripten::val cos_cache,// Shape: [max_sequence_length, head_size / 2]
+ emscripten::val sin_cache,// Shape: [max_sequence_length, head_size / 2]
+ emscripten::val position_ids,// Shape: [batch_size, sequence_length] or [1] |
Head branch was pushed to by a user without write access
|
/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline,Windows GPU WebGPU CI Pipeline,Windows OpenVINO CI Pipeline |
|
/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline |
|
/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI |
|
/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run Test Linux CUDA x64 Release,Test Linux TensorRT x64 Release,web_Debug / build_onnxruntime_web,web_Release / build_onnxruntime_web |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run Linux QNN CI Pipeline |
|
No pipelines are associated with this pull request. |
1 similar comment
|
No pipelines are associated with this pull request. |
|
Azure Pipelines successfully started running 2 pipeline(s). |
|
Azure Pipelines successfully started running 1 pipeline(s). |
3 participants
Add support for GroupQueryAttention with:
Add ApplyRotaryEmbedding helper function.
Fix decode stage by using qkv_sequence_length to distinguish prefill vs decode, and use runtime seqlens_k instead of static past_sequence_length for rotary position calculation.