[GPU] xattention_block_size 256 support. #33485
Conversation
…ed with float precision to avoid an onverflow zp.
riverlijunjie
left a comment
riverlijunjie
left a comment
There was a problem hiding this comment.
Some minor comments, totally LGTM.
| svmptr_t sparse_mask_base [[type("svmptr_t")]], | ||
| svmptr_t wg_sparse_mask_base [[type("svmptr_t")]], | ||
| bool validate, | ||
| int SPARSE_BLOCK_SIZE, |
There was a problem hiding this comment.
Do we really need this parameter if it is a macro?
There was a problem hiding this comment.
Yes, we need. SPARSE_BLOCK_SIZE is a runtime parameter now for PA kernel, instead of a compile time jit const.
| res_event = {execute_stage(res_event, instance, xattn_estimate_find_block)}; | ||
| res_event = {execute_stage(res_event, instance, xattn_estimate_post_proc)}; | ||
| if (!bypass_xattn(params)) { | ||
| if (rt_params->xattn_block_size == 128) { |
There was a problem hiding this comment.
If xattn_block_size is fixed value, we don't need add_stage for both 128 and 256.
There was a problem hiding this comment.
Unfortunately xattn_block_size is a compile time jit const for xattention kernels, while it is also a runtime parameter of model with PA node. This means users can dynamically switch it from time to time during inferencing. So this PR has to create two stages (one for 128, the other for 256) to switch in fly, accordingly.
4 participants
Details:
Tickets: