[SPIRV] Enable scf.forall-based workgroup distribution#23684
Merged
Conversation
hanhanW
force-pushed
the
users/hanhanW/spirv-forall-distribution
branch
from
56d65a7 to
b3ed406
Compare
hanhanW
changed the base branch from
main
to
users/hanhanW/diff-spirv-forall-distribution
Contributor
Author
kuhar
reviewed
Mar 6, 2026
| static llvm::cl::opt<bool> clSPIRVTileDispatchUsingForall( | ||
| "iree-spirv-tile-dispatch-using-forall", | ||
| llvm::cl::desc("Enable tile and distribute to workgroups using scf.forall"), | ||
| llvm::cl::init(true)); |
Member
There was a problem hiding this comment.
should this be hidden? I don't realistically expect anyone to flip this flag
Contributor
Author
There was a problem hiding this comment.
I forgot to mention that I'll remove the option if CI is happy. And yes, it should be hidden if we preserve the flag.
| // CHECK-COUNT-8: %{{.+}} = spirv.KHR.CooperativeMatrixMulAdd %{{.+}}, %{{.+}}, %{{.+}} | ||
|
|
||
| // Matmul results are stored to workgroup memory for the epilogue. | ||
| // CHECK-COUNT-4: spirv.KHR.CooperativeMatrixStore %{{.+}}, %{{.+}}, %[[C9]], <RowMajor> |
Member
There was a problem hiding this comment.
Can we restore checks for the order loaded values?
hanhanW
force-pushed
the
users/hanhanW/spirv-forall-distribution
branch
from
b3ed406 to
e299cfc
Compare
kuhar
approved these changes
Mar 7, 2026
hanhanW
force-pushed
the
users/hanhanW/spirv-forall-distribution
branch
from
e299cfc to
52c80d2
Compare
hanhanW
requested review from
AWoloszyn,
Groverkss,
IanWood1,
MaheshRavishankar,
Max191,
ScottTodd,
Yu-Zhewen,
antiagainst,
benvanik,
bjacob,
jtuyls,
krzysz00,
nirvedhmeshram and
stellaraccident
as code owners
hanhanW
changed the base branch from
users/hanhanW/diff-spirv-forall-distribution
to
main
hanhanW
removed request for
antiagainst,
benvanik,
bjacob,
krzysz00 and
stellaraccident
hanhanW
removed request for
AWoloszyn,
Groverkss,
IanWood1,
Max191,
ScottTodd,
Yu-Zhewen,
jtuyls and
nirvedhmeshram
hanhanW
force-pushed
the
users/hanhanW/spirv-forall-distribution
branch
from
52c80d2 to
8af48c7
Compare
Switch the SPIRV backend from the legacy `TileAndDistributeToWorkgroups` pass to `TileAndDistributeToWorkgroupsUsingForallOp`. This aligns SPIRV with the LLVMGPU backend's distribution approach. The forall path also requires `ConvertAccGEMMToGEMM` before distribution to avoid accumulating matmul's shared_outs being bufferized to workgroup memory with a copy-out outside the forall, which violates the workgroup distribution verifier. Signed-off-by: hanhanW <hanhan0912@gmail.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
hanhanW
force-pushed
the
users/hanhanW/spirv-forall-distribution
branch
from
8af48c7 to
1a68b3b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The forall path also requires
ConvertAccGEMMToGEMMbefore distribution to avoid accumulating matmul's shared_outs being bufferized to workgroup memory with a copy-out outside the forall, which violates the workgroupdistribution verifier.