Gen Composite Operators

[jeremylt]

Ok, the idea here is to launch all of the gen suboperators on separate streams and then sync once at the end.

Overall approach:

Loop over all operators

• create stream for suboperator
• try to run kernel for suboperator
• destroy stream

Sync all streams

Return active vec access

Loop over all operators

• fallback on any operators that couldn't run

[jrwrigh]

Would we want this for shared as well? In cases where e.g. Gen-simplex can't be used and falls back to shared.

Actually, can't this exist entirely outside of Gen vs Shared since it's a composite operator? (And thus doesn't necessarily have to do with how the operators are compiled, but how they are called?)

Probably showing my ignorance on how the GPU backends are setup... 😅

[jeremylt]

GPU-shared doesn't have its own operators. GPU-ref holds the operator implementation. The GPU-ref operators and GPU-gen operators don't really play nice together so I don't think we'll be able to easily launch both at the same time with how the code is currently designed.

[jeremylt]

@zatkins-dev we need to check perf, but this should help for composite operators where the suboperators that are all of the same basis type (all tensor or all non-tensor)

[jeremylt]

This also has a correctness fix, so I'll merge for now and we can perf tune going forward