Is batch Inference supported?

[imneov]

BATCH inference can greatly improve the inference speed. Is there any plan to support it?

[zeux]

Not planned right now as this is a latency optimized single user implementation.

[leiwen83]

I see current proj is using cuda core for fma, and from profiling, single batch seems more likely to be memory bound op. So for the batch > 1, it would require more computation and tensor core maybe needed in that case?

[zeux]

Yeah that's correct, tensor cores are not useful for single-user token generation, but they become critical for batch inference (they could be useful for prefill acceleration but that's not part of the workloads I'm personally interested in atm).