Torch 2.0

[varshith15]

Is there a way to leverage torch2.0's compile using tensorrt as a backend directly? without all the current tedious process? https://pytorch.org/docs/stable/dynamo/get-started.html

And any thoughts on torch 2.0 in general? Has anyone tried it out?
I've tried it out for a few of the transformer models, there doesn't seem to be any improvement.
@pommedeterresautee @ayoub-louati

[pommedeterresautee]

Yes tensorRT is supported out of the box. However, it adds its own overhead and is not always best choice in my tests. Kernl runs on top of PyTorch 2.0. The 2.0 targets mostly for now training (and not inference).

[varshith15]

@pommedeterresautee any thoughts on Apache tvm?

[pommedeterresautee]

TVM was best for non GPU stuff. Recently they started to support better GPU through cutlass + adding possibility to program at block of threads level (CTAs), but IMO Triton is a better choice for now when Nvidia hw is your target