Possible to call the .cu files in python?

[vgoklani]

This is the best inference code i've seen, and I'd like to call it from python.

Specifically I'd like to create a class that stores a reference to the model and then create a generate function that does the forward pass.

Could you point me to some examples on how to call the CUDA code from python? thanks!

[GilesBathgate]

You can't call the .cu file as that is the source code. You can call the executable binary from python using os.system('./llama2_q4'). However, if you want to do the inference entirely in python, you could possibly use the original AWQ Tinychat

[GilesBathgate]

@vgoklani Or it can be done like this #11