Genny CUDA memory exception

[lilhoser]

Describe the bug
Genny using onnxruntimegenai.cuda package fails to load https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx

To Reproduce
Steps to reproduce the behavior:

1. Run genny with Debug_Cuda configuration selected.
2. Open C:\Downloads\models\Phi-3-mini-128k-instruct-onnx\cuda\cuda-int4-rtn-block-32 or C:\Downloads\models\Phi-3-mini-128k-instruct-onnx\cuda\cuda-fp16
3. Type a message, hit enter
4. See error

Expected behavior

Is this model supported?

Screenshots

Desktop (please complete the following information):

• OS: Windows 11

[MaxAkbar]

I had the same issue, I think it's an issue with the fp16. I can run it on Phi-3.5-mini-instruct-cuda-fp32-onnx.

[natke]

Hi @lilhoser, thank you for reporting this issue. We are looking into it

[MaxAkbar]

I noticed that Microsoft has released: microsoft/Phi-3.5-mini-instruct-onnx. I just tested the cuda-fp16 and has the same issue as noted above.