[MLAS Feature Request] MatMulNBits faster implementation for fp16 input dtype

[qti-ashimaj]

Describe the feature request

Please add support for HQNBIT_CompInt8 computation path for MatMulNBits fp16 input datatype. Currently the performance of MatMulNBits is much slower for fp16 vs fp32 (~6x) on CPU.

Describe scenario use case

For FP16 input dtype, MatMulNBits computation is always falling back to HQNBIT_CompFp16 as there is no implementation present for HQNBIT_CompInt8 compute type.

[hariharans29]

Hi @qti-ashimaj : For the 6x slowness compared with fp32, how many bits in B per-element ? 4 or 8 ?

Also for the faster 8 bit compute, are you looking for 4 or 8 bit support for B ? What is the use-case ? Do you have a model ready for deployment ?