int8 vs int8_float16 compute type

[palladium123]

When running on a GPU, are these identical?

[guillaumekln]

No they are different.

• With int8, non quantized layers are run in FP32.
• With int8_float16, non quantized layers are run in FP16.

See also the corresponding CTranslate2 documentation: https://opennmt.net/CTranslate2/quantization.html#supported-types

[christhegrand]

Is int8_float16 faster for inference?