Sparsity training together with FP8 linear layer in torchao

[goldhuang]

Is sparsity training compatible with FP8 linear layer provided in torchao? They both convert nn.Linear to a new linear layer.
from torchao.sparsity.training import ( SemiSparseLinear, swap_linear_with_semi_sparse_linear )

from torchao.float8 import convert_to_float8_training

Do you have suggestions how to apply both in the same training?

[jerryzh168]

cc @vkuzo @jcaip

[jcaip]

@goldhuang Sorry, these two are not compatible currently. The fast sparsification routines we need for training are not available for FP8.