[Feature]: Support Re‑Quantizing Per‑Tensor FP8 Models to Other Dtypes

[yiliu30]

Feature Description

As the title.

Motivation and Use Case

• Input:
  Models whose weights have already been quantized to FP8 with per‑tensor granularity, such as: https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512
  Output:
  Re‑quantized models in other dtypes, e.g. W4A16

• The relevant implementation starting point is here:
  

  auto-round/auto_round/utils/model.py

  Lines 1070 to 1075 in 6190869

  	if layer.__class__.__name__ == "CompressedLinear":
  	dq_weight = layer.compressor.decompress_module(layer)
  	else:
  	weight_scale = layer.weight_scale if hasattr(layer, "weight_scale") else layer.weight_scale_inv
  	data_type = getattr(layer, "data_type", None)
  	dq_weight = dequant_block_fp8_weight(layer.weight, weight_scale, layer.block_size, data_type=data_type)

Alternatives Considered

No response

Definition of Done

• Allow taking mistralai/Devstral-2-123B-Instruct-2512 as input, and W4A16 as output.
• UT

Additional Context

No response

[SwekeR-463]

I would love to work on this.

My initial thoughts are for allowing Per-Tensor FP8 quant we have to make block_size optional in dequant_block_fp8_weight and accordingly add support for per tensor.

[yiliu30]

Hey @SwekeR-463, thanks for your interest in this feature!
We can add a new branch in convert_fp8_layer_to_linear to handle the per‑tensor case.

Also, @scopophobic has shared some design ideas to make that part more modular—you may want to take a look. #1283 (comment)
But no worries, we can handle it in parallel; feel free to focus on the per‑tensor case first.

[SwekeR-463]

Yes, first let me do the per-tensor case.

And for the test part where to add it and does it has to be separate or in test_autoround.py?

[yiliu30]

Yes, first let me do the per-tensor case.

And for the test part where to add it and does it has to be separate or in test_autoround.py?

Both are okay :)

[SwekeR-463]

@yiliu30 have raised the PR, do verify the tests part that I have added and waiting for review.