`_convert_scalars_to_attrs` writes every lifted scalar as float32, retyping integer subgraphs and breaking `prepare_pt2e`

[john-rocky]

What happens

XNNPACKQuantizer.transform_for_annotation lifts every scalar argument of aten.add.Tensor
and aten.mul.Tensor into a buffer, and writes it as a float32 tensor regardless of the
node's own dtype:

https://github.com/pytorch/executorch/blob/main/backends/xnnpack/quantizer/xnnpack_quantizer_utils.py#L1160

float_tensor = torch.tensor(float(args[i]))

An integer node such as arange(n) + 0 therefore comes back float32, and everything
downstream of it is retyped. It happens with an empty quantizer — nothing has to be
annotated for the graph to change.

Repro (12 lines, no model download)

import torch
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import XNNPACKQuantizer
from torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e

class Tiny(torch.nn.Module):
    def __init__(self):
        super().__init__(); self.fc = torch.nn.Linear(4, 4)
    def forward(self, x):
        return self.fc(x)[:, torch.arange(4) + 0]      # <- the python scalar

m, ex = Tiny().eval(), (torch.randn(2, 4),)
gm = torch.export.export(m, ex).module()
gm(*ex)                                                # runs

p = prepare_pt2e(torch.export.export(m, ex).module(), XNNPACKQuantizer())
print({n: getattr(p, n).dtype for n in dir(p) if n.startswith("_tensor_constant")})
p(*ex)                                                 # IndexError

exported runs: torch.Size([2, 4])
promoted constants -> {'_tensor_constant_0': torch.float32}
prepared runs: NO — IndexError tensors used as indices must be long, int, byte or bool tensors

The exported graph holds add = aten.add.Tensor(arange, 0); after
transform_for_annotation it holds add = aten.add.Tensor(arange, _tensor_constant_0) with
_tensor_constant_0 float32.

On real models: two different symptoms, one cause

Transformers computes position ids and attention-mask indices by adding a python int to an
arange, so the promotion lands on an integer chain that is then used as an index. I ran
torch.export.export(...).module() followed by prepare_pt2e on seven graphs (the exported
module itself runs in every case; only the prepared one fails):

graph	fails with	at
cross-encoder/ms-marco-MiniLM-L6-v2	IndexError: tensors used as indices must be long, int, byte or bool tensors	index.Tensor(to, [unsqueeze_2, unsqueeze_11])
cross-encoder/ms-marco-MiniLM-L12-v2	same	same
sentence-transformers/all-MiniLM-L6-v2	same	same
BAAI/bge-small-en-v1.5	same	same
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2	same	same
openai/whisper-tiny decoder	same	index.Tensor(embed_positions.weight, [repeat])
BAAI/bge-reranker-base	RuntimeError: Tensor dtype mismatch! Expected: Int, Got: float	_assert_tensor_metadata(mul, dtype=torch.int32)

The three source lines the promotion lands on, in transformers 5.15.0:

• masking_utils.py:509-510 — q_arange = torch.arange(q_length) + q_offset,
  kv_arange = torch.arange(kv_length) + kv_offset, consumed as indices at line 516.
  This is the BERT/MiniLM row above.
• models/whisper/modeling_whisper.py:749 — position_ids = torch.arange(...) + past_key_values_length,
  consumed by self.weight[position_ids] in WhisperPositionalEmbedding.forward.
• models/xlm_roberta/modeling_xlm_roberta.py:154 —
  (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask, an int32
  chain whose recorded dtype assertion then fails. This is why XLM-R reports a dtype
  mismatch rather than an IndexError.

In each case the offset is a python int and is 0 for a plain forward pass, so the
arithmetic is a no-op that only exists to be retyped.

Suggested fix

The quantizer only has business with floating-point arithmetic, so skip the rest:

     for n in model.graph.nodes:
         if n.op != "call_function" or n.target not in [
             torch.ops.aten.add.Tensor,
             torch.ops.aten.mul.Tensor,
         ]:
             continue
+        # Only float arithmetic is quantizable, and rewriting an integer scalar as a
+        # float32 buffer silently retypes the whole integer subgraph downstream.
+        val = n.meta.get("val")
+        if val is None or not val.dtype.is_floating_point:
+            continue
         args = list(n.args)

I applied exactly that diff to the installed
backends/xnnpack/quantizer/xnnpack_quantizer_utils.py and re-ran the same seven probes:
all seven prepare_pt2e graphs now run. Float scalars are still lifted — whisper-tiny's
decoder keeps eight promoted float32 constants — so the skip removes nothing the quantizer
wanted.

Quantization still happens after the skip: with it in place, dynamic int8 for
ms-marco-MiniLM-L6 lowers to a 58.8 MB .pte against 91.0 MB for fp32, and the five
embedding/reranker models above hold their quality (the rerankers reproduce the fp32
ranking order over a six-document candidate list; the embedding models read worst cosine
0.9980–0.9996 against eager over eight sentences).

Versions

• executorch 1.4.0 (pip); the line is unchanged on main at cff6f4d
• torch 2.13.0, torchao 0.18.0, transformers 5.15.0
• macOS arm64

torchao/testing/pt2e/_xnnpack_quantizer_utils.py:1122 carries the same line.