8da4w silently skips layers that do not divide the group size, and 8da8w is implemented but rejected by config validation

[john-rocky]

Two things in the 8da4w path make a small model quietly come out nearly unquantized, with no way to tell from the output of the export.

1. Layers whose in_features do not divide the group size are skipped silently

examples/models/llama/source_transformation/quantize.py:146:

def filter_fn(m, fqn):
    if not isinstance(m, nn.Linear):
        return False
    parts = fqn.split(".")
    if "lora_a" in parts or "lora_b" in parts:
        return False
    if group_size == 0:
        return True
    return m.weight.shape[1] % group_size == 0

group_size defaults to 128. A model whose width is not a multiple of 128 therefore has most of its linears left in fp32, and nothing says so — the only output is if verbose: print("quantized model:", model), which prints the whole module tree rather than a count.

SmolLM2-135M is 576 wide (examples/models/smollm2/135M_config.json). Exported with qmode: 8da4w and embedding_quantize: "8,0" it comes out at 475.7 MB, against roughly 540 MB for the same model in fp32. With group_size: 64 the same command gives 101.8 MB. The first number is the one that looks like a successful quantized export.

A line saying how many linears were quantized and how many were skipped would have made this a five-second problem instead of a "why is my 135M model 475 MB" problem.

2. 8da8w is implemented but unreachable

quantize.py:134 handles it:

elif qmode in ("8da4w", "8da8w"):
    ...
    weight_dtype = torch.int4 if qmode == "8da4w" else torch.int8

and there is a linear_forward_8da8w at quantize.py:445. But extension/llm/export/config/llm_config.py:465 will not let the value through:

QMODE_OPTIONS: ClassVar[List[str]] = ["int8", "8da4w", "8da4w-gptq", "4w"]

ValueError: Got qmode 8da8w, but expected one of ['int8', '8da4w', '8da4w-gptq', '4w'],
or one of the regex patterns ['torchao:8da(\\d+)w', 'torchao:fpa(\\d+)w'].

The suggested alternatives do not cover it for an XNNPACK export. torchao:8da8w is refused later:

ValueError: Cannot use low-bit Ao ops (from qmode=torchao:...) while also delegating to XNNPack.

and qmode: int8 falls over on grouped-query attention:

RuntimeError: a and b must have same reduction dim, but got [s11, 576] X [192, 576].

(SmolLM2-135M, n_heads 9, n_kv_heads 3, head_dim 64 → 192.)

Why the pair matters together

Small models are exactly the ones that need 8-bit weights. Compared against the untouched model on the same prompts, Qwen2.5-0.5B at int4 answers that water boils at 215 degrees Fahrenheit where eager says 212 °F or 100 °C; SmolLM2-135M at int4 produces mojibake and stray tokens where eager, though repetitive, stays in English. Adding "8da8w" to QMODE_OPTIONS would make the already-written path reachable for them.

Measured with the executorch 1.4.0 wheel; every line quoted above is present unchanged on main at e4576d0.