feat: custom gathered-expert MoE combine for silu/FP8 (specifically Qwen 3.5 family)

[NatTuck]

This speeds up Qwen3.5/3.6 token generation significantly at low batch size on Gaudi 2 - like 2x on Qwen 3.6 35B-A3B at BS=1. Also speeds up Qwen3.5-122B (nearly 2x) and 397B (1.5x) significantly, with smaller gains as BS increases up to about BS=8.

Vibe-coded based on PR #1673 for Minimax M3. Specific to fp8, probably doesn't need to be. Gated on an env var that probably wants to get cleaned up.

I think there's more performance to be gained on these MoE models, and this probably also applies to stuff like Minimax M2, but this looks like a big enough win that it's worth sharing before I keep going.

This seems to work based from:

vllm-gaudi: 9cae8b6
[CI] Fix Scorecard publish break; move SARIF filter to a separate job (#1718)

vllm core: 0820125ae99f719f3cd8b0c26c0d3f9789e187e3
[Bugfix][Parser] Emit REASONING_END for Inkling tool calls that follow no thinking block (#50528)

[NatTuck]

This should be up to date with main as of now.

It doesn't look like this approach applies to Minimax M2.7, so this is just Qwen 3.5 family (including Qwen 3.6 35B-A3B and probably Qwen 3.8).

I think this is ready for someone to take a look at it.

[iboiko-habana]

@NatTuck thanks for contribution. Please apply next comments

1. Please keep general vllm env vars convention
   All env vars in this plugin are prefixed with VLLM_ (e.g. VLLM_MINIMAX_M3_MOE_DECODE_GATHER, VLLM_HPU_FORCE_CHANNEL_FP8). The new knobs HPU_MOE_GATHER* don't follow this. Please rename them to VLLM_HPU_MOE_GATHER, VLLM_HPU_MOE_GATHER_MAX_TP, VLLM_HPU_MOE_GATHER_VERIFY, VLLM_HPU_MOE_GATHER_VERIFY_DIR, and VLLM_HPU_MOE_GATHER_VERIFY_LAYERS for consistency and discoverability.

2. Register them in envs.py (add to the TYPE_CHECKING block and the environment_variables dict), and read them as envs.VLLM_HPU_MOE_GATHER in the op instead of raw os.environ.
   i.e.

diff --git a/vllm_gaudi/envs.py b/vllm_gaudi/envs.py
--- a/vllm_gaudi/envs.py
+++ b/vllm_gaudi/envs.py
@@ -15,6 +15,11 @@ if TYPE_CHECKING:
     VLLM_MINIMAX_M3_MOE_TOKEN_TILE: int = 512
     VLLM_MINIMAX_M3_MOE_DECODE_GATHER: bool = True
     VLLM_MINIMAX_M3_MOE_GATHER_MAX_TOKENS: int = 16
+    VLLM_HPU_MOE_GATHER: bool = False
+    VLLM_HPU_MOE_GATHER_MAX_TP: int = 64
+    VLLM_HPU_MOE_GATHER_VERIFY: bool = False
+    VLLM_HPU_MOE_GATHER_VERIFY_DIR: Optional[str] = None
+    VLLM_HPU_MOE_GATHER_VERIFY_LAYERS: int = 40
 
 # The begin-* and end* here are used by the documentation generator
 # to extract the used env vars.
@@ -78,6 +83,29 @@ environment_variables: dict[str, Callable[[], Any]] = {
     "VLLM_MINIMAX_M3_MOE_GATHER_MAX_TOKENS":
     lambda: int(os.environ.get("VLLM_MINIMAX_M3_MOE_GATHER_MAX_TOKENS", "16")),
+
+    # EXPERIMENTAL custom gathered-expert FP8 MoE combine (silu only).
+    # Off by default; falls back to the stock Habana fused op when disabled
+    # or when tokens*top_k exceeds VLLM_HPU_MOE_GATHER_MAX_TP.
+    "VLLM_HPU_MOE_GATHER":
+    lambda: os.environ.get("VLLM_HPU_MOE_GATHER", "0").lower() in ("1", "true"),
+
+    # Upper bound on tokens*top_k for which the custom gather path is used.
+    # Above this the stock fused op is faster and is used instead.
+    "VLLM_HPU_MOE_GATHER_MAX_TP":
+    lambda: int(os.environ.get("VLLM_HPU_MOE_GATHER_MAX_TP", "64")),
+
+    # Validation mode: run both the custom and stock paths and record output
+    # pairs for offline FP8-ULP comparison. Only effective together with
+    # VLLM_HPU_MOE_GATHER.
+    "VLLM_HPU_MOE_GATHER_VERIFY":
+    lambda: os.environ.get("VLLM_HPU_MOE_GATHER_VERIFY", "0").lower() in ("1", "true"),
+
+    # Directory for verify-mode output captures. If unset, nothing is written.
+    "VLLM_HPU_MOE_GATHER_VERIFY_DIR":
+    lambda: os.environ.get("VLLM_HPU_MOE_GATHER_VERIFY_DIR", None),
+
+    # Max captures saved per token count T (per rank) in verify mode.
+    "VLLM_HPU_MOE_GATHER_VERIFY_LAYERS":
+    lambda: int(os.environ.get("VLLM_HPU_MOE_GATHER_VERIFY_LAYERS", "40")),
 }
 
 # end-env-vars-definition

3. Please read via envs instead of os.environ

diff --git a/vllm_gaudi/ops/hpu_fp8.py b/vllm_gaudi/ops/hpu_fp8.py
--- a/vllm_gaudi/ops/hpu_fp8.py
+++ b/vllm_gaudi/ops/hpu_fp8.py
 
@@ EXPERIMENTAL custom MoE combine header
-# Default stock. `HPU_MOE_GATHER_VERIFY=1` runs BOTH the custom path and the
-# Habana op on the same inputs and records FP8-ULP per layer.
-_HPU_MOE_GATHER = bool(os.environ.get("HPU_MOE_GATHER"))
-_HPU_MOE_GATHER_VERIFY = bool(os.environ.get("HPU_MOE_GATHER_VERIFY"))
-# Max tokens*topk (== gathered-expert count g) for which the custom gather path
-# is used. The gathered pure-PyTorch path wins below ~g=64 and LOSES to the stock
-# fused op once g approaches E (the dense gather + fp32 bmm path is slower than
-# the Habana op).
-_HPU_MOE_GATHER_MAX_TP = int(os.environ.get("HPU_MOE_GATHER_MAX_TP", "64"))
+# Default stock. `VLLM_HPU_MOE_GATHER_VERIFY=1` (with VLLM_HPU_MOE_GATHER=1)
+# runs BOTH the custom path and the Habana op on the same inputs and records
+# FP8-ULP per layer.
+import vllm_gaudi.envs as envs
+_HPU_MOE_GATHER = envs.VLLM_HPU_MOE_GATHER
+# Only meaningful together with the gather path.
+_HPU_MOE_GATHER_VERIFY = envs.VLLM_HPU_MOE_GATHER_VERIFY and _HPU_MOE_GATHER
+# Max tokens*topk (== gathered-expert count g) for which the custom gather path
+# is used. The gathered pure-PyTorch path wins below ~g=64 and LOSES to the stock
+# fused op once g approaches E (the dense gather + fp32 bmm path is slower than
+# the Habana op).
+_HPU_MOE_GATHER_MAX_TP = envs.VLLM_HPU_MOE_GATHER_MAX_TP
 if _HPU_MOE_GATHER:
     from vllm_gaudi.ops.hpu_moe_combine import gather_silu_fp8_moe  # noqa: E402
 else:
     gather_silu_fp8_moe = None
 
-_HPU_MOE_GATHER_VERIFY_DIR = os.environ.get("HPU_MOE_GATHER_VERIFY_DIR")
-_HPU_MOE_GATHER_VERIFY_LAYERS = int(os.environ.get("HPU_MOE_GATHER_VERIFY_LAYERS", "40"))
+_HPU_MOE_GATHER_VERIFY_DIR = envs.VLLM_HPU_MOE_GATHER_VERIFY_DIR
+_HPU_MOE_GATHER_VERIFY_LAYERS = envs.VLLM_HPU_MOE_GATHER_VERIFY_LAYERS

4. Please add description of new env vars to readme. i.e.

diff --git a/docs/configuration/env_variables.md b/docs/configuration/env_variables.md
--- a/docs/configuration/env_variables.md
+++ b/docs/configuration/env_variables.md
@@
 | `VLLM_MINIMAX_M3_MOE_GATHER_MAX_TOKENS` | Maximum token count for the MiniMax-M3 routed-expert gather path. Larger batches use the dense expert path. | `16` |
+
+## Experimental: Custom FP8 MoE Gather Combine
+
+These variables control an **experimental** pure-PyTorch gathered-expert MoE
+combine for silu + FP8-per-channel weights, an alternative to the Habana
+`mixture_of_experts` op. It is off by default and intended for low-token
+(small batch / decode) workloads. The `VERIFY` knobs run both the custom and
+stock paths and dump output pairs for offline FP8-ULP comparison — use them
+only for validation runs, not in production (they double compute and force a
+graph break).
+
+| Parameter name                      | Description                                                                                                                                                        | Default value |
+| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
+| `VLLM_HPU_MOE_GATHER`               | Enables the custom gathered-expert FP8 MoE combine (silu only). Falls back to the stock fused op when disabled or when the gather crossover is exceeded.          | `false`       |
+| `VLLM_HPU_MOE_GATHER_MAX_TP`        | Upper bound on `tokens * top_k` for which the custom gather path is used. Above this the stock fused op is faster and is used instead.                             | `64`          |
+| `VLLM_HPU_MOE_GATHER_VERIFY`        | Validation mode: runs both the custom and stock paths on the same inputs and records output pairs for offline FP8-ULP comparison. Requires `VLLM_HPU_MOE_GATHER`. | `false`       |
+| `VLLM_HPU_MOE_GATHER_VERIFY_DIR`    | Directory where verify-mode output pairs are saved (`moecomb_T<T>_n<n>_r<rank>.pt`). If unset, nothing is written even when verify mode is on.                     | `None`        |
+| `VLLM_HPU_MOE_GATHER_VERIFY_LAYERS` | Maximum number of captures saved per token count `T` (per rank) in verify mode.                                                                                   | `40`          |

5. Please correct order is ("RANK", "LOCAL_RANK"): try the globally-unique value first, fall back to LOCAL_RANK only for single-node launchers that set only LOCAL_RANK, then 0 if neither is present.

for _k in ("RANK", "LOCAL_RANK"):

6. Please make folder creation more safety

if not os.path.isdir(_HPU_MOE_GATHER_VERIFY_DIR):
    try:
        os.makedirs(_HPU_MOE_GATHER_VERIFY_DIR, exist_ok=True)
    except Exception as e:
        raise RuntimeError(
            f"Failed to create MoE-gather verify output directory: "
            f"{_HPU_MOE_GATHER_VERIFY_DIR}") from e

New review will be done after these changes

[iboiko-habana]

@NatTuck thanks for comments.
After extra security check - we can not allow any folder creation in vllm-gaudi during run execution. Also already implemented mechanism for data sharing between cards should be used. Please find more details below

PR can be changes in next way, it is onpy implementation proposal:
The original implementation accepted VLLM_HPU_MOE_GATHER_VERIFY_DIR, created
that user-selected directory with os.makedirs(), copied stock/custom outputs to
CPU, and wrote them with torch.save(). This creates an uncontrolled filesystem
output channel for inference-derived tensors.

The rank helper was also unnecessary and unreliable for this purpose. It used
ambient RANK / LOCAL_RANK values to construct filenames, rather than the actual
parallel-group topology; this can produce misleading rank attribution or naming
collisions across launch configurations.

what replaced the filesystem capture with in-memory verification:

• remove VERIFY_DIR and VERIFY_LAYERS flags;
• remove directory creation, CPU tensor transfer, and torch.save;
• calculate the local max absolute stock/custom error;
• all-reduce that scalar with MAX over get_ep_group().device_group.

EP is used because each rank owns a local expert shard. This validates every
expert-rank partial while exchanging only one scalar and creating no files.

diff --git a/docs/configuration/env_variables.md b/docs/configuration/env_variables.md
index 3326cbab..9d94bc87 100644
--- a/docs/configuration/env_variables.md
+++ b/docs/configuration/env_variables.md
@@ -33,6 +33,16 @@ This document lists the supported diagnostic and profiling, as well as performan
 | `VLLM_MINIMAX_M3_MOE_DECODE_GATHER` | Enables the MiniMax-M3 routed-expert gather path for low-token decode. Set to `0` or `false` to use the dense expert path. | `true` |
 | `VLLM_MINIMAX_M3_MOE_GATHER_MAX_TOKENS` | Maximum token count for the MiniMax-M3 routed-expert gather path. Larger batches use the dense expert path. | `16` |
 
+## Experimental: Custom FP8 MoE Gather Combine
+
+These variables control an experimental pure-PyTorch gathered-expert MoE combine for silu with FP8-per-channel weights. It is intended for low-token workloads. Verification runs both paths and reduces their maximum absolute error over the expert-parallel group without writing model-derived tensors to disk.
+
+| Parameter name | Description | Default value |
+| --- | --- | --- |
+| `VLLM_HPU_MOE_GATHER` | Enables the custom gathered-expert FP8 MoE combine for silu. Larger workloads use the stock fused op. | `false` |
+| `VLLM_HPU_MOE_GATHER_MAX_TP` | Maximum `tokens * top_k` for the gathered-expert path. | `64` |
+| `VLLM_HPU_MOE_GATHER_VERIFY` | Runs both paths and reduces their maximum absolute error across the EP group. Requires `VLLM_HPU_MOE_GATHER`. | `false` |
+
 Use `VLLM_BUCKETING_STRATEGY=exp` for the default exponential warm-up, `VLLM_BUCKETING_STRATEGY=lin` for explicitly configured linear ranges, or `VLLM_BUCKETING_STRATEGY=pad` for padding-aware ranges with absolute and relative padding limits.
 
 Leave `VLLM_EXPONENTIAL_BUCKETING` unset when using `VLLM_BUCKETING_STRATEGY`. The legacy flag is checked for backward compatibility and still overrides the selected strategy when present.
diff --git a/vllm_gaudi/envs.py b/vllm_gaudi/envs.py
index c1a1fddf..bdaf24d8 100644
--- a/vllm_gaudi/envs.py
+++ b/vllm_gaudi/envs.py
@@ -17,6 +17,9 @@ if TYPE_CHECKING:
     VLLM_MINIMAX_M3_MOE_DECODE_GATHER: bool = True
     VLLM_MINIMAX_M3_MOE_GATHER_MAX_TOKENS: int = 16
     VLLM_MM_WARMUP_OUTSIDE_COMPILE_ONLY: bool = False
+    VLLM_HPU_MOE_GATHER: bool = False
+    VLLM_HPU_MOE_GATHER_MAX_TP: int = 64
+    VLLM_HPU_MOE_GATHER_VERIFY: bool = False
 
 # The begin-* and end* here are used by the documentation generator
 # to extract the used env vars.
@@ -87,6 +90,13 @@ environment_variables: dict[str, Callable[[], Any]] = {
     # data-dependent output shapes that must be materialized during warmup.
     "VLLM_MM_WARMUP_OUTSIDE_COMPILE_ONLY":
     lambda: os.environ.get("VLLM_MM_WARMUP_OUTSIDE_COMPILE_ONLY", "false").strip().lower() in ("1", "true"),
+
+    "VLLM_HPU_MOE_GATHER":
+    lambda: os.environ.get("VLLM_HPU_MOE_GATHER", "0").lower() in ("1", "true"),
+    "VLLM_HPU_MOE_GATHER_MAX_TP":
+    lambda: int(os.environ.get("VLLM_HPU_MOE_GATHER_MAX_TP", "64")),
+    "VLLM_HPU_MOE_GATHER_VERIFY":
+    lambda: os.environ.get("VLLM_HPU_MOE_GATHER_VERIFY", "0").lower() in ("1", "true"),
 }
 
 # end-env-vars-definition
diff --git a/vllm_gaudi/ops/hpu_fp8.py b/vllm_gaudi/ops/hpu_fp8.py
index 7e7f0397..a5cf0eb9 100644
--- a/vllm_gaudi/ops/hpu_fp8.py
+++ b/vllm_gaudi/ops/hpu_fp8.py
@@ -2,6 +2,7 @@ from functools import partial
 from typing import Optional
 
 import torch
+from vllm.distributed import get_ep_group
 from vllm_gaudi import envs
 from torch.nn.parameter import Parameter
 from vllm.model_executor.layers.fused_moe.layer import FusedMoEFactory as FusedMoE
@@ -24,6 +25,29 @@ from vllm.model_executor.kernels.linear.scaled_mm.pytorch import (
 )
 
 
+_HPU_MOE_GATHER = envs.VLLM_HPU_MOE_GATHER
+_HPU_MOE_GATHER_VERIFY = envs.VLLM_HPU_MOE_GATHER_VERIFY and _HPU_MOE_GATHER
+_HPU_MOE_GATHER_MAX_TP = envs.VLLM_HPU_MOE_GATHER_MAX_TP
+
+if _HPU_MOE_GATHER:
+    from vllm_gaudi.ops.hpu_moe_combine import gather_silu_fp8_moe
+else:
+    gather_silu_fp8_moe = None
+
+
+def _verify_moe_combine(stock: torch.Tensor, custom: torch.Tensor) -> torch.Tensor:
+    max_error = (stock.float() - custom.float()).abs().amax()
+    if torch.distributed.is_available() and torch.distributed.is_initialized():
+        ep_group = get_ep_group()
+        if ep_group.world_size > 1:
+            torch.distributed.all_reduce(
+                max_error,
+                op=torch.distributed.ReduceOp.MAX,
+                group=ep_group.device_group,
+            )
+    return max_error
+
+
 class HPUPerTensorTorchFP8ScaledMMLinearKernel(PerTensorTorchFP8ScaledMMLinearKernel):
 
     @classmethod
@@ -268,13 +292,29 @@ class HPUFp8MoEMethod(Fp8MoEMethod):
         topk_ids = topk_ids.view(-1, topk_ids.shape[-1])
         topk_weights = topk_weights.view(-1, topk_weights.shape[-1])
 
-        output = layer.moe_op(
-            x,
-            topk_ids,
-            topk_weights,
-            permuted_weights=True,
-            activation=_normalize_moe_activation(layer.activation),
-        )
+        activation = _normalize_moe_activation(layer.activation)
+        use_gather = (_HPU_MOE_GATHER and activation == "silu"
+                      and x.shape[0] * topk_ids.shape[-1] <= _HPU_MOE_GATHER_MAX_TP)
+        if use_gather:
+            custom = gather_silu_fp8_moe(layer, x, topk_ids, topk_weights)
+            if _HPU_MOE_GATHER_VERIFY:
+                stock = layer.moe_op(
+                    x,
+                    topk_ids,
+                    topk_weights,
+                    permuted_weights=True,
+                    activation=activation,
+                )
+                _verify_moe_combine(stock, custom)
+            output = custom
+        else:
+            output = layer.moe_op(
+                x,
+                topk_ids,
+                topk_weights,
+                permuted_weights=True,
+                activation=activation,
+            )
         return output.view(*(output.size(0), *input_shape[1:]))
 
 
diff --git a/vllm_gaudi/ops/hpu_moe_combine.py b/vllm_gaudi/ops/hpu_moe_combine.py
new file mode 100644
index 00000000..94335219
--- /dev/null
+++ b/vllm_gaudi/ops/hpu_moe_combine.py
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: Apache-2.0
+
+import torch
+
+
+def _dynamic_quant(data: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+    from vllm_gaudi.extension.ops import dynamic_quant
+    return dynamic_quant(data)
+
+
+def gather_silu_fp8_moe(
+    layer,
+    x: torch.Tensor,
+    topk_ids: torch.Tensor,
+    topk_weights: torch.Tensor,
+) -> torch.Tensor:
+    """Compute the rank-local FP8 silu MoE partial for routed experts only."""
+    tokens, hidden_size = x.shape
+    num_topk = topk_ids.shape[-1]
+    w13 = layer.w13_weight
+    w2 = layer.w2_weight
+    scale13 = layer.w13_weight_scale_inv
+    scale2 = layer.w2_weight_scale_inv
+    local_experts = w13.shape[0]
+    intermediate_size = w13.shape[1] // 2
+
+    x_fp8, x_scale = _dynamic_quant(x)
+
+    experts_min = int(layer.moe_config.ep_rank * layer.local_num_experts)
+    local_ids = topk_ids - experts_min
+    in_range = (local_ids >= 0) & (local_ids < local_experts)
+    safe_ids = torch.where(in_range, local_ids, torch.zeros_like(local_ids))
+    safe_weights = torch.where(in_range, topk_weights, torch.zeros_like(topk_weights)).float()
+    gate_weights = x.new_zeros(tokens, local_experts, dtype=torch.float32)
+    gate_weights.scatter_add_(1, safe_ids, safe_weights)
+
+    gathered_experts = min(local_experts, tokens * num_topk)
+    hit_counts = (gate_weights != 0).float().sum(0)
+    gather_ids = torch.topk(hit_counts, gathered_experts, sorted=False).indices
+    gather_ids, _ = torch.sort(gather_ids)
+
+    w13_gathered = w13.index_select(0, gather_ids)
+    w2_gathered = w2.index_select(0, gather_ids)
+    scale13_gathered = scale13.index_select(0, gather_ids)
+    scale2_gathered = scale2.index_select(0, gather_ids)
+
+    w13_columns = w13_gathered.permute(2, 0, 1).reshape(hidden_size, -1)
+    scale13_columns = scale13_gathered.reshape(-1)
+    projected = torch.ops.hpu.fp8_gemm_v2(
+        A=x_fp8,
+        trans_A=False,
+        B=w13_columns,
+        trans_B=False,
+        D=None,
+        out_dtype=torch.float32,
+        A_scale_inv=x_scale,
+        B_scale_inv=scale13_columns,
+        B_scale_shape=[gathered_experts * 2 * intermediate_size],
+        bias=None,
+        accumulate=False,
+    )
+    projected = projected.reshape(tokens, gathered_experts, 2 * intermediate_size).permute(1, 0, 2)
+    gate, up = projected[..., :intermediate_size], projected[..., intermediate_size:]
+    activations = gate * torch.sigmoid(gate) * up
+
+    w2_float = w2_gathered.float() * scale2_gathered.unsqueeze(-1)
+    expert_outputs = torch.bmm(activations, w2_float.transpose(1, 2))
+    gathered_weights = gate_weights.index_select(1, gather_ids).t()
+    return (expert_outputs * gathered_weights.unsqueeze(-1)).sum(0).to(x.dtype)

[iboiko-habana]

@NatTuck please add 1 more critical fix, and CI will be run

Problem
The new gathered-expert path always dynamically quantizes the input:
x_fp8, x_scale = _dynamic_quant(x)

However, HPUFp8MoEMethod also supports FP8 models configured with activation_scheme == "static". For those models, the existing Habana MoE path must use checkpoint-provided scales:
w13_input_scale for the input/up-gate projection
w2_input_scale for each expert’s intermediate/down projection
The custom path ignores both. Enabling VLLM_HPU_MOE_GATHER=1 therefore changes the intended quantization behavior for static-FP8 models and can change their output.

Solution
Limit the experimental gathered-expert implementation to dynamic activation quantization, which is the mode it implements today. Static-FP8 models continue through the existing layer.moe_op implementation, where their configured scales are correctly applied.
This preserves existing static-FP8 correctness without blocking the intended low-token optimization for dynamic FP8 Qwen 3.5-family deployments.

diff --git a/vllm_gaudi/ops/hpu_fp8.py b/vllm_gaudi/ops/hpu_fp8.py
@@
         use_gather = (
             _HPU_MOE_GATHER
             and activation == "silu"
+            and self.quant_config.activation_scheme != "static"
             and x.shape[0] * topk_ids.shape[-1] <= _HPU_MOE_GATHER_MAX_TP
         )

[iboiko-habana]

CI run is failed on pre-commit issue: https://github.com/vllm-project/vllm-gaudi/actions/runs/33056998066/job/98466244312

For local pre-commit check please install and use next commands

pip install pre-commit
pre-commit install

[iboiko-habana]

@NatTuck please fix pre-commit issue
https://github.com/vllm-project/vllm-gaudi/actions/runs/33274841495/job/99159675259

[github-actions]

✅ CI Passed

All checks passed successfully against the following vllm commit:
39e276eaeb9daed06a180f6a8d187bbb8790e97b

[pawel-olejniczak]

gather_silu_fp8_moe reads layer.w13_weight_scale_inv / layer.w2_weight_scale_inv unconditionally, but under HPUFp8MoEMethod those parameters exist in only one configuration. Upstream names the parameter by layout (weight_scale_name = "weight_scale_inv" if self.block_quant else "weight_scale" in vLLM's layers/quantization/fp8.py), and the per-channel [E, 2I] / [E, H] form assumed here is created only at vllm_gaudi/extension/ops.py:1127-1128, i.e. by fp8_block_moe_prepare_weights with force_channel_fp8=True. fp8_channel_moe_prepare_weights guards with hasattr(layer, "w13_weight_scale_inv") at extension/ops.py:1146 for exactly that reason. Two reachable configurations break with VLLM_HPU_MOE_GATHER=1:

• non-block FP8 checkpoint (weight_block_size absent, activation_scheme: dynamic): the attribute never exists, so the first MoE layer raises AttributeError.
• block FP8 checkpoint with VLLM_HPU_FORCE_CHANNEL_FP8=0, or with QUANT_CONFIG set (which forces that flag false at vllm_gaudi/envs.py:37-39): the attribute exists but holds block scales shaped [E, 2I/block_n, H/block_k], and reshape(-1) feeds them to fp8_gemm_v2 while B_scale_shape declares [G*2I].

Suggest extending the gate the same way the static-FP8 case was handled: capture self._moe_gather_ok = self.block_quant and envs.VLLM_HPU_FORCE_CHANNEL_FP8 in process_weights_after_loading (next to has_moe_quant_config) and require it in use_gather. Reading w13_weight_scale as a fallback would not work, since on a non-block checkpoint that scale is per tensor ([E, 2]), not per channel.

[pawel-olejniczak]

_fp8_ulp casts both tensors to float8_e4m3fn before differencing bit patterns, so every element above the E4M3 max of 448 collapses onto the NaN code (0x7F / 0xFF) and compares equal. Running this exact function on CPU torch 2.6: stock 500 vs custom 50000 returns 0 ULP, and stock 448 vs custom 1e9 returns 1 ULP. Both pass the 2-ULP bar, so verify mode is blind precisely on the large-magnitude activation outliers where an FP8 up-projection is most likely to diverge. The docstring's "adjacent same-sign representable values differ by exactly +/-1 in uint8" also stops holding at 0x7F, which is not a finite value.

Suggest flagging out-of-range elements rather than folding them into the ULP count, for example warn when (stock.abs() > 448) | (custom.abs() > 448) holds anywhere, or compute a relative error for those elements only (their denominator is at least 448, so this cannot false-positive on small values).

[pawel-olejniczak]

The comment just above at hpu_fp8.py:41-44 states the crossover as "LOSES to the stock fused op once g approaches E", but this gate only bounds tokens * top_k by 64 and never compares it against local_num_experts. Since g = min(local_experts, tokens * K) (hpu_moe_combine.py:90), any deployment whose local expert count is at or below 64 sits inside the gate with g == local_experts: a Mixtral-style FP8 MoE with 8 experts at top_k 2 always takes the custom path with every expert gathered, and a 512-expert model at EP=8 reaches g == 64 == local_num_experts at 8 tokens with top_k 8. Those are the cases your own note says the stock op wins, and they also pay the full [G, H, I] fp32 w2 dequant at hpu_moe_combine.py:137.

Do you have the g-sweep behind VLLM_HPU_MOE_GATHER_MAX_TP=64, and was it measured at a single local_num_experts? If the real crossover is g versus E_local, adding and x.shape[0] * topk_ids.shape[-1] < layer.local_num_experts to the gate would express it directly.

[NatTuck]

Honestly, that was from an early test just with Qwen 3.6 35B on one HPU at concurrency 1.

I just did a longer test sweep over all three sizes (it looks like this just hits Qwen 3.5, so 35B, 122B, and 397B) and it looks like fraction of experts is, in fact, more related than the current gate. So I've swapped that in instead of the MAX_TP thing. Doesn't seem like it lines up perfectly, and the threshold isn't entirely clear, so it'll still be a tunable.

Now I've just got to wait for this one verification run to finish. Feature request: make vllm-gaudi load faster

[pawel-olejniczak]

Two small things, grouped:

• This early return is dead code: the only call site (hpu_fp8.py:346-349) is already inside if _HPU_MOE_GATHER_VERIFY. It also makes the function return either the stock tensor or a scalar ULP depending on a flag, and the call site discards the result. Dropping the guard and annotating -> None would be clearer.
• Verify mode reads a device tensor on the host at hpu_fp8.py:86 (max_ulp > ...) once per MoE layer per forward, on top of running both paths. The docs table only mentions the extra compute, so worth stating the per-layer host sync there too so nobody enables it on a perf run.

[pawel-olejniczak]

Thanks - the mask does close the blind spot above 448, but folding it into the same scalar opens a second one. _fp8_ulp now returns inf at out-of-range positions and _verify_moe_combine reduces with amax, so a single out-of-range element anywhere makes max_ulp infinite, this branch fires, and the elif that reports the actual ULP number never runs. Measured with this exact code on CPU torch 2.6:

• stock = custom = [600.0, 1.0], bit-identical: warns out-of-range element(s) detected.
• stock = [600.0, 1.0] vs custom = [600.0, 1.5]: same out-of-range warning, and the real in-range mismatch (4 ULP on that second element) is never printed.

So for any model whose MoE output holds one element above 448, verify mode emits the same message on every layer whether the two paths agree bit for bit or diverge badly inside the finite range. The docs row still promises "warns if any element exceeds 2 ULP", which is the part that stops being reachable.

Suggest keeping the two signals separate: have _fp8_ulp return finite ULPs plus the mask, take amax over in-range positions only, and reduce a 2-element tensor [max_in_range_ulp, any_out_of_range] in the one existing all_reduce. Both warnings can then fire independently and the numeric bar keeps working. If you would rather not hear about out-of-range elements when both paths agree on them, gate that warning on a relative error over those positions instead of on their presence.

[pawel-olejniczak]

The pinned yapf 0.43.0 hook wants this comparison joined onto one line - it fits inside the 120-column limit:

-                      and x.shape[0] * topk_ids.shape[-1]
-                      <= layer.local_num_experts * _HPU_MOE_GATHER_RATIO)
+                      and x.shape[0] * topk_ids.shape[-1] <= layer.local_num_experts * _HPU_MOE_GATHER_RATIO)

python -m yapf -d -r vllm_gaudi/ at this SHA flags this file as the only one in the tree needing a reformat, so pre-commit run --all-files --hook-stage manual will fail on it. ruff 0.11.7 and pymarkdown 0.9.29 are clean on the changed files.