Skip to content

Restore the TPC causal_conv1d update path, switchable by env var - #1714

Open
rsenaikh wants to merge 1 commit into
vllm-project:mainfrom
rsenaikh:restore-tpc-causal-conv1d-update
Open

Restore the TPC causal_conv1d update path, switchable by env var#1714
rsenaikh wants to merge 1 commit into
vllm-project:mainfrom
rsenaikh:restore-tpc-causal-conv1d-update

Conversation

@rsenaikh

@rsenaikh rsenaikh commented Aug 8, 2026

Copy link
Copy Markdown

Summary

Restores the TPC implementation of the hybrid-model causal_conv1d decode update in
hpu_causal_conv1d_update, and adds VLLM_HPU_CONV1D_DISABLE_TPC so that the TPC kernel and the
PyTorch reference can be selected from a single build. The TPC kernel is the default; the PyTorch
reference has to be asked for explicitly.

The path was removed in #1585 (cherry-picked to main in #1600) because a gsm8k accuracy
difference was observed against the PyTorch reference on a large hybrid MoE model.

Why restore it

The difference does not reproduce under controlled conditions.

The benchmark was run repeatedly from one build, on one machine, against one copy of the weights,
with the conv1d implementation as the only variable, six repeats per implementation. What that
shows:

  • The two implementations end up closer together than the benchmark's own run-to-run spread.
    Re-running the identical build already moves more questions than the difference under
    discussion.
  • That spread is larger on the PyTorch reference than on the TPC kernel, so the instability
    does not originate in the kernel.
  • No significance test separates the two arms — neither a run-level t-test, nor an exact
    permutation test over every possible relabelling of the runs, nor McNemar on the pooled
    per-question outcomes.
  • Per question, about as many outcomes improve under the TPC kernel as under the reference. A
    defect would concentrate the losses on one side; a reshuffle of borderline questions is what a
    last-bit numerical difference looks like.
  • Nearly all of the remaining difference between the arms comes from a small set of questions
    that each implementation answers reproducibly but oppositely. Which answer is correct there is
    arbitrary — the perturbation carries no information about it — and the split of that set alone
    varies by as much as the whole measured difference.

This was measured twice, in two independent configurations that differ in PyTorch version and in
kernel library build, with the same outcome both times.

Separately, an element-wise comparison of the two implementations on identical inputs — across
batch sizes, kernel widths, activations, both bias settings and repeated decode steps — agrees to
one or two bf16 ulp and writes bit-identical conv state, with no accumulation across steps.

The environment variable

Value Implementation
unset, 0, false TPC kernel (default)
1, true PyTorch reference

The variable is read per call rather than cached at import, so it applies in every worker process
regardless of how workers are spawned, and it can be flipped between runs of one build. This is
what makes the two implementations comparable without rebuilding or swapping commits — the reason
the original report was hard to act on.

Test plan

  • New tests/unit_tests/ops/test_causal_conv1d_update_dispatch.py asserts which implementation
    each setting of the variable selects, including the default and the case where the TPC op is
    absent from the build, and that the fallback to the PyTorch reference still produces correct
    results.
  • Run on Gaudi hardware: pytest tests/unit_tests/ops/test_causal_conv1d_update_dispatch.py
    all cases pass.
  • The accuracy experiment above was run end-to-end with this exact file installed.

The TPC implementation of the hybrid-model causal_conv1d decode update was
removed after a gsm8k accuracy difference was observed against the PyTorch
reference on a large MoE model.

A controlled repeat does not reproduce it. Running the benchmark repeatedly from
a single build, with the conv1d implementation as the only variable, leaves the
two implementations closer together than the benchmark's own run-to-run spread --
and that spread is larger on the PyTorch reference than on the TPC kernel, so it
does not originate in the kernel. No test separates the two arms. This was
measured twice in two independent configurations -- differing in PyTorch version
and kernel library build -- with the same outcome.

Of the questions whose outcome ever changes, about as many favour the TPC kernel
as favour the reference, which is what a last-bit numerical difference looks like
rather than a defect. Almost all of the difference between the two arms comes
from a small set of questions that each implementation answers reproducibly but
oppositely; which one is right there is arbitrary, and the split alone varies by
as much as the whole measured difference. An element-wise comparison of the two
implementations on identical inputs agrees to one or two bf16 ulp and writes
bit-identical state.

This restores the removed branch and adds VLLM_HPU_CONV1D_DISABLE_TPC so the two
implementations can be compared from one build without rebuilding or swapping
commits. The TPC kernel stays the default: unset, 0 or false selects it, 1 or
true selects the PyTorch reference. The variable is read per call rather than
cached at import so it reaches every worker process however they are spawned.

Signed-off-by: Raed Al-Senaikh <raedx.senayh@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant