Skip to content

fix: reject swiglu_limit=0.0 - #15

Open
aryagxr wants to merge 1 commit into
cursor:mainfrom
aryagxr:fix/nonzero-swiglu-limit
Open

fix: reject swiglu_limit=0.0#15
aryagxr wants to merge 1 commit into
cursor:mainfrom
aryagxr:fix/nonzero-swiglu-limit

Conversation

@aryagxr

@aryagxr aryagxr commented Aug 26, 2026

Copy link
Copy Markdown

Summary

Addresses bugbot on #11: zero swiglu_limit enables clamping and zeros expert activations

Problem

swiglu_limit=0.0 was allowed by the python validation but the CUDA path treats any provided limit as "enable clamping". With a limit of 0, the up projection gets clamped to zero, and as a result expert activations are wiped instead of running normal SwiGLU

Changes

  • Require swiglu_limit to be None or strictly positive in all six fused wrappers in mok/ops.py. Reject when swiglu_limit <= 0
  • Add a swiglu_limit=0.0 rejection case to each matching failure loop intests/test_ops.py

Validation

  • python3 -m py_compile mok/ops.py tests/test_ops.py — passed
  • git diff --check — passed
  • Confirmed 6 swiglu_limit <= 0 checks in mok/ops.py and 6 matching rejection cases in tests/test_ops.py
  • Full distributed CUDA operator tests were not run; no SM100/SM103 system/compiled extension runtime was available

Note

Low Risk
Input validation-only change with no CUDA or numerical path edits; callers passing zero will now fail fast instead of producing wrong activations.

Overview
Tightens swiglu_limit validation on all six fused MoE entry points in mok/ops.py (MXFP8/BF16 forward, recompute, and backward). None is still allowed; any numeric value must now be strictly positive (<= 0 raises ValueError), replacing the previous rule that allowed zero.

This blocks a silent correctness bug: the CUDA path treats a set limit as “enable clamping,” so swiglu_limit=0.0 clamps the up projection to zero and wipes routed expert activations instead of running normal SwiGLU.

Tests add a swiglu_limit: 0.0 failure case to each operator’s existing validation loop in tests/test_ops.py.

Reviewed by Cursor Bugbot for commit a8fd5bc. Bugbot is set up for automated code reviews on this repo. Configure here.

Signed-off-by: Arya Gopikrishnan <aryagxr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant