perf(k3): decode gates, GEMV route QA, and FlashInfer 0.6.18 re-tune - #1318
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 287b0d25a6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
287b0d2 to
e4107b0
Compare
e4107b0 to
5a8faea
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a8faeafc8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
2f5f90e to
250d162
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 250d1621c5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
f1917bb to
6736397
Compare
- KDA gate precompute on tensor cores: the per-layer gate is a plain f_a @ f_b.T, but shipped as a CUDA-core reduction. The tl.dot form takes the batched kernel 1366 -> 197us/step at bs=64; numerics stay within the spread the code's own legal tilings already exhibit. The reduction kernel remains for tiles under tl.dot's 16-row minimum. - Decode buckets skip the multimem all-reduce window: the is_decode axis the tail-tier comment asked for. Prefill keeps multimem (the ld_reduce win is real there); decode moves to FUSED_LANE_AR, worth -1.2 to -1.7% step time at bs>=32 on a 2-node TP8 GB300 pair, and the mechanism is idle removal. Same num_tokens arrives from both phases, so no token bound can separate them. - MEASURED_ROUTE: re-measure the 92 proposed drafter/M-gap entries three runs each against an unrounded >=1.08 bar. 57 confirmed, 2 flip backend (both arms beat the incumbent; the inter-backend edge is ~1%), 35 dropped as rounded-margin or run-flip noise, and one clean passer the original accounting missed is added. Add 17 DSpark-TP8 and 3 eagle3-TP8 entries from shapes observed at the kernel launch point (not derived from config). Table lands at 122 entries; a source-level duplicate-key assertion joins the tests, since a dict literal resolves duplicates silently last-wins. A dense 17..32 sweep then extends three TP16 widths where tgv keeps winning past one tile -- 768x1536 and 1152x1536 across the whole range at 1.26-1.41x, 2304x7168 where it clears the bar -- for 39 more entries (161 total); the sweep grid stays hole-free so exact-M keying never strands a batch between swept points. - Bound the fused qkv_a min-latency kernel at M <= 16: it wins to one tile and loses a flat ~1.35x from M=18 through 64 (measured on GB300; the parts this gate serves share that tile geometry). - Re-sweep the GB300 TP8/EP1 MoE tactic table for FlashInfer 0.6.18 (17 of 21 tactics changed). Note the packaged table is shadowed by the startup autotune window's in-memory cache; it only matters for deployments that skip that window. - tune_route.py: extend MS through the observed drafter range and add the TP8 drafter shapes. Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rYMfq4CsFuHfXcQWMJ3Cc
c4eacfc to
c487431
Compare
2 participants
Summary
Test Plan