Commit f53bbae
committed
Optimize expert selection
Simplify the case when norm_topk_prob is set. Avoid extra normalization and smaller softmax computation.
Results (Measured on M3 max):
Before:
📊 Benchmark Summary:
==================================================
Prompt: 1066.413 tokens/sec
Generation: 62.131 tokens/sec
==================================================
After:
📊 Benchmark Summary:
==================================================
Prompt: 1103.736 tokens/sec
Generation: 69.197 tokens/sec
==================================================
Testing: presubmit1 parent 34f0db3 commit f53bbae
1 file changed
Lines changed: 11 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
125 | 125 | | |
126 | 126 | | |
127 | 127 | | |
128 | | - | |
129 | | - | |
| 128 | + | |
130 | 129 | | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | 130 | | |
136 | | - | |
137 | | - | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
138 | 141 | | |
139 | 142 | | |
140 | 143 | | |
| |||
0 commit comments