Add vl support - #7
Merged
Merged
Conversation
…ss optimizer steps
transformers 5.x save_pretrained keeps only the new rope_parameters schema and drops legacy rope_scaling. Serving stacks and older transformers that read only the legacy key then silently lose YaRN on long-context checkpoints (the draft falls back to unscaled RoPE at serve time — accept length collapses beyond the original context). Mirror whichever schema survives into the other at save time; default (non-scaled) RoPE configs are left untouched.
docs: add npu tutorial
feat: --trim-loss-positions (compute teacher/logits/loss only at supe…
…-support feat: add Kimi-K3 DSpark template, draft config, and example run
Domino previously materialized corrected logits before computing its final and base cross-entropy losses. At production vocabulary sizes, that creates large corrected-logit and softmax intermediates and requires additional vocabulary passes. Add a fused Triton implementation that consumes base logits and compact suffix corrections directly. The forward kernel computes both weighted loss sums and predictions in one vocabulary scan. It saves per-row maximum and shifted exponential-sum statistics so backward can reconstruct softmax tiles and write base and correction gradients without materializing corrected logits or probability tensors. Integrate the loss into upstream's checkpointed block reduction and normalize once after summing all chunk numerators. Use the fused path by default for CUDA tensors; SPECFORGE_DOMINO_TRITON_CE=0 retains the PyTorch reference path. On an NVIDIA GB300 with Torch 2.11.0, CUDA 13.0, BF16, block size 16, and vocabulary size 248320, a forward-plus-backward microbenchmark measured 11.17 ms eager versus 1.82 ms fused at 2048 rows (6.1x), and 21.91 ms versus 3.27 ms at 4096 rows (6.7x). At 4096 rows, incremental peak allocation fell from 7.58 GiB to 3.67 GiB. These are loss-kernel measurements, not end-to-end training results. Keep a readable PyTorch reference implementation and add focused comparisons of losses, predictions, and gradients across correction layouts and vocabulary tiles, plus checkpointed model and offline FSDP integration coverage.
[Feature] Refactor configs dir
…ton-ce Fuse Domino cross entropy with Triton
# Conflicts: # tests/test_config/test_launch_topology.py # tests/test_config/test_unified_feature_reachability.py
Turn on model.use_liger_kernel for the two AMD Qwen3.5-4B DFlash example
configs (offline + online). This routes the draft model's RMSNorm and
SwiGLU MLP through Liger's fused Triton kernels.
Validated on a single idle MI355X (gfx950), torch 2.9.1+rocm7.2.0, real
`specforge train` (offline, batch_size=2, S=2048, 200 steps, steady-state
mean over 19 post-warmup steps):
use_liger_kernel=false -> 12.68 samples/s, 0.623 s/step, 35.37 GiB peak
use_liger_kernel=true -> 13.97 samples/s, 0.565 s/step, 30.18 GiB peak
=> +10.2% throughput, -5.2 GiB (-15%) peak memory, loss parity to 4
decimals (step200 7.0105 vs 7.0101).
Depends on the flex-backend fix (gate BACKEND=TRITON on torch>=2.11): on
torch<2.11 the Liger path drives flex-attention through the inductor
autotune lowering, which crashes with NameError: 'TRITON' is not defined
without that fix.
[CI] Del hardcode config tests
feat: add mtp support
…er-kernels perf(rocm): enable Liger fused RMSNorm/SwiGLU for AMD DFlash configs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
Related Issues
Accuracy Test
Benchmark & Profiling
Checklist