[Feat] Support platform-graph capture and replay for KDA chunk - #1212
Open
zhiyuan1i wants to merge 4 commits into
Open
[Feat] Support platform-graph capture and replay for KDA chunk#1212zhiyuan1i wants to merge 4 commits into
zhiyuan1i wants to merge 4 commits into
Conversation
…h support prepare_chunk_indices_static builds chunk_indices/chunk_offsets with on-device ops at a fixed shape so they can be recorded in a platform graph (CUDA graph) and recomputed from the live cu_seqlens on every replay. get_static_buffer hands out persistent intermediates whose addresses stay valid for the graph's whole lifetime. chunk_kda gains use_graph + max_num_seqs: the varlen chunk list is built on-device at a static shape, kernels early-return on sentinel rows, and backward intermediates come from persistent buffers. The eager path (use_graph=False) is unchanged.
- Pass use_graph to chunk_kda_fwd_intra_token_parallel: with a static token grid, tokens beyond cu_seqlens[-1] converged the in-kernel binary search to i_n=N and read cu_seqlens out of bounds; the kernel now returns early for them. - Tighten the max_num_seqs contract to an exact padded length and document the graph contract: undefined output/gradient rows outside real tokens, [max_num_seqs, ...] initial_state/dht, and eager warmup before capture. - Unify the sentinel guard to a single-line check and export prepare_chunk_indices_static from fla.ops.utils.
- FLACPContext: pre/post_num_ranks_dev device scalars, NUM_RANKS_ON_DEVICE, h_seq_idx for merge kernel, pre_process wrapper always launches - chunk_local_cumsum outputs zeros_like under use_graph so dA/dbias are not polluted by dirty padding rows - pass use_graph/chunk_offsets through NPU wrappers; NPU rejects use_graph=True explicitly, chunk_offsets no longer recomputed when provided, zeros_like parity
zhiyuan1i
marked this pull request as ready for review
September 2, 2026 09:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
use_graph+max_num_seqstochunk_kdaso the forward and backward can be captured in a platform graph (CUDA graph). The varlen chunk list is built on-device at a static shape viaprepare_chunk_indices_static, kernels early-return on sentinel rows, and backward intermediates come from persistent buffers viaget_static_buffer. The eager path (use_graph=False) is unchanged.Also supports
cp_context(context parallelism) in graph capture/replay: FLACPContext gains device scalars for pre/post num_ranks, the merge kernel handles NUM_RANKS_ON_DEVICE, and the pre_process wrapper always launches.NPU backends explicitly reject
use_graph=TruewithNotImplementedError.Changes
fla/ops/utils/graph.py(new, 27 lines):get_static_bufferhands out persistent intermediates whose addresses stay valid for the graph's lifetime.fla/ops/utils/index.py:prepare_chunk_indices_staticbuilds chunk_indices/chunk_offsets with on-device ops at a fixed shape for graph recording.fla/ops/kda/chunk.py:use_graph+max_num_seqsentry point; varlen chunk list built on-device; eager path unchanged.fla/ops/kda/chunk_fwd.py/chunk_bwd.py: kernels early-return on sentinel rows; backward intermediates from persistent buffers.fla/ops/kda/chunk_intra.py/chunk_intra_token_parallel.py:use_graphguard for tokens beyondcu_seqlens[-1].fla/ops/common/chunk_delta_h.py:use_graphsupport in the shared delta-rule state recurrence.fla/ops/cp/chunk_delta_h.py/context.py: CP graph capture/replay support.fla/ops/utils/cumsum.py:zeros_likeunderuse_graphso dA/dbias are not polluted by dirty padding rows.NotImplementedErroronuse_graph=True;chunk_offsetsno longer recomputed when provided;zeros_likeparity.Tests
tests/ops/test_kda_graph.py(386 lines): 16 tests covering fwd/bwd graph capture, varlen, state, gate, partial tokens.tests/context_parallel/test_cp_kda_graph.py(179 lines): CP graph capture/replay with world=2, 64/64 checks pass.Related
Checklist