CUDA: fix cpy transposed path corrupting non-contiguous dst#1570
Open
devYRPauli wants to merge 1 commit into
Open
CUDA: fix cpy transposed path corrupting non-contiguous dst#1570devYRPauli wants to merge 1 commit into
devYRPauli wants to merge 1 commit into
Conversation
ggml_cuda_cpy's can_be_transposed gate only inspected src0's layout, so a same-type F32/F16/BF16/I32 copy into a non-contiguous destination (e.g. a strided view_2d slot such as a KV-cache column write) still selected the transposed scalar kernel. That kernel writes dst with a contiguous index derived from ne00/ne01 and ignores the destination strides nb10..nb13 (marked GGML_UNUSED_VARS), silently corrupting the strided destination. Require ggml_is_contiguous(src1) for the transposed fast path. Strided destinations now fall back to the general cpy_scalar kernel, which honors nb10..nb13. Fixes ggml-org#1497. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ggml_cuda_cpy'scan_be_transposedgate only inspectssrc0's layout. When the destination (src1) is non-contiguous - for example a stridedview_2dslot such as a KV-cache column write - a same-type F32/F16/BF16/I32 copy still takes the transposed scalar kernel.That kernel (
cpy_scalar_transpose) writes the destination with a contiguous index derived fromne00/ne01and ignores the destination stridesnb10..nb13(they are handed toGGML_UNUSED_VARS). The result is silent corruption of the strided destination.Fix
Require
ggml_is_contiguous(src1)for the transposed fast path. Non-contiguous destinations now fall back to the generalcpy_scalarkernel, which honorsnb10..nb13. Contiguous destinations (the common case) are unaffected, so there is no change on the normal path.Fixes #1497, which has the root-cause analysis and a reproduction (16 ops plus a real transformer decode). Thanks to @OriPekelman for pinpointing the exact gate and proposing this fix.