feat: rebase qvac patches onto 2026-06-05 (upstream leejet/ggml@0ce7ad34)#21
Open
aegioscy wants to merge 15 commits into
Open
feat: rebase qvac patches onto 2026-06-05 (upstream leejet/ggml@0ce7ad34)#21aegioscy wants to merge 15 commits into
aegioscy wants to merge 15 commits into
Conversation
Allow ggml to export a static CPU backend while loading GPU backends dynamically, and align backend naming/config metadata with qvac packaging. Harden backend discovery for Android/OpenCL/Vulkan so packaged builds fail more gracefully.
Adds Metal backend support for ops required by Wan video diffusion models on Apple Silicon. Without this patch the Metal backend crashes with: ggml_metal_op_encode_impl: error: unsupported op 'IM2COL_3D'. Changes ------- - IM2COL_3D Metal kernel implementation (f32 and f16 variants) - Pipeline getter and op handler for IM2COL_3D dispatch - PAD kernel updated to support non-zero left padding (lp0..lp3) - ggml_metal_kargs_pad extended with left-padding fields - supports_op for PAD relaxed to allow any padding values DIAG_MASK_INF intentionally omitted ----------------------------------- The original CLDawes patch (ggml-org/llama.cpp#16669) also added a Metal DIAG_MASK_INF kernel, but it is not included here because: - GGML maintainers are deprecating GGML_OP_DIAG_MASK_INF in favor of generic ops (see ggml-org/llama.cpp#17654) - leejet/stable-diffusion.cpp#1230 (merged Jan 2026) refactored the attention path to build explicit -INFINITY mask tensors and no longer calls ggml_diag_mask_inf in app code - Wan attention sites pass mask=nullptr and never request DIAG_MASK_INF, so the kernel would be dead code on this stack Adapted from CLDawes' patch (https://github.com/CLDawes/ggml/tree/patch-qwen-image), updated to use the current ggml_metal_pipeline_with_params API. Verified on Apple M3 Ultra: Wan2.1 1.3B txt2video runs end-to-end on Metal with the host process at 0-1.5% CPU throughout. Made-with: Cursor
Per code review on #5 (jesusmb1995, gianni-cor), the Metal supports_op predicate for IM2COL_3D only constrained the dst type. The Metal kernel reads src[1] as `device const float *` and the CPU reference asserts `src1->type == GGML_TYPE_F32` (ops.cpp:6413), so the F16-input case was advertised as SUPPORTED and then aborted at runtime under test-backend-ops: ./bin/test-backend-ops support -b Metal -o IM2COL_3D \ -p "type_input=f16,type_kernel=f32,dst_type=f32" -> SUPPORTED (incorrect) ./bin/test-backend-ops test -b Metal -o IM2COL_3D \ -p "type_input=f16,type_kernel=f32,dst_type=f32" -> src/ggml-cpu/ops.cpp:6413: GGML_ASSERT(src1->type == GGML_TYPE_F32) This matches the existing predicate in the Vulkan backend (src/ggml-vulkan/ggml-vulkan.cpp:14913). Also adds the corresponding negative regression case to test-backend-ops so this specific pattern stays rejected: test_im2col_3d(GGML_TYPE_F16, GGML_TYPE_F32, GGML_TYPE_F32) Verified on Apple M3 Ultra: 2051/2051 IM2COL_3D Metal tests pass and the F16-input case is now reported as `not supported [Metal]`. Made-with: Cursor
…dst kernel match Per code review on #6 (gianni-cor), the prior predicate (which only required src[1]->type == F32 + dst in {F32,F16}) still left two advertise/abort gaps against the CPU reference: 1. F16-dst path (ggml_compute_forward_im2col_3d_f16, ops.cpp:6322) also asserts src0->type == F16. Metal's kernel does GGML_UNUSED(src0) so it executes any combo on the GPU, but the CPU reference aborts: ./bin/test-backend-ops support -b Metal -o IM2COL_3D \ -p "type_input=f32,type_kernel=f32,dst_type=f16" -> SUPPORTED (incorrect) ./bin/test-backend-ops test -b Metal -o IM2COL_3D \ -p "type_input=f32,type_kernel=f32,dst_type=f16" -> src/ggml-cpu/ops.cpp:6322: GGML_ASSERT(src0->type == GGML_TYPE_F16) failed 2. Both CPU IM2COL_3D paths assert nb10 == sizeof(float) (ops.cpp:6453). The Metal predicate didn't validate this; a non-stride-1 dim 0 on src[1] (e.g. a permuted view) passed Metal's supports_op and crashed in the CPU reference. The 2D GGML_OP_IM2COL arm one block above already enforces ggml_is_contiguous(op->src[1]). Tighten the Metal predicate to cover both: case GGML_OP_IM2COL_3D: return op->src[1]->type == GGML_TYPE_F32 && op->src[1]->nb[0] == sizeof(float) && (op->type == GGML_TYPE_F32 || (op->type == GGML_TYPE_F16 && op->src[0]->type == GGML_TYPE_F16)); This now matches all three CPU-reference invariants (src[1] type, src[1] contiguity in dim 0, and the F16-path src[0] requirement). Existing test-backend-ops cases use ggml_view_4d() with v=true, which preserves nb[0] = sizeof(float), so the new contiguity check does not regress existing positive cases. Also adds the matching negative regression case alongside the existing F16-input one, so both stay rejected: test_im2col_3d(GGML_TYPE_F16, GGML_TYPE_F32, GGML_TYPE_F32) // already in test_im2col_3d(GGML_TYPE_F32, GGML_TYPE_F32, GGML_TYPE_F16) // new Verified on Apple M3 Ultra: ./bin/test-backend-ops support -b Metal -o IM2COL_3D type_input=f32,type_kernel=f32,dst_type=f32: SUPPORTED type_input=f32,type_kernel=f16,dst_type=f32: SUPPORTED type_input=f32,type_kernel=f16,dst_type=f16: SUPPORTED type_input=f16,type_kernel=f32,dst_type=f32: NOT SUPPORTED type_input=f32,type_kernel=f32,dst_type=f16: NOT SUPPORTED (new) ./bin/test-backend-ops test -b Metal -o IM2COL_3D 2051/2051 tests passed; both negative cases logged as "not supported [Metal]" and skipped (no GGML_ASSERT abort). Made-with: Cursor
1. Fused RoPE (GGML_OP_ROPE_FLUX)
- New op: ggml_rope_flux(a, b) in ggml.h / ggml.c
- Metal kernel: kernel_rope_flux — applies interleaved rotary embedding
and permutes output layout in a single dispatch
- Metal kernel: kernel_permute_cont_021 — fused permute(0,2,1,3)+cont
for V tensor preparation (called via ggml_rope_flux(v, NULL))
- Dispatch: ggml-metal-ops.cpp selects kernel based on PE presence
- Supports_op: ggml-metal-device.m for F32 inputs
- Test: test-rope-flux.cpp — bit-exact correctness across 4 configs
2. Implicit GEMM conv2d (kernel_conv_2d)
- Rewrote naive per-pixel conv2d as implicit GEMM with simdgroup MMA
- 64x64 output tiles, half-precision loads, float accumulators
- 1x1 conv fast path, incremental k-decomposition
- Dispatch: ggml-metal-ops.cpp updated for 256 threads, threadgroup mem
- Pipeline: ggml-metal-device.cpp updated smem and bc_out condition
- Test: test-conv2d-direct.cpp — 14 configs, exact match vs im2col
3. Flash attention NQPTG>8 fix
- Added query block loops to QK and O accumulation sections
- Enables correct output for NQPTG>8 (though NQPTG=8 remains optimal)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Tighten fused RoPE validation and fallback behavior while covering the conv2d 1x1 edge case that could read out of bounds. Co-authored-by: Cursor <cursoragent@cursor.com>
Require simdgroup matrix multiply before advertising Metal conv2d support and add ROPE_FLUX to backend-op coverage. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep Metal ROPE_FLUX support checks aligned with the int32 dispatch limit so oversized tensors fall back instead of asserting at runtime. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep backend correctness tests on executable IM2COL_3D type combinations and make CPU/Vulkan supports_op reject combinations that would otherwise assert during compute. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds ggml_graph_leaf(), ggml_graph_leafs(), ggml_graph_n_leafs() to match the existing node equivalents. Needed by downstream consumers (e.g. stable-diffusion.cpp's ggml_graph_cut.cpp) that cannot include the private ggml-impl.h when building with SD_USE_SYSTEM_GGML. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
2026-01-30onto the new2026-06-05base (leejet/ggml@0ce7ad34— the exact ggml commit used byleejet/stable-diffusion.cpp@2026-06-04)ggml_graph_leaf(),ggml_graph_leafs(),ggml_graph_n_leafs(), andggml_graph_add_leaf()to the public API — required bystable-diffusion.cpp'sggml_graph_cut.cppwhen building withSD_USE_SYSTEM_GGML=ON(no submodule)Commits included
cmake: support qvac hybrid backend packagingmetal: add IM2COL_3D op and PAD left-padding support for Wan videometal: tighten / extend IM2COL_3D supports_opmetal: fused RoPE kernel, implicit GEMM conv2d, flash attention fixmetal: move RoPE flux dispatch to helpermetal: harden / gate / reject RoPE flux and conv2dmetal: sync / remove unused conv2d tile argscpu: optimize Flux RoPE row traversaltest: cover Flux RoPE strides and F32 conv2dbackend: reject unsupported IM2COL_3D casesapi: export ggml_graph_leaf/leafs/n_leafs/add_leaf public APIMade with Cursor