Granite-Switch Architecture by barvhaim · Pull Request #25107 · ggml-org/llama.cpp

barvhaim · 2026-06-28T16:02:44Z

Overview

Adaptation of Granite-Switch model (https://huggingface.co/ibm-granite/granite-switch-4.1-3b-preview) for llama.cpp,
Today Granite-Switch supports HF and vLLM backends.
Granite-Switch OSS project: https://github.com/generative-computing/granite-switch
Converted model with this code: https://huggingface.co/barha/granite-switch-4.1-3b-preview-GGUF/blob/main/granite-switch-4.1-3b-preview-f16.gguf

The model is dense, not MoE. This value is only needed because ggml_mul_mat_id uses n_expert_used as the number of selected matrix IDs per token. For Granite Switch this is always one selected adapter slot per token.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, parity with existing backends + documentations, assisted by Claude Code harness

P.S, maintainers, I'm looking forward for your comments on how to progress with getting this supported arch, so pretty much opened that PR for getting feedbacks. thanks!

ggml-gh-bot · 2026-06-28T16:07:01Z

Hi @barvhaim, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

barvhaim · 2026-06-28T17:16:14Z

Hi @barvhaim, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

I already added before "AI usage disclosure"

gabe-l-hart

A bunch of thoughts that loosely fall into the following buckets:

Should we explore implementing the adapters themselves using the existing adapter mechanism so it's more extensible in the future?
Some restructuring in the conversion code to hopefully simplify a little and extend flexibility

@CISC I'm particularly curious your thoughts on reusing existing adapter mechanisms vs this implementation that reimplements adapters in just the right places for this model so that it doesn't need to change anything about the existing adapter code paths.

gabe-l-hart · 2026-06-29T15:44:05Z


+@ModelBase.register("GraniteSwitchForCausalLM")
+class GraniteSwitchModel(GraniteMoeModel):
+    """Dense, all-attention Granite model with N embedded LoRA adapters selected


I know the current model is built on the Dense 4.1 bases. I believe the intent is to also release switch models for other architectures going forward (including MoE). I haven't made it through the review yet, but I think we'll want to try to make sure the switch functionality is loosely coupled to the core model so it can easily extend to other core model architectures.

with removing the hack, expert_count now follows num_local_experts from the config
(still need approval on shared code modification)

gabe-l-hart · 2026-06-29T15:53:53Z

+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+
+        # the model is dense; force the dense invariant in case the llama parent


The GraniteMoeModel inherits directly from GraniteModel and only overrides set_gguf_parameters to optionally add the sahred_intermedaite_size (GraniteMoeShared). I don't think these explicit overrides are needed and they make the architecture less flexible to future core model architectures. In fact, I think the best course of action may be to inherit from GraniteHybridModel which itself acts as a child of both GraniteModel and GraniteMoeModel depending on the presence of SSM layers and experts. Some of the other logic may get more complex, but it's worth trying.

Dropped the hardcoded add_expert_count(0), it's config-driven now. Kept one dense guard on expert_used_count: the model's config.json carries num_experts_per_tok: 2 because the upstream HF config (GraniteSwitchConfig extends transformers.GraniteMoeHybridConfig) sets it, so without the guard a dense model emits expert_used_count=2 with expert_count=0 and fails the loader assert.

On reparenting to GraniteHybridModel: I tried it, seems like it's the wrong base here. Its __init__ auto-downgrades the arch when there are no SSM layers, and since the config has num_experts_per_tok: 2 it rewrites graniteswitch → granitemoe. It'd also pull in Mamba2 machinery (hparam_prefixes, SSM params, mamba set_vocab) that a dense all-attention model doesn't use. GraniteMoeModel stays the cleaner parent, the flexibility comes from the config-driven expert count, not the base class.

gabe-l-hart · 2026-06-29T17:49:30Z

+    auto * inp_attn = build_attn_inp_kv();
+
+    // router attention: a single causal head whose K/V live in the KV cache at
+    // layer R, recovering the per-token adapter index in-graph. Only dim 0 carries


Ah, I see: you keep the KV cache for this layer at the end, not the beginning, so we don't have to shift everything else.

gabe-l-hart · 2026-06-29T17:55:41Z

+    ggml_tensor * Kcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd_kv, qkv->ne[1], qkv->nb[1], n_embd_q*ggml_element_size(qkv)));
+    ggml_tensor * Vcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd_kv, qkv->ne[1], qkv->nb[1], (n_embd_q + n_embd_kv)*ggml_element_size(qkv)));
+
+    Qcur = ggml_add(ctx0, Qcur, build_switched_lora_delta(sl.a_q, sl.b_q, cur, adapter_ids));


Ok, I think this is the core of the in-model adapter implementation point. If we explore the route of using the existing llama_lora_adapter infra, this would get deferred into the build_attn below automatically I think.

CISC · 2026-06-29T18:47:52Z

@CISC I'm particularly curious your thoughts on reusing existing adapter mechanisms vs this implementation that reimplements adapters in just the right places for this model so that it doesn't need to change anything about the existing adapter code paths.

We have avoided this path a couple of times in the past already, but I think it's long overdue to try and tackle this, I think it would be worth refactoring adapters to handle this, unfortunately the timing is bad for me as I am on forced vacation. :)

I'm sure @ngxson has some thoughts on this as well.

gabe-l-hart · 2026-06-29T19:55:45Z

unfortunately the timing is bad for me as I am on forced vacation. :)

Zero problem (I hope everything is ok!). @ngxson if you're eager, I'd love to hear your thoughts, but I'm also game to take a pass at this if you're swamped. I had started it already, but hadn't gotten much further since @barvhaim was able to get things working end-to-end.

CISC · 2026-06-29T20:44:59Z

unfortunately the timing is bad for me as I am on forced vacation. :)

Zero problem (I hope everything is ok!).

Oh yes, given the circumstances fantastic in fact. :) Long story short I'm between jobs, all will be revealed by August...

ngxson

Given recent models like deepseekv4 / minimax also have complicated cgraph and kv logic, I think it could be fine to add this one. We can accept it as long as the impl is mostly confined inside granite-switch.cpp

Things that I'm less comfortable about the PR:

Excessive AI-generated comments. Most comments are just noise and doesn't add much to my understanding of the code
Still a lot of hacks outside of granite-switch.cpp

gabe-l-hart

A few more thoughts/comments, especially around simplifying the added tensor names with suffixes.

gabe-l-hart · 2026-06-30T17:19:16Z

+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+
+        # dense model: override any expert counts the llama parent emitted


I'm still not sure we want to explicitly override like this. The upstream parents should handle this correctly for dense or MoE target models, so this just unnecessarily limits to dense-only.

barvhaim · 2026-07-02T12:09:32Z

Given recent models like deepseekv4 / minimax also have complicated cgraph and kv logic, I think it could be fine to add this one. We can accept it as long as the impl is mostly confined inside granite-switch.cpp

Things that I'm less comfortable about the PR:

Excessive AI-generated comments. Most comments are just noise and doesn't add much to my understanding of the code

Still a lot of hacks outside of granite-switch.cpp

+1
Removed the redundant AI generated comments

gabe-l-hart

One tiny NIT, but otherwise this looks like it's in good shape to me. @ngxson I think it's ready for CI and your next review when you have time.

gabe-l-hart · 2026-07-06T15:50:43Z

+// slots in dim 2 (slot 0 is zero-filled so base tokens add an exact-zero delta).
+// A: {n_embd_in, max_lora_rank, N}   B: {max_lora_rank, n_embd_out, N}
+// 7 injection sites (q, k, v, o, gate, up, down), each with an A and a B = 14 tensors.
+struct llama_layer_switch_lora {


Given @ngxson 's input, I think we're going to avoid the "overhaul how adapters work" piece for now.

ngxson

CI need to be fixed too

…g, TODO)

barvhaim · 2026-07-12T13:57:49Z

CI need to be fixed too

skipped the arch in test-llama-archs and added FIXME for now

ngxson

let's also wait for @CISC review for libllama changes

NOTE: This shadows the work done for Granite Speech ggml-org#25107 Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

gabe-l-hart · 2026-07-14T19:10:55Z

@ggerganov With @CISC out of pocket is there someone else that can take a look at the core library changes on this PR?

New llama_hparams.router_layer: https://github.com/ggml-org/llama.cpp/pull/25107/changes#diff-ee47e8a0fad92d8bf326a0355b15607854a618311472962cab3fa4ffb0d0562aR58
New llama_hparams.has_rope(uint32_t il) const: https://github.com/ggml-org/llama.cpp/pull/25107/changes#diff-ee47e8a0fad92d8bf326a0355b15607854a618311472962cab3fa4ffb0d0562aR368
Graph shift logic based on has_rope: https://github.com/ggml-org/llama.cpp/pull/25107/changes#diff-4ab5300c99a0180f396cedfc9ce8b1948a5d51ff6e0e2f744398eda7df2c2ce3R1928
Allow GGML_OP_MUL_MAT_ID w/out MoE when IDs used for router layer: https://github.com/ggml-org/llama.cpp/pull/25107/changes#diff-8d21845db02f372f1fd8af99813f355fec8742691c17558d7fd3ed983d49995eR935

ngxson · 2026-07-23T14:09:13Z

Quick note on this, running code-review reveals some correctness issues that must be address. You can get the skill here: #26042

barvhaim requested review from CISC and ggerganov as code owners June 28, 2026 16:02

github-actions Bot added model Model specific conversion labels Jun 28, 2026

barvhaim marked this pull request as draft June 28, 2026 18:04

pwilkin marked this pull request as ready for review June 28, 2026 18:14

pwilkin marked this pull request as draft June 28, 2026 18:14

barvhaim marked this pull request as ready for review June 28, 2026 18:42

gabe-l-hart self-requested a review June 29, 2026 15:40

gabe-l-hart reviewed Jun 29, 2026

View reviewed changes

barvhaim force-pushed the feature/granite-switch branch from 111d333 to 9249531 Compare June 30, 2026 06:53

ngxson reviewed Jun 30, 2026

View reviewed changes

gabe-l-hart reviewed Jun 30, 2026

View reviewed changes

barvhaim force-pushed the feature/granite-switch branch from 758fbbb to aa441eb Compare July 1, 2026 14:17

gabe-l-hart approved these changes Jul 6, 2026

View reviewed changes

barvhaim force-pushed the feature/granite-switch branch from bae32b6 to 0a7e411 Compare July 7, 2026 07:08

barvhaim requested a review from ngxson July 7, 2026 21:34

ngxson reviewed Jul 7, 2026

View reviewed changes

Comment thread src/llama-kv-cache.cpp Outdated

ngxson reviewed Jul 7, 2026

View reviewed changes

barvhaim force-pushed the feature/granite-switch branch from 290d8d4 to 97700cf Compare July 8, 2026 16:43

barvhaim requested a review from ngxson July 8, 2026 18:22

gabe-l-hart mentioned this pull request Jul 9, 2026

model: GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM #25505

Draft

ngxson reviewed Jul 12, 2026

View reviewed changes

Comment thread gguf-py/gguf/constants.py Outdated

Comment thread src/models/granite-switch.cpp Outdated

barvhaim requested a review from JohannesGaessler as a code owner July 12, 2026 12:23

barvhaim added 22 commits July 12, 2026 16:04

granite-switch: derive n_slots()

e5b1539

granite-switch: move llm_graph_input_switch into granite-switch.cpp

37a3f87

granite-switch: cut AI-style narration comments

0173b54

granite-switch: collapse multi-line comments

9b6cb51

granite-switch: rename control_token_* maps to adapter_token_*

4a54c03

granite-switch: cut noise comments

4675356

granite-switch: rename embedded LoRA tensors to <base>.lora_a/lora_b

ea39782

granite-switch: GGML_ASSERT token input to avoid UB on embeddings

600ef1b

granite-switch: TODO for raw embedding input support

8379826

granite-switch: collapse LoRA tensor constants to .lora_a/.lora_b suffix

010f22d

granite-switch: drop n_expert_used hack, guard mul_mat_id buft probe

48fc26f

granite-switch: stop forcing dense expert counts, read from config

1870e92

granite-switch: renamed control_token_gain metadata key to router_gain

c9a2850

granite-switch: trim header comments to match native style

dbfd779

granite-switch: collapse LoRA tensors to base name + suffix

17ad165

granite-switch: inline suffix checks in tensor op resolution

9fc6854

granite-switch: drop switch-lora struct comment

fc5e001

granite-switch: guard router layer index and inline n_slots

0cd4391

granite-switch: group adapter metadata under {arch}.adapters.* namespace

f769da1

granite-switch: add hparams.has_rope(il) for KV-shift rope skipping

aac9a24

granite-switch: skip arch in test-llama-archs (adapter fixture missin…

1a0acb5

…g, TODO)

granite-switch: Keys.Adapters namespace + simplify n_slots

f57f905

barvhaim force-pushed the feature/granite-switch branch from 1cb71b5 to f57f905 Compare July 12, 2026 13:07

barvhaim requested a review from ngxson July 12, 2026 13:57

ngxson approved these changes Jul 12, 2026

View reviewed changes

ngxson mentioned this pull request Jul 23, 2026

llama: various bug fixes #26051

Merged

Uh oh!

Conversation

barvhaim commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Requirements

Uh oh!

ggml-gh-bot Bot commented Jun 28, 2026

Uh oh!

barvhaim commented Jun 28, 2026

Uh oh!

gabe-l-hart left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CISC commented Jun 29, 2026

Uh oh!

gabe-l-hart commented Jun 29, 2026

Uh oh!

CISC commented Jun 29, 2026

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabe-l-hart left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barvhaim commented Jul 2, 2026

Uh oh!

gabe-l-hart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barvhaim commented Jun 28, 2026 •

edited

Loading