Replies: 2 comments
|
+1 I think this would significantly enhance the abilities of llama.cpp as well as enabling multi-modal deployments in VRAM constrained environments as models can switch devices using RAM as necessary. Although |
|
Update: I've been working on a server-level implementation for this, and have a draft PR up that I'm working out of (#25963). I've implemented basic versions of three methods that support most of the features I described in the original post: the ability to reinitialize context, and to enable/disable both the mmproj and speculative decoder. I'm including notes on the API and next steps below and would appreciate feedback 😄 Server and Model APIsThe server-context API I'm running with for now is as follows: // reinitialize the model's context without reloading weights
// returns true on success
bool reload_context(common_params & params);
// unload an existing mmproj and/or load a new one
// returns true on success
bool reload_mmproj(common_params & params);
// unload an existing speculative decoder and/or load a new one
// returns true on success
bool reload_speculative(common_params & params);I've also added context-only reload methods to // (from common.h)
struct common_init_result {
// ... other methods
// reinitialize the model's context without touching weights
llama_context * reinit_context(common_params & params);
// helper method added to capture some behavior from common_init_from_params
void finalize_and_warmup(common_params & params);
private:
// helper method used by both constructor and reinit_context
llama_context * init_context_inner(common_params & params);
};
// (from speculative.h)
struct common_speculative_init_result {
// ... other methods
// reinitialize the speculative model's context without touching weights
llama_context * reinit_context(common_params & params, llama_model * model_tgt, llama_context * ctx_tgt);
};TestingI've done some basic testing on reload_context (growing/shrinking context window, and adjusting kvcache precision); that test file is included in the PR but is mostly vibecoded and not intended to land in the main repo. That said, it seems to work well! Running with Qwen3.6-27B, resizing a large context seems to take ~200 ms, which is about 1/10th the time it takes to load the entire model. If you want to run my tests, here are the basic commands: Checkout/Buildgit clone https://github.com/wadealexc/llama.cpp
cd llama.cpp
git checkout feat/context-reinit
cmake -B build -S . -DLLAMA_BUILD_TESTS=ON -DGGML_CUDA=ON
cmake --build build --target test-reload-context test-llama-archsRun with generated fixturesctest --test-dir build -R '^test-reload-context$' --output-on-failure -VRun against a real model(example model; replace with whatever you want) build/bin/test-reload-context \
-m /path/to/Qwen3.5-9B-MTP-Q4_K_M.gguf \
--spec-type draft-mtp --spec-draft-n-max 2 \
-c 4096 -np 1 -ngl 99Next StepsI don't have an HTTP API implemented yet. The main reason for this is that I need to think more carefully about how I would use this in my current inference harness, and whether additional endpoints/operations/changes are needed. Some additional things I suspect will be needed:
Future WorkCross-device reconfiguration. I haven't touched anything that would require moving layers off of one device and onto another, but I think that's necessary to support a lot of reconfiguration use-cases. For example, maybe I want 175k context total, but I can fit 150k on my GPU. So my inference backend runs on my GPU until we hit ~150k, then offloads layers to CPU until I can fit 175k. Testing. Since these changes are such a large refactor to llama-server's model lifecycle, it might be time to introduce some actual tests for Open QuestionsI'd appreciate feedback on any of the above, but specifically:
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Update: I've added my progress and notes on the current direction to the comment below! I'll keep this post mostly untouched.
llama.cpp's server runtime maintains two different kinds of state:
Main model state
Auxiliary/Inference state
Today, these are initialized and destroyed together. Loading a model means loading not just its weights, but also an mmproj, a speculative decoder, and a kv cache.
Problem: Inference Configuration on Model Load
Model and inference parameters can both be configured via CLI arguments, but this only happens at load time. Changing any configuration requires fully killing the process and restarting: removing model weights and inference state, then reinitializing all of it. For example, adding an mmproj to a model after it's been loaded:
graph LR User["User"] subgraph ServerA["llama-server"] A1["Read model from disk"] A2["Init kv cache"] A1 --> A2 end User -- "1: load model" --> ServerA subgraph ServerB["llama-server"] B1["Read model from disk"] B2["Init kv cache"] B3["Init mmproj"] B1 --> B2 B2 --> B3 end User -- "2: kill process" --> ServerA User -- "3: load model + mmproj" --> ServerBThis pattern treats inference configuration as a decision to be made infrequently, as it has a significant time penalty attached. However, runtime requirements often change over the lifetime of a model.
As an example, here are some scenarios that might cause an inference backend to reconfigure the models it's serving:
Configure-on-load pushes configuration decisions to account for the "worst case" runtime requirements of all of these scenarios, making the average case worse.
Proposal: Runtime Configuration
In general, runtime inference configuration means reconstructing/reconfiguring a model's runtime context while its weights stay put (or without fully restarting). This can take many forms, but a few options I've been thinking of are:
While working on llama-server, I noticed that most of the ideas I had in this vein shared the same underlying limitation: model weight and mmproj/mtp/config lifetimes are tightly coupled. This coupling is the primary architectural limitation preventing runtime configuration from being implemented, and addressing it would require refactoring llama-server somewhat.
Additionally, I imagine exposing these runtime operations via HTTP endpoints, as that feels like the natural way to interact with them given how the built-in router mode (and other llama.cpp management libraries) work.
Example Use Case: Serving Multiple Models
By exposing these and similar methods for granular inference management, llama-server significantly enhances the model-serving usecases it already supports. It especially becomes easier to serve small task models alongside a single larger model, as memory requirements are more flexible.
For example, consider a chat application where a user talks to a single larger model, which is set up alongside a smaller model used for various agentic tasks.
For ease of visualization, let's assume we're loading both onto a GPU with 24 GB of capacity. There's a cost associated with each loaded component, deducted from our capacity:
Both are initialized at startup with some basic capabilities, and as the user chats, the main model's context expands into available, unallocated space:
--- title: "llama-server (GPU: 20/24 GB)" --- graph LR subgraph Main["Main (GPU: 15 GB)"] A1["Main model<br>+ mtp<br>+ mmproj"] A2["30k context"] A1 --- A2 end subgraph Task["Task (GPU: 5 GB)"] B1["Task model"] B2["Empty context"] B1 --- B2 end Main ~~~ TaskA request comes in that requires the task model to summarize results from a web search. This requires the task model's context window to expand to handle 50k context. 4 GB are already available, and the additional 1 GB is freed by unloading the main model's mmproj, which has not been used in the current session:
--- title: "llama-server (GPU: 24/24 GB)" --- graph LR subgraph Main["Main (GPU: 14 GB)"] A1["Main model<br>+ mtp<br><span style='color:red'>- mmproj</span>"] A2["30k context"] A1 --- A2 end subgraph Task["Task (GPU: 10 GB)"] B1["Task model"] B2["50k context"] B1 --- B2 end Main ~~~ TaskAs the task model completes its task, the bulk web page results can be cleared, and its entire context can be disposed. A summary of the results are sent to the main model, and its mmproj is restored:
--- title: "llama-server (GPU: 22/24 GB)" --- graph LR subgraph Main["Main (GPU: 17 GB)"] A1["Main model<br>+ mtp<br><span style='color:green'>+ mmproj</span>"] A2["50k context"] A1 --- A2 end subgraph Task["Task (GPU: 5 GB)"] B1["Task model"] B2["Empty context"] B1 --- B2 end Main ~~~ TaskAll reactions