Native Context Folding

[CaasiMule]

Context Folding is a technique that can enhance long-context workloads by decomposing tasks and using subagents. In that paper, it was done at the harness level. However, I believe that llama.cpp can implement it at the inference provider level. This has numerous benefits, which I will outline after describing what the system may look like.

An efficient system prompt (or RL like the above paper used) is likely required for this to work properly, but llama.cpp can make this easier and faster with a fixed framework that tinkerers can later optimize for.

Let's say you have a large codebase and you want an LLM to fix a known issue. You give it your prompt, and tool definitions and system prompt are prefilled. That KV cache is then marked as a checkpoint.

Then, you let the LLM decompose the task, but this isn't normal subagents. The LLM calls a delegate tool that hooks into llama.cpp internally. Perhaps the LLM decides to send a subagent to give a summary of the project structure and "go ham with tokens." The "subagent" reads plenty of files to thoroughly understand the structure, then returns a fold call with just the important information. llama.cpp will then roll back the KV cache to the checkpoint, append the body of the fold call, and then compute from there. From there it can keep going, diagnosing if the issue is with the frontend, the backend, etc. Subagents can spawn their own subagents (with a depth limit) and only return important context.

I know this is fundamentally a harness-level feature and is near-identical to existing subagent systems, but implementing it at the inference provider level has at least the following benefits.

• The main prefill is NEVER recomputed. Prompt caching is always a best-guess, but by implementing it at this level, it is guaranteed because llama.cpp itself knows about the subagents. There is no "do I have to throw out the main thread's context because a subagent is taking all of the ram," there is just "a subagent is done, rolling back the KV cache. . ."
• The main "thread" or agent never wastes resources. Each delegate call suspends the parent and lets the child run with the full resources and very little memory overhead (pretty must just "where is the checkpoint?"). A subagent can run for 2 hours and the main thread will resume near-instantly after it's done.
• Sending multiple subagents at a time executes them sequentially while not returning "agent 1 done, agent 2 waiting" to the main thread and having it waste resources, which is desirable for local AI's extreme efficiency requirements.

I'd assume there are other benefits to keeping it internal which I have not thought of yet.

llama.cpp already has server-sided tools implemented as an experiment, and I think this could use those existing systems. This might involve inventing an entirely-new part of the stack (inference provider tools that have to sync to the harness), but those issues can be tackled if/when this is deemed feasible and implemented.

Please feel free to roast this approach! If it's not suitable for llama.cpp, just explain why so we can both move on with our lives.