Edit Prediction rewrites: what happens when token generation is 10-100x faster?

[NicoConstant]

Zed's Edit Prediction is one of the most interesting AI interactions in any editor right now. The idea of rewriting entire functions or files as a native editing primitive is genuinely new.

But there's a hard physical constraint underneath it: standard LLM inference generates tokens sequentially, one at a time. For short completions, it's fine. For full-function or full-file rewrites, exactly the cases where Edit Prediction is most powerful, that sequential ceiling becomes the bottleneck.

We've been working on this specific problem at Kog. Last year we published a benchmark with AMD showing 3.5× faster token generation vs vLLM and TensorRT-LLM on MI300X, engine alone, general-purpose model, no task-specific
optimization.

The next step we're now testing combines three layers:

• Bare metal engine (Assembly/C++), eliminating all software overhead
• Parallel model architecture generating entire code sequences simultaneously rather than token by token
• Model fine-tuned specifically on code rewrite data

The question we're trying to answer: Does x100 generation velocity on full-file rewrites fundamentally change how Edit Prediction feels to a developer?

We're looking for one team to run this benchmark on a real workflow under NDA, we handle 100% of the work.

Would love to hear how the Zed team is thinking about the generation speed ceiling for longer rewrites.