Add tiered KV cache offloading blog post

[orozery]

Covers the host-centric architecture, secondary tiers (filesystem, object storage, P2P), hybrid model support, observability, KV events, and performance benchmarks.

[liranschour]

You mention kv_transfer_params but the code section below captures the kv-transfer-config cli. I suggest to add a small section describing the kv_transfer_params protocol for p2p/pd. You can take it from here: p2p-issue

[dannyharnik]

@liranschour I will add a link to the llm-d guide for this.
https://github.com/llm-d/llm-d/tree/main/guides/p2p-kv-cache-sharing

[ivanium]

Thanks for the nice feature and the blog! I think tiered offloading is definitely an important feature. The current blog feels a bit high-level, though, and I’d love to see more technical details and performance results.

For example, it would be great to include more details, such as the unified KV cache layout, the overall system architecture, how this works with routers like llm-d and Dynamo, etc.

I also think the results would be more convincing with performance numbers at a larger scale, ideally in some (simulated) production settings, along with more metrics such as TTFT, TPOT, cache hit rate, etc.

[dannyharnik]

Thanks @ivanium for your feedback. The high-level feel is by design - we want to ensure this is a good read that gets the main points through. Low level details can be found in the guides or the actual code. That being said, we added a bit more information about the layout and integration with the router and added relevant links to where more details can be found. We also added a link to the actual benchmarking code and full data which includes breakdowns of other metrics like TTFT and TPOT.