VLLM / LLama.cpp integration

[ExtReMLapin]

Hello

VLLM and llama.cpp are the main inferences engines available (vllm is also in python), is there any plan to open a PR to get support of KVzip directly inside those inference engines ?

vllm also implemented few days ago DCA (dual chunk attention)

[samidten]

+1

[Janghyun1230]

Yes, we'll definitely try to integrate our method with inference engines, including vLLM. I'm wrapping up some personal stuffs and plan to start the work as soon as possible.