New feature about KV cache truncate and sliding window attention for all models #12388
ai-easy-cpu
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
KV cache truncate at runtime (sliding window attention) seems to be a easy workaround to trade some quality with efficiency in long-context, where RAM is limited and caches are all evicted.
Currently I do not see any server config on this. Is this a possible new feature?
Beta Was this translation helpful? Give feedback.
All reactions