Unable to get KV cache from GPU when KV cache is quantized to q8_0

[KarSri7694]

Hello everyone,

I am currently building a prototype pipeline for the GUI Agent project (GSoC Project 1) and have run into a blocker I'd appreciate some guidance on.

As part of the pipeline, I need to capture a snapshot of the KV cache prior to image input being processed. To handle longer contexts more efficiently, I attempted to quantize the KV cache to q8_0.

Issue
Upon enabling q8_0 quantization, get_state and set_state operations on KV cache tensors fail with the following error:

RuntimeError: [GPU] get_state API is supported only when KV-cache compression is disabled

It appears that KV cache compression (quantization) and the state snapshot API are mutually exclusive in the current implementation. get_state and set_state will not work when KV cache is quantized

What Works
Setting KV precision to f16 resolves the issue — get_state and set_state operate as expected under that configuration.

Question
Is there a known workaround to capture a snapshot of a quantized (q8_0) KV cache, or is falling back to f16 precision currently the only viable option? Any pointers to relevant internals or planned support for this would be greatly appreciated.

Thank you in advance for your help!

[Ganesh-403]

@KarSri7694 The error message is actually quite informative:

[GPU] get_state API is supported only when KV-cache compression is disabled

From what I could find, this is currently a limitation of the GPU plugin rather than a bug.

The OpenVINO documentation explicitly states that when KV-cache quantization/compression is enabled on the GPU:

The GPU plugin does not support the get_state() and set_state() APIs. :contentReference[oaicite:0]{index=0}

So with a configuration such as:

ov_config = {
    "KV_CACHE_PRECISION": "u8",   # q8_0 equivalent KV compression
}

the following is expected to fail:

infer_request.get_state()
infer_request.set_state()

On the other hand, using an uncompressed KV cache (e.g. f16) allows state snapshotting, which matches what you're observing. :contentReference[oaicite:1]{index=1}

So, regarding your question:

Is there a workaround to snapshot a quantized KV cache?

From what I can tell, I couldn't find any documented workaround that allows get_state()/set_state() to operate on a GPU-compressed KV cache. At the moment, your options appear to be:

1. Use f16 (or another uncompressed KV precision) if you need to snapshot and restore KV state.
2. Keep KV-cache compression enabled and avoid relying on the state API.

I also couldn't find any public roadmap or issue indicating planned support for combining GPU KV-cache compression with state snapshot APIs.

Out of curiosity, is your pipeline tied specifically to the GPU plugin?

The reason I ask is that this limitation is documented specifically for GPU. If your prototype can tolerate CPU execution for the snapshot stage, it would be interesting to know whether that satisfies your use case, or whether the snapshot/restore must happen entirely on the GPU.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.

[KarSri7694]

Hi @Ganesh-403

Yes, The pipeline is specifically tied to using GPU, as without it, inference would be way too slow, I was able to create logic using claude to implement get_state() and set_state() for compressed KV cache.

You can find the code I implemented here.

[Ganesh-403]

Alright!

[Ganesh-403]

Hi @Ganesh-403

Yes, The pipeline is specifically tied to using GPU, as without it, inference would be way too slow, I was able to create logic using claude to implement get_state() and set_state() for compressed KV cache.

You can find the code I implemented here.

could mark this answer as accepted so it closed this discussion?