Question about streaming design and implementation

[winke88]

Dear authors, thanks for your wonderful work!

I have a question about your streaming inference design and implementation. I notice that in inference.py, you implement the streaming inference similar to multi-turn conversation, which waits for user's response text, update the kv cache (keep recent text and vision tokens) and process the next video chunk.

However, from my perspective in a real world setting, video streams should be continually processed to meet the real-time requirements, and it may not be proper to wait for user's response and this will block the process of video streams. I admit that encoding user's responses as historical information will be beneficial for video understanding, is there any method that can do this in an asynchronous way?

Can you elaborate it with your design? Thanks!

[winke88]

Dear authors, could you explain this question? @xrorrim @Guangxuan-Xiao

[xrorrim]

We do not actually wait for the user’s response text; instead, we simply adopt the conversation format used during model pretraining. However, we acknowledge that handling video inputs and model computation can indeed be done asynchronously.