[Feature Request] Implementation of Parallel Frame Processing / Batch Inference to maximize GPU Utilization

[amalrajan]

It looks like this application processes frames in a strictly sequential, single-threaded pipeline (Read -> Upload -> Inference -> Download -> Encode).

Even when using the TensorRT backend, I am seeing a hard performance ceiling of ~9 FPS regardless of whether the output resolution is 1440p or 4K. Monitoring via nvidia-smi reveals:

• GPU Utilization: ~60% (indicating the GPU is frequently idling while waiting for the next frame).
• VRAM Usage: ~2.1 GB out of 6 GB available (leaving ~4 GB of wasted headroom).
• Power Draw: Stagnant at ~72W, well below the thermal/power capacity of the hardware.

I would like to request a feature to enable Parallel Frame Processing or Batch Inference.Specifically: Batch Size / Parallel Frames: An option in the "Settings" or "Render" tab to process $N$ frames simultaneously.

[NevermindNilas]

LLM generated issue requires an LLM generated answer:

A parallel processing / batch-based processing approach in a Python environment is extremely complex to implement, especially for a pipeline like this.

The current workflow relies on strict sequential processing (Read → Upload(?) → Inference → Download(?) → Encode). Operations such as VFI interpolation combined with upscaling and other frame-dependent processing inherently depend on neighboring frames and ordered execution. Because of this, introducing parallel frame processing is not simply a matter of increasing batch size or spawning more threads.

In practice, implementing true parallelism here would require major architectural changes, including redesigning the pipeline for asynchronous GPU queues, frame dependency management, memory synchronization, and encoder coordination. In Python specifically, this becomes even more complicated due to the overhead of inter-process communication, GPU memory handling, and maintaining deterministic frame order.

As a result, a batch or multi-frame parallel processing system would be extremely complex to implement and would effectively require a significant rewrite of the current processing architecture.

[amalrajan]

I used an LLM to rephrase my question clearly, and if you look closely, the hardware bottleneck is 100% real.

maintaining deterministic frame order.

Is that not a solved problem?

[NevermindNilas]

Is that not a solved problem?

Sequentially, sure.. dynamically allocated with perfect frame accuracy and ALSO performant.. I have yet to see a pure python implementation of such. Vapoursynth could do it ( written in C++ with a Python API* ), but it comes with other caveats.

[NevermindNilas]

Say you do 2 batch interpolation, you need a queue just for decoding, then you need another queue to store the frames inbetween I0 and I1 and another queue for I1 and I2 ( let's not mention scenechanges here, be it pre-process or mid-process. You would then want to upscale those, then you'd need to fetch all the frames from each of those queues and the main qeueu which has I0 and I1 and I2 and make sure to upscale them in order.

You can definitely do it with a really good indexing system within a Queue with say tupple ( index 0 ,frame0 ).. (index, frame N ). But that's a real big pain in the ass to implement.

[NevermindNilas]

If what you wish for is True parallelism, where the decode, inference ( let's ignore batch processing here), encode are all decoupled, move to C++ or Free Threaded Python ( which is imho still not there yet ). Otherwise, the GIL will screw you over... you can definitely do some parallelism on the CUDA side of things, but the GIL will ALWAYS get in your way imho.

One workaround that is using subprocesses, but since those don't share the same memory, you'll practically have to have 3 instances of python running at the same time, each using their own unique memory and sending data between them is not trivial ( not really that hard, but not worth it imho ).

Sequentially doing it, with a really good pipeline is the best you will get without moving into C++ or losing significant performance by having to store frames in memory and then magically seeking them as you need them.

[amalrajan]

Appreciate the detailed breakdown. Since it's outside the current scope, I'll leave this here for future consideration if the backend ever moves towards true parallelism. Thanks for the work you've put in so far