Directly rendering a given VideoFame?

[marcchristie35]

I am working on a sports video analysis tool with Qt. Decoded frames are stored in a FIFO FrameBus waiting to be processed. My approach currently consists in converting the mdk::VideoFrames to cv::Mat / QImage and pushing them on the FrameBus. Ideally, I'd like to keep the mdk::VideoFrame on the frameBus for some amount of time, so it can then be rendered later, typically through a render(mdk::VideoFrame &) method to keep the benefits of an efficient (in my case, OpenGL) rendering.

I believe the memory of a given VideoFrame is quickly invalid after the frame has been processed and rendered, so it would require a deep copy of the VideoFrame.

Therefore, a method to get a copy of the current frame, together with a method to render it later would be nice (unless there's a simple workaround). It could also be useful when creating a simple caching system to very quickly scrub through a sequence of images with the mouse/keyboard (without converting all of them to cv::Mat or QImage).

And, by the way, great work on this mdk-sdk :)

[wang-bin]

what's your platform? the convertion can be very slow and maybe impossible for some decoders. you can try Player.setProperty("video.decoder", "copy=1") so you can always get frame data but rendering performance will be worse. In callback of Player.onFrame<VideoFrame>(callback) you can copy a frame and VideoFrame.to() to convert to a different format or size.

There's no way to control when to render time directly. A possible solution is creating 2 playes, one for decoding but no renderer, another for rendering.

   Player frameGen;
   frameGen.setMedia(...);
   frameGen.setProperty("video.decoder", "copy=1");
   frameGen.onFrame([&](VideoFrame& v, int){
        if (v.timestamp() == TimestampEOS) { // eof frame format is invalid
            if (ret == 0) { // -3: seek error
                ret = decoded ? 0 : -1;
            }
            //p.set(State::Stopped);
            return 0;
        }
        if (!v) { // an invalid frame is sent before/after seek, and before the 1st frame
            return 0;
        }
        // convert and/or copy frame
        rgba = v.to(PixelFormat::RGBA, width, height);
        // add to your frame bus
        return 0;
    });
    frameGen.set(State::Playing);

    Player renderer; // just a renderer, no demuxing and decoding
    renderer.setRenderAPI(...);
    renderer.setVideoSurfaceSize(....);
    renderer.setRenderCallback(....); // dispatch update event

    // when you want to display a frame from frame bus
    render.enqueue(aFrameFromBus);