How to render frames from h264_mediacodec directly, without conversion, accelerated?

[geminixdev]

In #273 you explain the process between decoding and rendering clearly:

Let me explain how "everything" works:

ffmpeg decodes a source using hardware acceleration and puts data to D3D11 or Metal or VDPAU or VAAPI or whatever textures.
Pointer to texture is returned in av_frame->data[x]
To get access to the data for the frame in this case you should use av_hwframe_transfer_data: https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavvideobuffer_gpu.cpp#L22 - it is done when you call QAVVideoFrame::map() --- it downloads data from gpu to cpu!
What should you do when you would like to render it without mapping in RHI?

F.e. using Metal textures: we convert QAVVideoFrame to QVideoFrame and set QVideoFrame::RhiTextureHandle - > defines that the frame contains raw texture handle.
RHI knows that current platform is macOS and handleType() points to RhiTextureHandle so no need to try to reuse this texture handle instead of calling QVideoFrame::map().

For Windows, D3D11 render is used by default. So if you use QVideoFrame::RhiTextureHandle , Qt is expecting to have pointer to ID3D11Texture2D.
2.1 So if ffmpeg creates this texture for us, we can try to return it - > it does not work since pixel format is NV12 which expects to have 2 textures - color and data.
2.2 Creating 2 instances of ID3D11Texture2D textures and copying data to it from original -> did not work for me yet -> Qt should reuse these textures and avoid mapping but crashes. Need to dive deeper, maybe doing something wrong.

How about Android with h264_mediacodec?

The frames delivered by avcodec_receive_frame seem to be NV12 and seem not to get converted or mapped, when sending them to the videosink they are still NV12 (at least I don't see where mapping or conversion would happen).

In armeabi-v7a devices the h264_mediacodec decoding is too slow, already not smooth anymore with HD.
Top shows 100% CPU load, for mediacodec??? (On arm64-v8a only around 20% for mediacodec.)

So for arm64-v8a devices the decoding and the process after avcodec_receive_frame seems to be very fast, all is playing smoothly, even on older devices.

However on armeabi-v7a devices, usually Android boxes, despite having fast CPUs, the transfer of data in and out of mediacodec plus the decoding is ridiculously slow, and the CPU usage (SD 40%, HD 100%) far too high for hardware decoding.

What could be wrong there, or different there?

[geminixdev]

Just to make that clear: the slowness is not visible when testing with arm64-v8a devices. Unfortunately all Android Boxes are armeabi-v7a, and there it makes them unusable for anything better than SD.

[valbok]

You said that QtMultimedia for 5.15 does not have perf issues?
there are 2 ways to render from QtMM:
https://github.com/qt/qtmultimedia/blob/5.15/src/qtmultimediaquicktools/qdeclarativevideooutput_render.cpp#L344

Uploads data to opengl textures, (and as I remember it is used by default for Android). And should be used with QtAVPlayer and its QVideoFrame

but also it can use window based renderer https://github.com/qt/qtmultimedia/blob/5.15/src/qtmultimediaquicktools/qdeclarativevideooutput_window.cpp#L88
which should be totally copy-free, and rendering is done directly to a window without any videoframes.

Also when you convert QAVVideoFrame to QVideoFrame there is hardcoded converting
https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavvideoframe.cpp#L314

which downloads data from GPU, since there is no support for mediacodec in QRHI (as I remember)

[valbok]

Can you confirm that qml rendering is lagging? Using VideoOutput? Can you also disable sending video frames to VideoOutput, but received from the player and confirm that CPU is low and GUI is acting fast?

1. Checking if decoding does not consume CPU and does not impact to GUI
2. Checking if rendering is not efficient enough, it can be even checked without a player, just send many QVideoFrames to VideoOutput ?

[geminixdev]

Also when you convert QAVVideoFrame to QVideoFrame there is hardcoded converting https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavvideoframe.cpp#L314

which downloads data from GPU, since there is no support for mediacodec in QRHI (as I remember)

Line 314:

            result = convertTo(AV_PIX_FMT_YUV420P);

But in the context, in the lines before that, there is the check for AV_PIX_FMT_NV12, see below:

        case AV_PIX_FMT_NV12:
            format = VideoFrame::Format_NV12;
            break;
        default:
            // TODO: Add more supported formats instead of converting
            result = convertTo(AV_PIX_FMT_YUV420P);
            format = VideoFrame::Format_YUV420P;
            break;

and in my checks (at least with the development devices in arm64-v8a) the frames delivered from avcodec_receive_frame are AV_PIX_FMT_NV12, and still are AV_PIX_FMT_NV12 when going into that QAVVideoFrame to QVideoFrame conversion and also out of that. That makes me thinking that there is no converion happening, thus no slowdown caused by that conversion.

I will rerun timing and pixel format tests with an armeabi-v7a device, to make sure that it is the same there.

[geminixdev]

Can you confirm that qml rendering is lagging? Using VideoOutput? Can you also disable sending video frames to VideoOutput, but received from the player and confirm that CPU is low and GUI is acting fast?

1. Checking if decoding does not consume CPU and does not impact to GUI
2. Checking if rendering is not efficient enough, it can be even checked without a player, just send many QVideoFrames to VideoOutput ?

One of my tests on an armeabi-v7a device was to comment out the videoSink->setVideoFrame() command, and that alone had absolutely no influence on the decoding time and mediacodec CPU, did not reduce them.

I will also rerun that, and may be drop the frames right after avcodec_receive_frame.

[geminixdev]

You said that QtMultimedia for 5.15 does not have perf issues?

yes, definitely, Full HD no problem at all, smooth and GUI not lagging, on the same armeabi-v7a devices.

Especially that is irritating, as it contradicts the slow mediacodec decoding and the 100% CPU of it which I see in my tests on exactly the same devices.

[geminixdev]

Also when you convert QAVVideoFrame to QVideoFrame there is hardcoded converting https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavvideoframe.cpp#L314
which downloads data from GPU, since there is no support for mediacodec in QRHI (as I remember)

Line 314:

            result = convertTo(AV_PIX_FMT_YUV420P);

But in the context, in the lines before that, there is the check for AV_PIX_FMT_NV12, see below:

        case AV_PIX_FMT_NV12:
            format = VideoFrame::Format_NV12;
            break;
        default:
            // TODO: Add more supported formats instead of converting
            result = convertTo(AV_PIX_FMT_YUV420P);
            format = VideoFrame::Format_YUV420P;
            break;

and in my checks (at least with the development devices in arm64-v8a) the frames delivered from avcodec_receive_frame are AV_PIX_FMT_NV12, and still are AV_PIX_FMT_NV12 when going into that QAVVideoFrame to QVideoFrame conversion and also out of that. That makes me thinking that there is no converion happening, thus no slowdown caused by that conversion.

I will rerun timing and pixel format tests with an armeabi-v7a device, to make sure that it is the same there.

A first test result, it confirms also on an armeabi-v7a device:

• pixel format is always NV12 (23) from avcodec_receive_frame() up to sending the frame to videosink with videoSink->setVideoFrame().
• the line with result = convertTo(AV_PIX_FMT_YUV420P); is not triggered to run, not executed.

Tests are ongoing.

[geminixdev]

Can you confirm that qml rendering is lagging? Using VideoOutput? Can you also disable sending video frames to VideoOutput, but received from the player and confirm that CPU is low and GUI is acting fast?

1. Checking if decoding does not consume CPU and does not impact to GUI
2. Checking if rendering is not efficient enough, it can be even checked without a player, just send many QVideoFrames to VideoOutput ?

Also here a first test result:

Not executing videoSink->setVideoFrame() apparently does not change anything. Still the same high CPU for Mediacodec (99 - 100% for HD and Full HD) , still the the same overload of decoding with FullHD.
(The unsmoothness of HD cannot get checked without actually seeing the pictures)

Tests are ongoing.

[geminixdev]

In QAVFrameCodec::decode() there is avcodec_receive_frame(). Logging the time it runs gives these results:

HD:

05-03 00:13:59.037  8197  8399 I Player  : [03 0:13:59.036 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1280 x 720 frame->format: 23 frame->pts: 4338057600
05-03 00:13:59.073  8197  8399 I Player  : [03 0:13:59.073 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1280 x 720 frame->format: 23 frame->pts: 4338061200
05-03 00:13:59.111  8197  8399 I Player  : [03 0:13:59.111 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1280 x 720 frame->format: 23 frame->pts: 4338064800

This shows that the second avcodec_receive_frame() took 37 msec, and the third 38 ms. These times are consistent, always more or less the same. Considering 40 ms available for25 fps we are at the limit here.

Full HD:

05-03 01:02:10.570  8197  8397 I Player  : [03 1:02:10.570 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594467600
05-03 01:02:10.651  8197  8397 I Player  : [03 1:02:10.651 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594471200
05-03 01:02:10.733  8197  8397 I Player  : [03 1:02:10.733 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594474800
05-03 01:02:10.892  8197  8397 I Player  : [03 1:02:10.892 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594478400
05-03 01:02:10.899  8197  8397 I Player  : [03 1:02:10.898 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594482000
05-03 01:02:11.055  8197  8397 I Player  : [03 1:02:11.055 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594485600
05-03 01:02:11.068  8197  8397 I Player  : [03 1:02:11.067 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594489200
05-03 01:02:11.221  8197  8397 I Player  : [03 1:02:11.221 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594492800
05-03 01:02:11.232  8197  8397 I Player  : [03 1:02:11.231 +07 I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 23 frame->pts: 4594496400

There is a big fluctuation, between 6 ms and 150 ms, with an average of about 80 ms. Far too long for the 40 ms we have at 25 fps.

So the question seems to be, why is it so slow?
Possibly an option or something in the codec context which must be set or set differently?

[valbok]

1. about NV12 , interesting, but AV_PIX_FMT_MEDIACODEC should be used, otherwise it looks like software decoding.
2. wondering if QAVPlayer itself consumes too much CPU
3. you say that pts diff between 2 frames are increasing?

[geminixdev]

The tests above where with an HK1 X4 Box. Now below tests with a Tanix TX6 Box.
And it gets weirder, here

• it is not NV12, but YUV420P all the way through.
• Top shows 49% for the player and < 20 % for mediacodec.
• The avcodec_receive_frame() times are 4 - 6 ms also with Full HD

05-03 03:13:10.200  7868 22724 I Player  : [03 3:13:10.200 MYT I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 0 frame->pts: 7366644000
05-03 03:13:10.206  7868 22724 I Player  : [03 3:13:10.206 MYT I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 0 frame->pts: 7366647600
05-03 03:13:10.210  7868 22724 I Player  : [03 3:13:10.210 MYT I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 0 frame->pts: 7366651200
05-03 03:13:10.214  7868 22724 I Player  : [03 3:13:10.214 MYT I]   decode_mediacodec, after avcodec_receive_frame. frames.size() 1920 x 1080 frame->format: 0 frame->pts: 7366654800

Why is playback not smooth here for Full HD and HD (HD is almost OK)?
Possibly the high IOW of 23%

User 2%, System 1%, IOW 23%, IRQ 0%
User 287 + Nice 4 + Sys 166 + Idle 10063 + IOW 3327 + IRQ 0 + SIRQ 22 = 13869

  PID USER     PR  NI CPU% S  #THR     VSS     RSS PCY Name
 7868 u0_a89   20   0   1% S    49 2154028K 662952K unk my.player
 2006 mediacod 20   0   0% S    14 166156K  66288K unk media.codec
 1859 system   12  -8   0% S    22 119036K   9612K unk /system/bin/surfaceflinger
28418 u0_a12   20   0   0% S    28 1265700K  95768K unk com.google.android.gms
 1997 audioser 20   0   0% S     9  39648K   4816K unk /system/bin/audioserver
30208 root     20   0   0% R     1   4780K   1568K unk top

(This is still without executing videoSink->setVideoFrame() )

[geminixdev]

• about NV12 , interesting, but AV_PIX_FMT_MEDIACODEC should be used, otherwise it looks like software decoding.

Yes, that is what I expected too. However we seem to get NV12 and YUV420P.

About possible software decoding, I specifically set 'h264_mediacodec' and logcat seems to confirm that its used.

• wondering if QAVPlayer itself consumes too much CPU

I assume that QAVPlayer as library is shown within the Player CPU%. According to top then its not too much.

• you say that pts diff between 2 frames are increasing?

Not the pts diff, that is always ok, 3600. With times I refer to the logged times at each log entry.
[03 3:13:10.200 MYT I] and [03 3:13:10.206 MYT I] show 6 ms difference, thus that avcodec_receive_frame() needed 6 ms.

I will recheck with the first box, the HK1, if there is an error in logcat about h264_mediacodec.

[geminixdev]

I will recheck with the first box, the HK1, if there is an error in logcat about h264_mediacodec.

Logcat shown no indication that software decoding would be used. There are entries like that:

05-03 03:05:00.903   412  4104 D AmlogicVideoDecoderAwesome2: [22]"codecInit done"
05-03 03:05:00.903   412  4104 D AmlogicVideoDecoderAwesome2: [22]"mOutWidth is 1920 mOutHeight is 1080 mFlvFlag=0 mOutBufferCount is 10"
05-03 03:05:00.903   412  4104 D AmlogicVideoDecoderAwesome2: [22]"mOutBufferCount =10 mDecOutWidth 1920 mDecOutHeight 1088\n"
05-03 03:05:00.903   412  4104 D AmlogicVideoDecoderAwesome2: [22]"mIsNativeBuffers =0\n"
05-03 03:05:00.903   412  4104 D AmlogicVideoDecoderAwesome2: [22]"setUp mOutPortChanged=0\n"
05-03 03:05:00.903   412  4104 D AmlogicVideoDecoderAwesome2: [22]"use nv12\n"
05-03 03:05:00.903   412  4104 I OmxComponent: STATE_DONE:  OMX_StateLoaded => OMX_StateIdle : OMX.amlogic.avc.decoder.awesome2
05-03 03:05:00.904   412  4104 I OmxComponent: OMX_CommandStateSet 850 Cmd 0 nParam1 0x3
05-03 03:05:00.904   412  4104 I OmxComponent: OMX-31 STATE_SET:   OMX_StateIdle => OMX_StateExecuting : OMX.amlogic.avc.decoder.awesome2
05-03 03:05:00.904   412  4104 V AmlogicVideoDecoderAwesome2: [22]prepare:315 >
05-03 03:05:00.904   412  4104 I AmlogicVideoDecoderAwesome2: [22]"AllocDmaBuffers uvm mDecOutWidth:1920 mDecOutHeight:1088, 1920x1088"

which might explain why NV12 is used.

[valbok]

You should track Cpu of the application. There is simple way how to determine it is hw accelerated or not. Just compare Cpu usage of the app with mediacodec and without.

[valbok]

I assume that QAVPlayer as library is shown within the Player CPU%. According to top then its not too much.

Does it mean you see low CPU% but lags and delays in receiving,decoding frames?