Is it possible the library cache is sending video late?

[mkrn]

I got a situation where stream went on for 48 minutes and only 40 minutes of video is recorded.
It looks like in transcoding in the second part of video it started to lag significantly falling behind to 0.7x
Inspecting video recorded on server it looks like library may have been sending frames late without dropping frames..
Here's the analysis of the frames in recorded video:

Attaching progress log:
ffmpeg-OMTGVO-progress.log

Is this possible that under certain situation the library sends old frames way behind @shogo4405 ?

[shogo4405]

The library itself does not have the capability to drop frames. If video frames cannot be sent due to temporary congestion or other factors, there may be a delay in frames, but when the bandwidth recovers, it should catch up and recover.

For frame dropping and bitrate control, it's discussed here. The approach is to have library users implement these features themselves.#1252

[mkrn]

Thanks for explaining.
Yes my question is - how to decide to drop frames? Based on what criteria?
Ideally it should NOT push frames older than a few seconds (i.e. buffer should be only certain size)
What is the limit to buffer of older frames in the library right now? In frames/ seconds or bytes?
Is it possible that it's 8 minutes late out of 48 minutes?
We're getting this problem recently regardless of adaptive bitrate implementation

in func streamWillDropFrame(_ stream: NetStream) -> Bool { function there's no access to rtmpConnection to be able to tell how much back are the frames currently sending.

Can we control the buffer window size?

[mkrn]

@shogo4405
I'm experimenting with increasing buffer size, and implementing drop frame based on the queue. I had to make
socket of RTMPConnection public, so that I can access the outputBufferFullness

func streamWillDropFrame(_ stream: NetStream) -> Bool {
        let bufferFullness = rtmpConnection.socket.queueBytesOut.value
        print("streamWillDropFrame", bufferFullness, rtmpConnection.socket.outputBufferSize)
        let shouldDrop = bufferFullness > Int64(0.6 * Double(rtmpConnection.socket.outputBufferSize));
        return shouldDrop
}

[shogo4405]

1. First, the library does not do anything at all, so it will indefinitely lag as long as there is memory capacity available.
2. I haven't implemented features like buffer window size adjustment.

Recently, we have been sending larger volumes, so I'm planning to implement bitrate change algorithms in multiple scenarios.

[shogo4405]

Let's assume a video size of 3Mbps at 30FPS as the expectation.

Since the VideoToolbox encoder is ABR (Adaptive Bitrate), dropping a few frames within one second will get us closer to 3Mbps. The effective way to avoid delaying frames is to lower the bitrate for the video.

It's advisable to lower the framerate to 15FPS when reducing the adaptive bitrate down to 1.5Mbps.
Reducing it to around 10FPS might be a good choice when lowering the bitrate to 1Mbps.
The value of RTMPConneciton.previousQueueBytesOut determines the delay. If it's 1 times the set bitrate (audio + video bitrate) divided by 8, it results in a 1-second delay. If it's 2 times, then there's a 2-second delay.
Reference Link

[mkrn]

OK, thanks @shogo4405 I was able to tune the algorithm to take into account the previousQueueBytesOut to keep the queue to minimum