Bug: Duplicate and out-of-sync frames if first GOP starts with B pictures

[shree1717]

Assume the very first GOP has the following structure (in display order): BBIBBPBBP... and the GOP is not marked as closed (D2VWitch does this). Also assume that pictures are requested linearly, i.e., 0, 1, 2, ... Then when src/core/decode.cpp:decodeframe() is asked for picture 0, it will return picture 0 — fine. If now asked for picture 1, however, it will return frame 0 again: decodeframe() finds that the first I picture follows the requested picture, decides it has to start the decoding process again (which is superfluous, but not wrong), and set the picture offset to 0 (cf. “offset = n > f.offset ? 0 : f.offset - n” on line 313). When asked for picture 2 (i.e., the I picture), it again sets the offset to 0 (same source line) and will deliver picture 0 again, and after that it will deliver picture 1 when asked for 3, 2 for 4, etc.

This “only” results in duplicate pictures, but it also breaks frame-accurate access, because if frames are requested linearly, say, 0, 1, 2, ..., 100, spanning several GOPs, you will get pictures 0, 0, 0, 1, 2, ..., 98, but if you request picture 100 by jumping directly to it, you do get the correct picture, i.e., 100.

I have attached a patch but because I could not get my head around the current code, it is basically a complete re-write of decodeframe(). Feel free to laugh your behind off and dismiss it out of hand.

0001-BUG-If-the-very-first-GOP-is-a-sequence-of-say-BBIBB.patch.txt

PS: ffms2 seems to have the same problem.

[dwbuiten]

Will take a look tomorrow; this slipped my mind... for a long time.

[dubhater]

These first two B frames appear (in coded order) after just one reference frame, right? So it's impossible to decode them. Should D2V Witch even write these two frames into the d2v file?

[dwbuiten]

If they're not written, does it 'fix' the seeking issue?

[dubhater]

If these leading B frames are not written in the d2v, the two extra copies of the I frame are not returned, so it fixes that problem.

I'm not sure there is a seeking problem, though. If there is, I don't have the right video to reproduce it. My seek test script doesn't find any problem.

I suspect @shree1717 is simply thinking about this a bit wrong: if you request pictures in order from the start, you get pictures 0, 1, 2, 3..., not pictures 0, 0, 0, 1, 2... It's just that pictures 0 and 1 happen to be copies of picture 2. If you request picture 100, you get picture 100, no matter how you got there. (The I picture at the start is not picture 0 in display order, it's picture 2.)

The first GOP of the sample I tried looks like this:

900 5 0 2324 0 0 0 73 70 d1 f2 f3 e0 f1 f2 e3 f0 f1 e2 f3 f0 e1 f2 f3 e0
                    |  |  |
                    |  |  |- 2 in display order, 0 in coded order, I frame, decodable without the previous GOP
                    |  |
                    |  |- 1 in display order, 2 in coded order, B frame, not decodable due to missing 2nd reference frame
                    |
                    |- 0 in display order, 1 in coded order, B frame, not decodable due to missing 2nd reference frame

[dubhater]

This guide says that B frames don't have to use two reference pictures, meaning that leading B frames may be decodable sometimes. D2V Witch doesn't check if this is the case. Also removing the two B frames probably interferes with audio/video sync, so D2V Witch will not remove them.

[shree1717]

I suspect @shree1717 is simply thinking about this a bit wrong: if you request pictures in order from the start, you get pictures 0, 1, 2, 3..., not pictures 0, 0, 0, 1, 2... It's just that pictures 0 and 1 happen to be copies of picture 2. If you request picture 100, you get picture 100, no matter how you got there. (The I picture at the start is not picture 0 in display order, it's picture 2.)

I don’t think so. If you jump to frame 100, d2vsource does the right thing: It back-steps into the previous GOP, starts decoding there, and goes on linearly until frame 100, discarding all previous frames (I think; I haven’t looked at the code for a long time). Therefore I know how picture 100 looks like.

If you start decoding at picture 0 and go on linearly, you get three duplicates at the beginning, and will arrive at picture 100 when you have requested 102 frames. That’s how you know that the first three pictures delivered are actually picture 0. The picture sequence I mentioned in my bug report (i.e., 0, 0, 0, 1, 2, ..., 98) is what I actually observed (I think; my memory is a bit foggy, but please don’t take this as veiled reproach).

[dubhater]

Please provide a sample where you see this problem.

[motbob]

https://0x0.st/issu.ts

This is a sample that breaks down completely when you load a d2vwitch-generated d2v with d2vsource and feed it to a deinterlacer (with fpsdivisor=1, especially). Regrettably, i don't know if it's relevant to the OP's issue.

The reason I am posting this here is that d2vsource has not handled any .ts that I have worked with correctly (though I get them all from the same person). I have had success demuxing the .ts files with eac3to and loading the m2v with lsmash. If you want more samples—from other TV channels, even—i can provide them.

[myrsloik]

Sample still required for testing or no progress can be made

[videoh]

Some perspective from DG land...

For DGDecode, when a stream starts with B frames (open GOP) , they are delivered as copies of the first decodable (I) frame. This is done to retain audio/video sync. The philosophy is that every coded frame should deliver an output frame, but not a macroblocked one. As somebody here pointed out this affects only the start of the stream because on random access, decoding starts at the previous GOP.

[myrsloik]

A small sample is very welcome if you have one. I'm not sure exactly what the behavior is after my changes in the newffmpeg branch.

[videoh]

I need to leave for several hours but will post one when I return.

[videoh]

Here is a link to an open GOP MPEG2 elementary stream.

http://rationalqm.us/misc/zdf.m2v

Right click and save as.

[myrsloik]

This is definitely completely broken and I'm not sure if the seeking is closer to one or the other behavior