FIX: long-term reference list reordering in non-Baseline CABAC streams#3954
Open
niw wants to merge 1 commit into
Open
FIX: long-term reference list reordering in non-Baseline CABAC streams#3954niw wants to merge 1 commit into
niw wants to merge 1 commit into
Conversation
**Problem** When a slice carries `ref_pic_list_reordering` commands for long-term references (`reordering_of_pic_nums_idc == 2`), `WelsReorderRefList2()` matched the requested `long_term_pic_num` against each picture's `uiLongTermPicNum` field. That field is unreliable because long-term references are created via MMCO 3 (short-to-long) and MMCO 6 (mark current as long), both of which only carry `long_term_frame_idx` in the bitstream and never a `long_term_pic_num`. As a result `AddLongTermToList()` stores `uiLongTermPicNum` as `0` for every such picture. With multiple long-term references present, they all compare equal (`0`), so the reorder failed to select the requested entry (matching either the wrong one or none at all), leaving the wrong picture at the head of the reference list and corrupting inter prediction. It only surfaces when multiple long-term references and a long-term reorder command occur together. In practice this is hit when decoding AirPlay Mirroring streams from iOS and macOS, whose High-profile CABAC streams use multiple long-term references with reordering, producing visibly corrupted video. Note that `WelsReorderRefList2()` is only used for non-Baseline CABAC streams (`uiProfileIdc != 66 && bEntropyCodingModeFlag`). The problem was introduced disabled (`#if 0`) by 1b8caef, and later enabled by fa6d099, which flipped it to `#if 1` and exposed it. **Solution** Match by `iLongTermFrameIdx` instead of `uiLongTermPicNum`, in both the selection loop and the subsequent compaction loop. `iLongTermFrameIdx` is the identifier that is actually maintained on the picture (set in `AddLongTermToList`) and, per the spec (8.2.4.1), for frame coding `LongTermPicNum == LongTermFrameIdx`, so the value parsed as `long_term_pic_num` matches it directly. This aligns the reorder path with how the rest of the code already identifies long-term pictures, e.g. `WelsDelLongFromList()` used by MMCO 2, which likewise matches against `iLongTermFrameIdx`. **Testing** Added a focused unit test (`DecUT_RefListReorder.cpp`) that reproduces the case via three long-term references sharing `uiLongTermPicNum == 0` with distinct `iLongTermFrameIdx`, plus an example High-profile CABAC regression stream (`res/ltr_reorder_high_cabac.264`) wired into the decoder API test. **Considerations** Keeping the match on `uiLongTermPicNum` but populating it correctly, i.e. deriving `LongTermPicNum` from `LongTermFrameIdx` per slice (8.2.4.1) or setting it at marking time, would be an alternative option. However, `LongTermPicNum` is a per-slice derived quantity, not a persistent property of the picture, so storing it is spec-awkward. For frame coding, it just equals `LongTermFrameIdx`, making the field redundant. It would also touch the marking path (`AddLongTermToList`, `MarkAsLongTerm`) for no functional gain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a slice carries
ref_pic_list_reorderingcommands for long-term references (reordering_of_pic_nums_idc == 2),WelsReorderRefList2()matched the requestedlong_term_pic_numagainst each picture'suiLongTermPicNumfield.That field is unreliable because long-term references are created via MMCO 3 (short-to-long) and MMCO 6 (mark current as long), both of which only carry
long_term_frame_idxin the bitstream and never along_term_pic_num. As a resultAddLongTermToList()storesuiLongTermPicNumas0for every such picture.With multiple long-term references present, they all compare equal (
0), so the reorder failed to select the requested entry (matching either the wrong one or none at all), leaving the wrong picture at the head of the reference list and corrupting inter prediction.It only surfaces when multiple long-term references and a long-term reorder command occur together. In practice this is hit when decoding AirPlay Mirroring streams from iOS and macOS, whose High-profile CABAC streams use multiple long-term references with reordering, producing visibly corrupted video.
Note that
WelsReorderRefList2()is only used for non-Baseline CABAC streams (uiProfileIdc != 66 && bEntropyCodingModeFlag).The problem was introduced disabled (
#if 0) by 1b8caef, and later enabled by fa6d099, which flipped it to#if 1and exposed it.Solution
Match by
iLongTermFrameIdxinstead ofuiLongTermPicNum, in both the selection loop and the subsequent compaction loop.iLongTermFrameIdxis the identifier that is actually maintained on the picture (set inAddLongTermToList) and, per the spec (8.2.4.1), for frame codingLongTermPicNum == LongTermFrameIdx, so the value parsed aslong_term_pic_nummatches it directly.This aligns the reorder path with how the rest of the code already identifies long-term pictures, e.g.
WelsDelLongFromList()used by MMCO 2, which likewise matches againstiLongTermFrameIdx.Testing
Added a focused unit test (
DecUT_RefListReorder.cpp) that reproduces the case via three long-term references sharinguiLongTermPicNum == 0with distinctiLongTermFrameIdx, plus an example High-profile CABAC regression stream (res/ltr_reorder_high_cabac.264) wired into the decoder API test.Considerations
Keeping the match on
uiLongTermPicNumbut populating it correctly, i.e. derivingLongTermPicNumfromLongTermFrameIdxper slice (8.2.4.1) or setting it at marking time, would be an alternative option. However,LongTermPicNumis a per-slice derived quantity, not a persistent property of the picture, so storing it is spec-awkward. For frame coding, it just equalsLongTermFrameIdx, making the field redundant.It would also touch the marking path (
AddLongTermToList,MarkAsLongTerm) for no functional gain.