Fix a corner case in DPB management logic.

Additional details and rationale are documented directly in the code comments.

Author: krajunv

vk_video_encoder/libs/VkVideoEncoder/VkEncoderDpbH264.cpp

@@ -318,6 +318,56 @@ int8_t VkEncDpbH264::DpbPictureEnd(const PicInfoH264 *pPicInfo,
                     if (i < MAX_DPB_SLOTS) {
                         DpbBumping(false);
                     } else {
+                        // If we reach this point, it means, DPB is full and no slot is available for the current picture.
+                        // Vulkan video encoding requires a DPB slot for the current picture whenever VkVideoEncodeFrameInfo::pSetupReferenceSlot
+                        // is being set. This application always populates this field, and without a valid DPB slot the API cannot be
+                        // configured correctly.
+
+                        // Corner case: DpbBumping() attempts to free a DPB slot when certain conditions are met.
+                        // One of the condition checked is whether the picture is marked as "needed for output".
+                        // As part of this process, it first outputs the picture by marking the it as "not needed for output".
+                        // Before actually clearing the slot, it checks whether the picture is "unused for reference".
+                        // If so, the slot is emptied. However, if the picture is still "used for reference", the function
+                        // only marks the picture as "not needed for output" and stops there.
+                        //
+                        // The issue arises when a picture is still "used for reference" at the moment DpbBumping() process, but later
+                        // during subsequent DPB management, it becomes "unused for reference". Because the picture was already
+                        // marked as "not needed for output" at the time last DpbBumping() process, it will no longer be considered by
+                        // the bumping logic when the next DpbBumping() process is called. This leaves the slot stuck in the DPB
+                        // even though it is no longer needed for output and not needed for reference.
+                        //
+                        // To avoid leaving such stale entries in the DPB, perform an explicit check for any slot that is
+                        // both "not needed for output" and "unused for reference", and remove it when encountered.
+                        //
+                        //
+                        // Because there is no reference‑counting tied to the "needed for output"/"used for reference"
+                        // and "not needed for output"/"unused for reference", simply marking a picture as "unused for reference"
+                        // does not immediately make its dpbImageView safe to reuse. Pictures still in the encode queue may
+                        // reference it, so releasing the dpbImageView too early can lead to invalid references.
+                        // Moving this cleanup logic into DpbBumping() can therefore cause problems if the picture is still
+                        // referenced at that moment. To avoid such issues, execute this logic only when it is clear that no empty
+                        // DPB slot can be found for the current picture.
+                        //
+                        // TODO: As an enhacement,
+                        //  1. the dpb slot with lowest PicOrderCnt value can be emptied first.
+                        //  2. tie the reference count to the markings "needed for output"/"used for reference"
+                        //     and "not needed for output"/"unused for reference" for cleaner implementation.
+                        int32_t i = 0;
+                        for (i = 0; i < MAX_DPB_SLOTS; i++) {
+                            if ((m_DPB[i].state & DPB_TOP) && !m_DPB[i].top_needed_for_output && (m_DPB[i].top_field_marking == MARKING_UNUSED) &&
+                                (m_DPB[i].state & DPB_BOTTOM) && !m_DPB[i].bottom_needed_for_output && (m_DPB[i].bottom_field_marking == MARKING_UNUSED)) {
+
+                                m_DPB[i].state = DPB_EMPTY;
+                                ReleaseFrame(m_DPB[i].dpbImageView); // release the image resource
+                                break;
+                            }
+                        }
+                        if (i >= MAX_DPB_SLOTS) {
+                            // Reaching this point indicates an error in the DPB management logic.
+                            assert(!"Failed to find a free DPB slot");
+                            break; // exit while (1)
+                        }
+#if 0
                         // DPB is full, current has lowest value of PicOrderCnt
                         if (!pPicInfo->field_pic_flag) {
                             // frame: output current picture immediately
@@ -334,6 +384,7 @@ int8_t VkEncDpbH264::DpbPictureEnd(const PicInfoH264 *pPicInfo,
                         }
 
                         break;  // exit while (1)
+#endif
                     }
                 } else {
                     for (m_currDpbIdx = 0; m_currDpbIdx < MAX_DPB_SLOTS; m_currDpbIdx++) {