Expand native NVDEC decoding coverage - #1635
Merged
Merged
Conversation
NVDEC can output planar 4:4:4 surfaces (YUV444 and YUV444_16Bit), but we only ever asked it for NV12 or P016, so every 4:4:4 stream fell back to CPU decoding. Pick the surface from the stream's chroma subsampling as well as the output dtype, add the CUDA 4:4:4 -> RGB kernels the new surfaces need, and lay the extra chroma plane out in the decoded AVFrame. Two knock-on fixes: - is_cpu_fallback() inferred CPU-ness from the pixel format, which no longer works now that yuv444p can come from either NVDEC or the CPU fallback. Frames record it explicitly instead. - get_pre_allocation_dtype() assumed any non-NV12 surface was 16-bit, which the 8-bit YUV444 surface breaks. The pre-allocated buffer's dtype follows the surface; the requested dtype is applied afterwards. Covered by new HEVC 4:4:4 assets at 8, 10 and 12 bits, through both VideoDecoder and the Blocks APIs.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1635
Note: Links to docs will display an error until the docs builds have been completed. ❌ 4 New Failures, 5 Cancelled Jobs, 4 PendingAs of commit c51e5a1 with merge base 1ebfad0 ( NEW FAILURES - The following jobs have failed:
CANCELLED JOBS - The following jobs were cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This reverts commit b2189af.
NVDEC offers only a P016 output surface for AV1 4:2:0 10-bit, no NV12. Asking for uint8 therefore found no 8-bit surface and fell back to the CPU, while asking for float32 decoded natively -- an odd split, given uint8 is the default. The widening branch added for 4:4:4 covers this too; this just adds the asset and the coverage.
NicolasHug
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NVDEC can output planar 4:4:4 surfaces (YUV444 and YUV444_16Bit), but we only ever asked it for NV12 or P016, so every 4:4:4 stream fell back to CPU decoding. Additionally some codecs like AV1 only support a 16bit surface for >8 bit sources, and we were enforcing NV12 surface (causing a fallback) - now we allow P016.
This required creating new CUDA color-conversion kernels for 444 -> RGB.
Some weird stuff was also needed to figure out whether a frame is on the fallback path or not, but that'll be removed in #1636
Stack created with GitHub Stacks CLI • Give Feedback 💬