Add cropping metadata #346

mattrwoz · 2024-03-19T20:55:33Z

This adds cropping metadata as proposed in CWG-D038. Accepted in the 10/10/2023 CWG meeting.
Most of the text is the same as the proposal document with some minor adjustments:

Replaced "valid picture data" with "rendered"
Clarified the scope of the metadata to be from "metadata_crop OBU to Keyframe and/or next metadata_crop OBU" instead of just "until next metadata_crop OBU"

podborski · 2024-04-01T17:09:40Z

06.bitstream.syntax.md

+| metadata_crop( ) {                                        | **Type**
+|     @@crop_width_minus_1                                  | f(16)
+|     @@crop_height_minus_1                                 | f(16)
+|     @@crop_offset_present                                 | f(1)


Why don't we make it byte-aligned in the first place? having something like:

f(7) reserved f(1) crop_offset_present;

would be better IMO than relying on the trailing_bits at the end of the OBU.

07.bitstream.semantics.md

podborski · 2024-04-01T17:23:26Z

07.bitstream.semantics.md

+#### Metadata crop semantics
+
+When present the metadata_crop OBU applies starting at the next frame in the sequence 
+with matching temporal_id and spatial_id, and shall apply to all matching frames until the next Key Frame


When present the metadata_crop OBU applies starting at the next frame in the sequence
with matching temporal_id and spatial_id, ...

Alexis and myself are wondering here what the behavior should be if no extension header is present. Perhaps in that case it should apply to all layers? Then the language on the persistence should also be clarified.

Also instead of:

and shall apply to all matching frames

it would be better to write "and shall apply to all frames with a matching temporal_id and spatial_id"

Yes. I think it should apply to all layers if no header is present. I will add wording clarifying that.

podborski · 2024-04-01T17:37:17Z

07.bitstream.semantics.md

+When present the metadata_crop OBU applies starting at the next frame in the sequence 
+with matching temporal_id and spatial_id, and shall apply to all matching frames until the next Key Frame
+or metadata_crop OBU. The output picture should be cropped to the region as specified in the 
+cropping metadata OBU. When applied, the crop shall be after all normal decode operations as a 


When applied, the crop shall be after all normal decode operations as a post-processing step (after film grain synthesis in the output process). This metadata has no effect on the decoding process.

Maybe it would be better to re-write like this:

The crop shall be applied after all normal decode operations as a post-processing step. This metadata information has no effect on the decoding process.

Used your new wording.

podborski · 2024-04-01T17:41:36Z

07.bitstream.semantics.md

+
+**crop_y_offset** specifies the minimum pixel row containing picture data which should be rendered.
+
+**crop_width_minus_1** specifies the number of pixel columns minus 1 which which should be rendered after applying crop_x_offset.


which which => which

podborski · 2024-04-01T18:25:07Z

07.bitstream.semantics.md

+When muxed into a container that supports signaling cropping information, this metadata should 
+be removed from the bitstream and included in the container’s signaling mechanism.  
+If both the container and bitstream signal cropping information, then the container’s cropping 
+information takes precedence.


Instead of:

When muxed into a container that supports signaling cropping information, this metadata should
be removed from the bitstream and included in the container’s signaling mechanism.
If both the container and bitstream signal cropping information, then the container’s cropping
information takes precedence.

We could write:

In cases where cropping information is present in both the bitstream and the delivery or container format, the latter should be preferred.

NOTE: Container or delivery formats that package the AV1 bitstream are recommended to address this redundancy, potentially by excluding bitstream cropping information when it is already available in the container or delivery format.

Then we will have to update the AV1 ISOBMFF spec and say there that we use the clap box and remove this metadata OBU. Thoughts @cconcolato ?

Updated to new wording.

podborski · 2024-06-25T15:08:15Z

Some relevant discussion in the file format group on how such metadata can co-exist MPEGGroup/FileFormat#77

Unlike H264/H265, AV1 contains no fields to crop encoded output to specific sizes. AMD's hardware cannot handle encoding of unaligned dimensions for AV1, hence it codes 1920x1080 as 1920x1088. Add side data to crop the output back to the original dimensions. There's an AV1-spec extension planned to fix this here: AOMediaCodec/av1-spec#346 But it seems to have stuck for now.

GrabbenD · 2025-01-22T09:09:29Z

This spec would fix black padding with av1 encoded media on AMDGPU

https://gitlab.freedesktop.org/mesa/mesa/-/issues/9185

imoccaga · 2025-02-21T23:36:50Z

This adds cropping metadata as proposed in CWG-D038. Accepted in the 10/10/2023 CWG meeting.
Most of the text is the same as the proposal document with some minor adjustments:

I suggest to update CWG-D038 or submit a new doc with the adjustments and discuss it in CWG. Changes should be made after CWG has reviewed and approved the adjustments.

aklemets · 2025-02-26T18:37:14Z

This adds cropping metadata as proposed in CWG-D038. Accepted in the 10/10/2023 CWG meeting.
Most of the text is the same as the proposal document with some minor adjustments:

I suggest to update CWG-D038 or submit a new doc with the adjustments and discuss it in CWG. Changes should be made after CWG has reviewed and approved the adjustments.

Iole, is there a particular change that you think justifies bringing this already approved change back to CWG?

I have diffed the approved document, "CWG-D038_CroppingMetadata_V4.docx" against the text that was submitted in the initial PR commit, and I see no technical differences, but only some grammatical changes.
(The description that was in this initial commit to the PR that claimed differences between the PR and the approved document, appears to be inaccurate.)

imoccaga · 2025-03-01T13:44:29Z

Thanks for double checking.

And, I see in https://github.com/AOMediaCodec/av1-spec/pull/346/files#diff-f215e95f9dd8408e553c3865e9cf72448db6bc62e4ffc27c21f5f74b73edf252
it has been clarified that
"This metadata information has no effect on the decoding process."

So, I have no concern on the current text.

Sadarex · 2025-03-25T14:45:31Z

Any news on this?

Some other projects are still blocked:
https://patchwork.ffmpeg.org/project/ffmpeg/patch/[email protected]/
https://gitlab.freedesktop.org/mesa/mesa/-/issues/9185

Disable cropping to render resolution by default. This functionality was added in https://webrtc-review.googlesource.com/c/src/+/381340 and https://webrtc-review.googlesource.com/c/src/+/382241 and was enabled by default to make things work with HW encoders which use render resolution to signal cropping. After discussions with AV1 folks the consensus is that encoders should not use render resolution to signal crop. Currently there is no way to signal crop in AV1 bitstream. There is a proposal for adding this info to AV1 metadata: AOMediaCodec/av1-spec#346. Bug: webrtc:405341160, b:402019802, b:405299912 Change-Id: I9cff9f2f8e7c9b8933fd0fbb100ce37221eaf084 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/382640 Reviewed-by: Danil Chapovalov <[email protected]> Commit-Queue: Sergey Silkin <[email protected]> Cr-Commit-Position: refs/heads/main@{#44228}

Upstream commit: https://webrtc.googlesource.com/src/+/ecde63eab5b643e9d84220b8a50208b4cd8ad866 Do not crop to render resolution by default Disable cropping to render resolution by default. This functionality was added in https://webrtc-review.googlesource.com/c/src/+/381340 and https://webrtc-review.googlesource.com/c/src/+/382241 and was enabled by default to make things work with HW encoders which use render resolution to signal cropping. After discussions with AV1 folks the consensus is that encoders should not use render resolution to signal crop. Currently there is no way to signal crop in AV1 bitstream. There is a proposal for adding this info to AV1 metadata: AOMediaCodec/av1-spec#346. Bug: webrtc:405341160, b:402019802, b:405299912 Change-Id: I9cff9f2f8e7c9b8933fd0fbb100ce37221eaf084 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/382640 Reviewed-by: Danil Chapovalov <[email protected]> Commit-Queue: Sergey Silkin <[email protected]> Cr-Commit-Position: refs/heads/main@{#44228}

Upstream commit: https://webrtc.googlesource.com/src/+/ecde63eab5b643e9d84220b8a50208b4cd8ad866 Do not crop to render resolution by default Disable cropping to render resolution by default. This functionality was added in https://webrtc-review.googlesource.com/c/src/+/381340 and https://webrtc-review.googlesource.com/c/src/+/382241 and was enabled by default to make things work with HW encoders which use render resolution to signal cropping. After discussions with AV1 folks the consensus is that encoders should not use render resolution to signal crop. Currently there is no way to signal crop in AV1 bitstream. There is a proposal for adding this info to AV1 metadata: AOMediaCodec/av1-spec#346. Bug: webrtc:405341160, b:402019802, b:405299912 Change-Id: I9cff9f2f8e7c9b8933fd0fbb100ce37221eaf084 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/382640 Reviewed-by: Danil Chapovalov <danilchapwebrtc.org> Commit-Queue: Sergey Silkin <ssilkinwebrtc.org> Cr-Commit-Position: refs/heads/main{#44228} UltraBlame original commit: 03d8ebe4ce8f1794a65113a7d3b05147e10be5a7

taste1981 · 2025-08-05T04:51:27Z

@mattrwoz , any update on this spec change proposal?

mattrwoz added 3 commits March 18, 2024 16:46

Define new metadata for cropping

3b33d6f

Re-scope to key frames

2ed153a

clarify both key and additional metadata

5ac8502

podborski reviewed Apr 1, 2024

View reviewed changes

07.bitstream.semantics.md Outdated Show resolved Hide resolved

podborski reviewed Apr 1, 2024

View reviewed changes

Feedback fixes

bb04abd

Rongronggg9 mentioned this pull request Aug 26, 2025

Streamed resolution doesn't match set resolution in moonlight (blurryness) moonlight-stream/moonlight-qt#1466

Open


		crop_y_offset specifies the minimum pixel row containing picture data which should be rendered.

		crop_width_minus_1 specifies the number of pixel columns minus 1 which which should be rendered after applying crop_x_offset.

Add cropping metadata #346

Are you sure you want to change the base?

Add cropping metadata #346

Uh oh!

Conversation

mattrwoz commented Mar 19, 2024

Uh oh!

podborski Apr 1, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

podborski Apr 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattrwoz Apr 2, 2024

Choose a reason for hiding this comment

Uh oh!

podborski Apr 1, 2024

Choose a reason for hiding this comment

Uh oh!

mattrwoz Apr 8, 2024

Choose a reason for hiding this comment

Uh oh!

podborski Apr 1, 2024

Choose a reason for hiding this comment

Uh oh!

mattrwoz Apr 8, 2024

Choose a reason for hiding this comment

Uh oh!

podborski Apr 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattrwoz Apr 8, 2024

Choose a reason for hiding this comment

Uh oh!

podborski commented Jun 25, 2024

Uh oh!

GrabbenD commented Jan 22, 2025

Uh oh!

imoccaga commented Feb 21, 2025

Uh oh!

aklemets commented Feb 26, 2025

Uh oh!

imoccaga commented Mar 1, 2025

Uh oh!

Sadarex commented Mar 25, 2025

Uh oh!

taste1981 commented Aug 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

podborski Apr 1, 2024 •

edited

Loading

podborski Apr 1, 2024 •

edited

Loading