Add ability to signal alpha channel type?

[haasn]

In the wild, there is a significant disagreement about whether images containing alpha channels should be "premultiplied" or "straight / independent" alpha. Some codes specify this as part of their specifications (notably, PNG); while some others have tagging for it (e.g. JPEG-XL). This can cause intense headaches when transcoding between formats, especially when using raw or lossless intermediate formats that generally don't tag/preserve the alpha mode. It would be nice to have ability to tag the correct alpha mode on the container level in Matroska.

I saw that there already exists an "AlphaMode" tag, but that it is advised in practice to avoid using undefined values due to differing behavior between firefox and chromium.

Do you think we could extend this defininition to introduce the value of 2 = Premultiplied alpha, and retroactively redefine 1 = Straight / independent alpha? In implementations that ignore the values > 1, it will be treated as implicitly blended to black. So the main problem comes with browsers that would then start interpreting these (premultiplied) files as straight alpha, leading to incorrect compositing.

Or maybe it's better to introduce some sort of new tag?

[robUx4]

The AlphaMode is basically a boolean to tell the content has more than the regular 3 components. This is useful to classify files with or without alpha or prepare special handling. And that this alpha data is part of the BlockAdditional data. A codec ID that would always have alpha inside shouldn't use this tag as it doesn't use BlockAdditional for that.

At the container level we don't know what is inside the codec. There are chroma subsampling information and things like that, but that's mostly necessary for raw formats as all compressed codec usually have this information in the bitstream. This is the same thing with BlockAdditional data. They are to be interpreted at the codec level, not the container level.

In the only I know with alpha (VP8 and VP9) we don't know what's inside the alpha bitstream (for VP9 it seems to be a 1 channel bitstream, for VP8 it's less clear). But that information is only known by the decoder, not the container. Just like we don't know the chroma of the pixels in those bitstreams (for VP9 you'd have to parse the CodecPrivate). We don't know anything about the alpha data. But you parse at the codec level, you know exactly what you have and it's the same for the alpha bitstream (it's an independent channel). For VPx that's straight alpha. If you start using premultiplied existing browsers won't be able to use it.

If the goal is to add premultiplied alpha for VPx (or AV1) it would be easier to use another BlockAddIDType to define another form of alpha data. It won't be misinterpreted by older browsers and that's the preferred way to use BlockAdditional data (there's Dolby Vision in there for example). You also have the \Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDExtraData element which is similar to the CodecPrivate for BlockAdditional. You could put a flag in there to define the type of alpha channel used.

[haasn]

I would specifically be interested in adding the ability to store premultiplied files as AV1 or VP9 inside mkv.

What do you think is cleaner; adding a new BlockAddIDType or adding a BlockAddIDExtraData flag? As I understand it, only the former will play well with existing implementations, which would then silently ignore the alpha channel (for premultiplied alpha, that gives a well-defined result - implicit blend to black).