Skip to content

Commit 5f88d99

Browse files
Edit ImageSupport.md using inpage editor
Summary: Image support updates to the vrs open source documentation, to reflect open sourcing xprs and the addition of `image/jxl` and `image/custom_codec`. NOTE: If you want to update this diff, go via the preview link inside the static docs section below. Ensure you are editing the same page that was used to create this diff. Differential Revision: D78191179 fbshipit-source-id: e68c2ee5f2efb70ae8cc63f33aa5f915aad0c0b9
1 parent 3c977b2 commit 5f88d99

1 file changed

Lines changed: 32 additions & 20 deletions

File tree

website/docs/ImageSupport.md

Lines changed: 32 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -9,14 +9,15 @@ Prerequisite: Understanding how `RecordFormat` works.
99

1010
Image content blocks can be any of the following different subtypes:
1111

12-
- `image/raw`
13-
The image is stored as a buffer of raw pixels, which might a compacted pixel format such as [RAW10](https://developer.android.com/reference/android/graphics/ImageFormat#RAW10), but not compressed using a `PixelFormat`.
14-
The exact [list of pixel formats](https://github.com/facebookresearch/vrs/blob/main/vrs/RecordFormat.h#L49-L68) grows over time.
15-
- `image/jpg` and `image/png`
16-
The image is compressed using jpeg or png compression. The payload of the image content block contains exactly what a jpg or png file would contain.
17-
- `image/video`
18-
The image is compressed according to the spec of a video codec standard, such as H.264 or H.265.
19-
_Note that the list of supported video codecs is deliberately not part of the spec, but rather, is meant to be implementation dependent, part of a VRS extension. VRS Open Source does not currently provide any implementation._
12+
- `image/raw`\
13+
The image is stored as a buffer of raw pixels, which might a compacted pixel format such as [RAW10](https://developer.android.com/reference/android/graphics/ImageFormat#RAW10), but not compressed using a `PixelFormat`.\
14+
The exact [list of pixel formats](https://github.com/facebookresearch/vrs/blob/main/vrs/RecordFormat.h#L63-L92) grows over time.
15+
- `image/png`, `image/jpg`, `image/jxl`, and `image/custom_codec`\
16+
The image is compressed using png, jpeg, jpeg-xl, or a custom codec. The payload of the image content block contains exactly what regular file of that image type would contain\
17+
_Note that when using a custom codec, you must specify a codec name, or the image content block will be reported as unsupported on read._
18+
- `image/video`\
19+
The image is compressed according to the spec of a video codec standard, such as H.264 or H.265.\
20+
_Note that the list of supported video codecs is deliberately not part of the spec, but rather, is meant to be implementation dependent, by an extension. VRS Open Source provides the vrs/xprs module to interface with ffmpeg and expose the codecs it was built with._
2021

2122
## Additional Image Properties
2223

@@ -46,9 +47,10 @@ The `DataLayout` fields are:
4647
DataPieceValue<ImageSpecType> width{kImageWidth};
4748
DataPieceValue<ImageSpecType> height{kImageHeight};
4849
DataPieceValue<ImageSpecType> stride{kImageStride};
50+
DataPieceValue<ImageSpecType> stride2{kImageStride2};
4951
DataPieceEnum<PixelFormat, ImageSpecType> pixelFormat{kImagePixelFormat};
5052

51-
// For image/video only
53+
// For image/custom_codec and image/video only
5254
DataPieceString codecName{kImageCodecName}; // required
5355
DataPieceValue<ImageSpecType> codecQuality{kImageCodecQuality}; // optional
5456
```
@@ -67,17 +69,22 @@ For `image/video` images, specify the keyframe timestamp and the keyframe index,
6769
Without the following properties, `image/raw` image content blocks can not be interpreted:
6870

6971
- Required properties: width, height, pixel format.
70-
- Optional properties: stride.
72+
- Optional properties: stride, stride2.
7173

7274
When properties are provided using the Datalayout Conventions, all additional properties must be provided by a single `DataLayout` structure in one record. For example, you must not put the pixel format in the configuration record, and then put the image dimensions in the data record. You must either put both the pixel format and the image dimensions in a configuration record's `DataLayout`, or you must put them both in a `DataLayout` just before the image content block.
7375

74-
### `image/jpg` and `image/png`
76+
### `image/png`, `image/jpg` and `image/jxl`
7577

76-
jpg and png payloads are exactly the same as jpg and png files, which are fully self-described. Therefore, when those image formats are used, properties specified using the Datalayout Conventions are ignored by VRS.
78+
png, jpg, and jxl payloads are exactly the same as png, jpg, and jxl files, which are fully self-described. Therefore, when those image formats are used, properties specified using the Datalayout Conventions are ignored by VRS.
7779

78-
### `image/video`
80+
### `image/custom_codec`
81+
82+
- Required properties: codec name.
83+
- Optional properties: width, height, pixel format, codec name, keyframe timestamp, keyframe index.
84+
85+
The only property VRS itself requires is a codec name for an image content block to be recognized. However, a particular custom codec may require additional properties, such as width, height, and pixel format. Whether a custom codec requires that information or not an implementation detail of each specific custom codec. VRS does not support any particular custom codec by default, they are meant for experimentations and special cases.
7986

80-
_Support for `image/video` images is not ready for open sourcing at this time._
87+
### `image/video`
8188

8289
Video image content blocks require additional properties to decode images:
8390

@@ -103,19 +110,24 @@ The `RecordFormatStreamPlayer::onDataLayoutRead()` callback happens **after** th
103110
The `RecordFormatStreamPlayer::onImageRead()` callback happens **before** any image data has been read. However, the provided `contentBlock` object holds all the image properties, so the following information can be found:
104111
105112
- For all types: the `contentBlock` size is known. That's the size of the buffer you will need to read the image data stored in the image content block.
106-
- For `image/raw` images: the pixel format and resolution are defined.
107-
- For `image/video` images: the pixel format, resolution, keyframe timestamp, and keyframe index are defined.
113+
- For `image/raw` images: resolution, strides, and pixel format are defined.
114+
- For `image/custom_codec` images: codec name is defined. Resolution, strides, and pixel format might be defined too, if provided in the stream using the Datalayout Conventions.
115+
- For `image/video` images: the pixel format, resolution, codec name, keyframe timestamp, and keyframe index are defined.
108116
109117
The image data itself can be read using the `RecordReader` object provided by the `CurrentRecord` object. The image is described by an `ImageContentBlockSpec` object provided by `contentBlock.image()`.
110118
111119
### `image/raw` images
112120
113-
Allocate or reuse a buffer where you can read the image data. Beware of the image’s stride, which is how it was when the image was saved.
121+
Allocate or reuse a buffer where you can read the image data. Be careful to properly handle the image’s stride and stride2. `stride` describes the stride of the first plane (the number of bytes separating the first byte of each successive lines), while `stride2` describes the stride of all the following planes, if the images is stored in a multi-plane format.
122+
123+
### `image/png`, `image/jpg`, and `image/jxl` images
124+
125+
You will need to read and decode the compressed image data using a standard implementation.
114126
115-
### `image/jpg` and `image/png` images
127+
### `image/custom_codec` images
116128
117-
You will need to read and decode the compressed image data using a standard jpg or png library.
129+
You will need to read and decode the compressed image data using your own implementation of the custom codec, which implementation is presumably unknown to VRS.
118130
119131
### `image/video` images
120132
121-
Support for `image/video` images is not currently ready for open sourcing. Stay tuned!
133+
Support for `image/video` images is provided by the vrs/xprs module, that depends on ffmpeg, and exposes the codecs built with it.

0 commit comments

Comments
 (0)