You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Summary:
Image support updates to the vrs open source documentation, to reflect open sourcing xprs and the addition of `image/jxl` and `image/custom_codec`.
NOTE: If you want to update this diff, go via the preview link inside the static docs section below.
Ensure you are editing the same page that was used to create this diff.
Differential Revision: D78191179
fbshipit-source-id: e68c2ee5f2efb70ae8cc63f33aa5f915aad0c0b9
Copy file name to clipboardExpand all lines: website/docs/ImageSupport.md
+32-20Lines changed: 32 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,14 +9,15 @@ Prerequisite: Understanding how `RecordFormat` works.
9
9
10
10
Image content blocks can be any of the following different subtypes:
11
11
12
-
-`image/raw`
13
-
The image is stored as a buffer of raw pixels, which might a compacted pixel format such as [RAW10](https://developer.android.com/reference/android/graphics/ImageFormat#RAW10), but not compressed using a `PixelFormat`.
14
-
The exact [list of pixel formats](https://github.com/facebookresearch/vrs/blob/main/vrs/RecordFormat.h#L49-L68) grows over time.
15
-
-`image/jpg` and `image/png`
16
-
The image is compressed using jpeg or png compression. The payload of the image content block contains exactly what a jpg or png file would contain.
17
-
-`image/video`
18
-
The image is compressed according to the spec of a video codec standard, such as H.264 or H.265.
19
-
_Note that the list of supported video codecs is deliberately not part of the spec, but rather, is meant to be implementation dependent, part of a VRS extension. VRS Open Source does not currently provide any implementation._
12
+
-`image/raw`\
13
+
The image is stored as a buffer of raw pixels, which might a compacted pixel format such as [RAW10](https://developer.android.com/reference/android/graphics/ImageFormat#RAW10), but not compressed using a `PixelFormat`.\
14
+
The exact [list of pixel formats](https://github.com/facebookresearch/vrs/blob/main/vrs/RecordFormat.h#L63-L92) grows over time.
15
+
-`image/png`, `image/jpg`, `image/jxl`, and `image/custom_codec`\
16
+
The image is compressed using png, jpeg, jpeg-xl, or a custom codec. The payload of the image content block contains exactly what regular file of that image type would contain\
17
+
_Note that when using a custom codec, you must specify a codec name, or the image content block will be reported as unsupported on read._
18
+
-`image/video`\
19
+
The image is compressed according to the spec of a video codec standard, such as H.264 or H.265.\
20
+
_Note that the list of supported video codecs is deliberately not part of the spec, but rather, is meant to be implementation dependent, by an extension. VRS Open Source provides the vrs/xprs module to interface with ffmpeg and expose the codecs it was built with._
When properties are provided using the Datalayout Conventions, all additional properties must be provided by a single `DataLayout` structure in one record. For example, you must not put the pixel format in the configuration record, and then put the image dimensions in the data record. You must either put both the pixel format and the image dimensions in a configuration record's `DataLayout`, or you must put them both in a `DataLayout` just before the image content block.
73
75
74
-
### `image/jpg` and `image/png`
76
+
### `image/png`, `image/jpg` and `image/jxl`
75
77
76
-
jpg and png payloads are exactly the same as jpg and png files, which are fully self-described. Therefore, when those image formats are used, properties specified using the Datalayout Conventions are ignored by VRS.
78
+
png, jpg, and jxl payloads are exactly the same as png, jpg, and jxl files, which are fully self-described. Therefore, when those image formats are used, properties specified using the Datalayout Conventions are ignored by VRS.
The only property VRS itself requires is a codec name for an image content block to be recognized. However, a particular custom codec may require additional properties, such as width, height, and pixel format. Whether a custom codec requires that information or not an implementation detail of each specific custom codec. VRS does not support any particular custom codec by default, they are meant for experimentations and special cases.
79
86
80
-
_Support for `image/video` images is not ready for open sourcing at this time._
87
+
### `image/video`
81
88
82
89
Video image content blocks require additional properties to decode images:
83
90
@@ -103,19 +110,24 @@ The `RecordFormatStreamPlayer::onDataLayoutRead()` callback happens **after** th
103
110
The `RecordFormatStreamPlayer::onImageRead()` callback happens **before** any image data has been read. However, the provided `contentBlock` object holds all the image properties, so the following information can be found:
104
111
105
112
- For all types: the `contentBlock` size is known. That's the size of the buffer you will need to read the image data stored in the image content block.
106
-
- For `image/raw` images: the pixel format and resolution are defined.
107
-
- For `image/video` images: the pixel format, resolution, keyframe timestamp, and keyframe index are defined.
113
+
- For `image/raw` images: resolution, strides, and pixel format are defined.
114
+
- For `image/custom_codec` images: codec name is defined. Resolution, strides, and pixel format might be defined too, if provided in the stream using the Datalayout Conventions.
115
+
- For `image/video` images: the pixel format, resolution, codec name, keyframe timestamp, and keyframe index are defined.
108
116
109
117
The image data itself can be read using the `RecordReader` object provided by the `CurrentRecord` object. The image is described by an `ImageContentBlockSpec` object provided by `contentBlock.image()`.
110
118
111
119
### `image/raw` images
112
120
113
-
Allocate or reuse a buffer where you can read the image data. Beware of the image’s stride, which is how it was when the image was saved.
121
+
Allocate or reuse a buffer where you can read the image data. Be careful to properly handle the image’s stride and stride2. `stride` describes the stride of the first plane (the number of bytes separating the first byte of each successive lines), while `stride2` describes the stride of all the following planes, if the images is stored in a multi-plane format.
122
+
123
+
### `image/png`, `image/jpg`, and `image/jxl` images
124
+
125
+
You will need to read and decode the compressed image data using a standard implementation.
114
126
115
-
### `image/jpg` and `image/png` images
127
+
### `image/custom_codec` images
116
128
117
-
You will need to read and decode the compressed image data using a standard jpg or png library.
129
+
You will need to read and decode the compressed image data using your own implementation of the custom codec, which implementation is presumably unknown to VRS.
118
130
119
131
### `image/video` images
120
132
121
-
Support for `image/video` images is not currently ready for open sourcing. Stay tuned!
133
+
Support for `image/video` images is provided by the vrs/xprs module, that depends on ffmpeg, and exposes the codecs built with it.
0 commit comments