Float16Array + HTML Canvas

[palemieux]

FYI. Below is input from the Color on the Web CG regarding Float16Array in the context of HTML Canvas. This input was prompted by w3c/ColorWeb-CG#87

-- W3C Color on the Web CG chairs
https://www.w3.org/community/colorweb/
public-colorweb@w3.org

====================

CG proposal re: Float16Array + HTML Canvas

The ColorWeb CG is seeking consensus with parties interested in
discussing floating-point backing stores for canvas rendering
contexts.

The CG spec proposal is:
https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md

The primary outstanding issue at present is how to represent values
which are read back from the canvas into typed arrays which can be
manipulated from ECMAScript.

The current spec proposal supports readbacks into both
Uint8ClampedArray and Float32Array via ImageDataColorType.

It has been suggested to also add support for readbacks into
Float16Array (w3c/ColorWeb-CG#87). This
would make the specification dependent on the Float16Array type
currently being defined by the ECMAScript committee (ISO/TC 39).

The CG supports the idea of adding support for Float16Array to
ECMAScript. However, the CG feels it is most appropriate to decouple
the ongoing improvements to ECMAScript from the work on floating-point
canvases. Users of the web platform are expressing the need for
floating-point canvas backing stores now; in Chromium, implementation
is underway in crbug.com/1230619. Moreover, readback and CPU-side
manipulation paths are not high-performance, so the additional memory
bandwidth of using Float32Array for this task will not critically
impact applications.

The ColorWeb CG therefore suggests:

1. Moving ahead with
   https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md in its
   current form, with unorm8 and float32 as options for readback.

2. In parallel, working on adding Float16Array to ECMAScript, and
   prototyping it in browsers.

3. Specifying a "float16" enum in ImageDataColorType once
   implementation experience has been achieved with Float16Array, .
   Applications can feature-detect its support in browsers by catching
   exceptions raised by getImageData or createImageData.

Background

For context, introducing a Float16Array into the Typed Array hierarchy
was discussed several years ago among members of TC39, as well as the
Khronos Group where Typed Arrays originated. At the time, the costs of
supporting this type natively in ECMAScript engines outweighed the
benefits for the following reasons:

1. For practical purposes, Float16Array could be sufficiently
   polyfilled in ECMAScript:

https://github.com/petamoriken/float16

Corner cases existed regarding denormalized values, NaNs, and
infinities - but applications could generally achieve their desired
results, with good performance on all ECMAScript engines, using
existing primitives.

2. Direct CPU support for half-float numbers did not seem to exist. At
   the time, C libraries which worked with this data type universally
   emulated it, rather than using compiler intrinsics on certain
   platforms.

3. A significant amount of work would be required in every ECMAScript
   engine to make this typed array type perform well.

4. A significant amount of work would be required to specify the
   behavior of this numeric type in ECMAScript.

The landscape has changed in recent years. Per
https://en.wikipedia.org/wiki/Half-precision_floating-point_format#Hardware_support,
ARM and x86 processors either support FP16 natively, or will in the
near future. FP16 formats are increasingly heavily used in neural
network evaluation and image processing, including on the web. The
ECMAScript editors may be able to save spec work by referencing IEEE
specifications (which, to be fair, already existed when Float16Array
was originally considered).

[annevk]

I'm worried that this removes the incentive to get the work done on Float16Array.

It also ends up cementing the wrong default:

If the CanvasImageData has a colorType of "float16", then the ImageData's colorType will be "float32"

We don't want float32 there in the future.

[annevk]

cc @whatwg/canvas

[palemieux]

@annevk Can you confirm that your concern is specifically with selecting Float32Array as the default readback type? If so, we can use Uint8ClampedArray as the default readback type instead.

We are also happy to support the Float16Array proposal in TC 39. Let us know how to best coordinate the effort/proceed.

In the meantime, are there objections to supporting Float32Array as one of the (non-default) readback types and adding Float16Array once the proposal makes its way through the TC 39 process?

[annevk]

What are the use cases for Float32Array? And would it not be weird if the canvas is float16 and you get uint8 data out even if Float16Array is a thing?

[palemieux]

The use case for Float32Array that is of concern to the Color CG is the manipulation of HDR pixels, for which UInt8 is not sufficient.

The idea is for getImageData() to continue returning UInt8 data by default, even if the contents of the Canvas is higher bit-depth. An application could request higher bit depths, e.g., Float16Array , by passing a configuration parameter to getImageData().

[annevk]

Right, but Float16Array meets that need as well and shouldn't be difficult to add to the platform. And would allow for more useful defaults as well.

[palemieux]

Please provide a concrete proposal and the steps needed to get there, and the CG can consider it.

[annevk]

1. The Color CG or one of its participants follows through on Please create Stage 1 float16 repo tc39/proposals#453 to get Float16Array in ECMAScript/JavaScript done.
2. We add Float16Array to Web IDL.
3. We incorporate it in this API (and others).

[palemieux]

@annevk Are you ready to be a proponent to the work in TC 39, and are ready to implement the feature?

[bakkot]

I've made a repo to track the proposal in TC39. I'm not very familiar with the space, so while I'd be willing to champion the proposal in TC39 it would be helpful if someone from ColorWebCG could lay out why it would be helpful to have this (and, specifically, why Uint16Array and Float32Array aren't sufficient). The best thing would be if you could open an issue on that repository with your use case, but if you respond here or point me to a summary I can try to copy that into an issue myself.

[palemieux]

@bakkot Thanks for getting this started. I am happy to coordinate input from the Color CG. In the meantime:

• Uint16Array is sufficient for high-dynamic range (HDR) pixels with non-linear color values, e.g. PQ and HLG pixels.
• Uint16Array is not sufficient for high-dynamic range (HDR) pixels with linear color values. Float16 or Float32 are needed in that case.

Do you think the explainer should include background on HDR and WCG imagery?