Piping element transforms into element.style.transform is problematic

[fantasai]

I want to make several arguments against the drawElementImage() API design, which expects the author to capture the returned transformation matrix and pipe it into the element's transform property via the style attribute in order to transform the element's input coordinates.

const transform = ctx.drawElementImage(drawableElement, 100, 0);
drawableElement.style.transform = transform.toString();

1. Using the style attribute and transform property as a core part of this drawElementImage() feature, which has nothing to do with CSS or its transforms feature, is awkward and weird and hacky.
2. Asking authors to reliably follow this non-obvious pattern of capturing and setting the transform manually, which they can reasonably forget, skip, or get wrong, is not great API design. The drawElementImage() API should pipe the transform to the element automatically, unless instructed not to. The "right thing to do" should happen by default.
3. Piping a transform into the element by stringifying it and then parsing it back in and running style recalculation is not performant. That is so much wasted computation.
4. The transform property can only handle linear transforms, but this is not enough to address many of the use cases for canvas-drawn elements, which will often need non-linear transformations. Consider, for example, wrapping text around a sphere. The characters at the edges are compressed closer together while the characters in the middle are stretched apart. If you try to insert a cursor 25% of the way into the text visually, that will be more than 25% of the way into the text in terms of caret positioning. Many of the visual distortions proposed as use cases for this feature are even more complex.

---

To address these problems I propose the following (as a starting point for discussion):

• Introduce a dedicated Element.inputTransform property, where the InputTransform object represents the transformation of the element for the purpose of capturing its input on the <canvas>. This avoids messing with Element.style.
• Give the InputTransform object the following mutable properties:
  • .transform - accepts a DOMMatrix (the drawElementImage() return value) directly, avoiding the stringification issue.
  • .matchDrawElementImage - boolean that says whether drawElementImage() calls should update .transform automatically. Defaults to true (common case), but allows the author to easily and persistently opt out for certain elements.
  • .map - accepts a function that maps input coordinates received by the element's <canvas> rectangle (after undoing .transform) to the input that the element should receive in its own, untransformed coordinate space. To re-use the sphere example, it might map y=50% x=25% (clicked position relative to the left edge of the image of the sphere) to y=50% x=38% (clicked position relative to the left edge of the element as originally laid out for drawing).
• and the following methods:
  • .set(DOMMatrix) - sets the transformation explicitly, unsetting .map and setting .matchDrawElementImage to false.
  • .set(string) - like .set(DOMMatrix), but using CSS transform syntax.
  • .set(function) - sets .map, setting .transform to identity and .matchDrawElementImage to false.
• Give drawElementImage() an updateTransform argument, defaulting to auto (check matchDrawElementImage), and allowing true and false. Or maybe a skipTransformUpdate argument, defaulting to false (check matchDrawElementImage) and allowing true (allowing to draw e.g. a reflection without updating the element's position).

This gives authors a much more flexible and powerful API, one that can address common cases easily while also handling complex cases straightforwardly. It also avoids unnecessary stringification and style calculation work, and by internalizing the most common case (keeping things in sync), it allows the UA more opportunities for optimization.

[progers]

The synchronized transform will affect an element's hit testing, accessibility, getBoundingClientRect(), intersection observer, and similar APIs, so we may want to choose a fairly general name that isn't specific to input. The breadth of affected APIs makes it difficult to support general transformations, such as the sphere case, so we may want to start with the transformations available to CSS as a starting point, which would still allow for more general hit testing to be layered on in the future (particularly if hit testing is controlled by the canvas). This could be done by adding a new transform to Element (exposed with element.getCanvasTransform). Like scrollTop, changes to this transform would not invalidate style or layout.

So that authors can still use CSS transforms, an Element's canvas transform should apply before CSS transforms. In the future, we can add an API like https://www.w3.org/TR/cssom-view-1/#the-geometryutils-interface to make it easier to use CSS transforms. Note that an element's canvas transform would not affect the rendering of the element, so updating it would not cause the canvas paint event to fire.

If we are to use the drawable idea in #134 (comment), where drawable elements are hit test by the canvas, we can solve #94 along with this issue by having the drawing APIs modify the hit test order so that the most recently drawn boxes are hit test first. Note that an element's hit test order would not affect the rendering of the element, so updating it would not cause the canvas paint event to fire.

To improve ergonomics, a new, optional, DrawElementOptions dictionary can be added to the Element drawing APIs (drawElementImage, etc). This can have a true-by-default boolean for updating the Element's canvas transform and hit test order.

On a worker thread, after each drawing API call (drawElementImage, etc), the drawn ElementImage is added to an internal pending notification queue and a microtask is queued to notify the main thread. When the microtask runs, if the pending notification queue is not empty, a task is posted to the main thread with all of the drawn ElementImages. When this task runs on the main thread, the associated elements synchronously update their canvas transforms, the canvas synchronously re-orders the CSS box hit test order, and a canvas.onelementsdrawnonworker event is fired synchronously with a list of Elements that were drawn (if a drawn Element no longer exists when the event fires, it is omitted from the list). This batching approach, as opposed to issuing a stand-alone update for each drawElementImage call, is similar to existing “observer” API’s, and ensures that:
Updates are applied atomically, so no tasks can run between updates.
Developers will know that all updates have been received; they never have to worry whether any given update is that last one in a sequence.

With these changes, maybe something like the following would work?

partial interface Element {
  DOMMatrixReadOnly getCanvasTransform();
};

dictionary DrawElementOptions {
  // The current transform from the canvas grid origin to where the element will
  // be drawn. Used for updating the Element's canvas transform, but does not
  // affect rendering. Defaults to the canvas CTM for 2D, and the identity matrix for
  // WebGL/WebGPU. This is primarily used by WebGL/WebGPU. An error is thrown
  // if a non-null currentTransform is specified and updateGeometry is false.
  DOMMatrixInit currentTransform;

  // True if the Element's canvas transform and hit test order should be
  // automatically updated.
  boolean updateGeometry = true;
};

// 2D
interface mixin CanvasDrawElementImage {
  DOMMatrix drawElementImage((Element or ElementImage) element,
                        unrestricted double dx, unrestricted double dy,
                        optional DrawElementOptions options = {});
  // Other drawElementImage overloads are similarly modified to have an
  // optional DrawElementOptions.
}
CanvasRenderingContext2D includes CanvasDrawElementImage;
OffscreenCanvasRenderingContext2D includes CanvasDrawElementImage;

// 3D
WebGLCopyElementImageConfig could be changed to inherit from DrawElementOptions
GPUCopyElementImageDestination could be changed to inherit from DrawElementOptions

dictionary ElementsDrawnOnWorkerEventInit : EventInit {
  sequence<Element> elements = [];
};
[Exposed=Window]
interface ElementsDrawnOnWorkerEvent : Event {
  constructor(DOMString type, optional ElementsDrawnOnWorkerEventInit eventInitDict);
  readonly attribute FrozenArray<Element> elements;
};
partial interface HTMLCanvasElement {
  attribute EventHandler onelementsdrawnonworker;
};

Example of API usage:

<canvas id="canvas" style="width: 400px; height: 400px;" layoutsubtree>
  <form id="form">
    <label for="name">name:</label>
    <input id="name">
  </form>
  <div id="menu">
    <div id="option_a">Option A</div>
    <div id="option_b">Option B</div>
  </div>
</canvas>

<script>
  canvas.onpaint = () => {
    const ctx = canvas.getContext('2d');
    ctx.reset();
    ctx.drawElementImage(form, 100, 0);
    ctx.drawElementImage(menu, 0, 0);

    // form.getCanvasTransform() would be a DOMMatrix with X translation of 100
    // menu.getCanvasTransform() would be an identity DOMMatrix
    // form.getBoundingClientRect() would return a DOMRect at 100,0
    // menu.getBoundingClientRect() would return a DOMRect at 0,0
  };
</script>

Offscreen example:

<!DOCTYPE html>
<canvas id="canvas" style="width: 400px; height: 400px;" layoutsubtree>
  <form id="form">
    <label for="name">name:</label>
    <input id="name">
  </form>
  <div id="menu">
    <div id="option_a">Option A</div>
    <div id="option_b">Option B</div>
  </div>
</canvas>

<script>
  const workerCode = `
    let ctx;
    self.onmessage = (e) => {
      if (e.data.canvas) {
        ctx = e.data.canvas.getContext('2d');
      }
      
      // Draw and sync.
      if (e.data.form && e.data.menu) {
        ctx.reset();        
        ctx.drawElementImage(e.data.form, 100, 0);
        ctx.drawElementImage(e.data.menu, 0, 0);
      }
    };
  `;

  const worker = new Worker(URL.createObjectURL(new Blob([workerCode])));
  const offscreen = canvas.transferControlToOffscreen();
  
  worker.postMessage({ canvas: offscreen }, [offscreen]);

  canvas.onpaint = () => {
    const formImg = canvas.captureElementImage(form);
    const menuImg = canvas.captureElementImage(menu);
    worker.postMessage({ form: formImg, menu: menuImg }, [formImg, menuImg]);
  };

  // Sync notification.
  canvas.onelementsdrawnonworker = () => {
    // form.getCanvasTransform() would be a DOMMatrix with X translation of 100
    // menu.getCanvasTransform() would be an identity DOMMatrix
    // form.getBoundingClientRect() would return a DOMRect at 100,0
    // menu.getBoundingClientRect() would return a DOMRect at 0,0
  };
</script>

[schenney-chromium]

@fantasai @tabatkins @shallawa @smfr @emilio

It would be great if we could get feedback on this proposed change before we get too far along implementing it in Chrome. Please pass the word around to anyone else who might be interested in giving feedback.

From an implementation perspective, the Chrome plan as I understand it is to keep track of the element bounding boxes on the canvas itself for the purposes of hit testing and hit test ordering, and then forward events to the elements. This is to avoid re-parenting or shifting the elements around in the DOM while still supporting interaction. The transforms are pushed back to the elements after drawing so that features like intersection observer still function.

[shallawa]

@progers, can you please explain how your proposed DrawElementOptions be explicitly passed? Your examples just used the default.

I am not sure also how currentTransform can be calculated. Suppose I want to do the default behavior myself. If I the form element is drawn at point { 100, 0 }, will currentTransform be the canvas ctm multiplied by translate(100, 0)?

[stubbornella]

Hi @progers, Elika and Said are on leave the next couple of weeks. We'll give this full attention when they are back.

[fantasai]

so we may want to choose a fairly general name that isn't specific to input ... CanvasTransform

CanvasTransforms seems like a reasonable name, sure.

The breadth of affected APIs makes it difficult to support general transformations, such as the sphere case, so we may want to start with the transformations available to CSS as a starting point

Given you already have at least one demo hacking this functionality in, I think this is something to tackle sooner than later. Even if we start by shipping only linear transforms, we should have in mind an API that extends to non-linear ones, and be prepared to follow up quickly. The current proposal is written as if only linear transforms are possible, since the "canvas transform" the API refers to can only be a linear transform.

DOMMatrixReadOnly getCanvasTransform();

This functionality was previously effectively read-write in your API design (and in fact, the default was to not keep the painting and hit-testing transforms in sync). Why switch to read-only (forced to match) now?

DOMMatrix drawElementImage()

Why is this still returning a DOMMatrix? This return value won't be extensible to handle non-linear transforms in the future. If we want to return the transform, having a CanvasTransform object to hold the DOMMatrix would be better, since we can extend that with more information later.

So that authors can still use CSS transforms, an Element's canvas transform should apply before CSS transforms.

I assume this means that CSS transforms transform both the element and its hit-testing "rectangle"? Ordering seems fine to me.

// True if the Element's canvas transform and hit test order should be
// automatically updated.
boolean updateGeometry = true;

Should it be possible to update the transform and not the hit test order or vice versa?

[progers]

@progers, can you please explain how your proposed DrawElementOptions be explicitly passed? Your examples just used the default.

I am not sure also how currentTransform can be calculated. Suppose I want to do the default behavior myself. If I the form element is drawn at point { 100, 0 }, will currentTransform be the canvas ctm multiplied by translate(100, 0)?

For 2d contexts, DrawElementOptions would typically be omitted and the context's CTM would be used for currentTransform. For 3d contexts, DrawElementOptions would typically be provided with a currentTransform representing the model-view-projection matrix for webgl/webgpu.

To answer your specific question, the arguments to drawElementImage such as dx would not be included in currentTransform. For example:

const ctx = canvas.getContext('2d');
ctx.reset();
ctx.drawElementImage(target, 100, 0, {currentTransform: ctx.getTransform()});

I think we can update the comment above currentTransform to make this clearer:

dictionary DrawElementOptions {
  // The base transform applied to the element, used to compute its final canvas
  // transform and hit-test geometry. The browser automatically composes this
  // matrix with the translation and scaling derived from the draw call's
  // arguments (e.g., dx, dy, dwidth, dheight).
  //
  // Defaults to the canvas CTM for 2D contexts, and the identity matrix for
  // WebGL/WebGPU. This is primarily explicitly provided for WebGL/WebGPU (e.g.,
  // passing a model-view matrix). An error is thrown if a non-null
  // currentTransform is specified and updateGeometry is false.
  DOMMatrixInit currentTransform;

  ...
};

[progers]

DOMMatrixReadOnly getCanvasTransform();

This functionality was previously effectively read-write in your API design (and in fact, the default was to not keep the painting and hit-testing transforms in sync). Why switch to read-only (forced to match) now?

Regarding the read-only type: a mutable DOMMatrix is difficult to expose directly on Element (or on a wrapper object) because of mutability. [SameObject] readonly attribute DOMMatrix canvasTransform wouldn't support element.canvasTransform = new DOMMatrix(), and attribute DOMMatrix canvasTransform wouldn't work as expected for element.canvasTransform.rotateSelf(...). For this reason, I think it's best to follow the pattern used by CanvasRenderingContext2D and use getters/setters. That said, regardless of being read-only, DOMMatrix getCanvasTransform() seems better than DOMMatrixReadOnly getCanvasTransform().

Regarding write-ability: It is easy to add setCanvasTransform(...) and I think we should do it, but the setter for hit test order is more complex. If usecases are identified, we could follow up with a setter for the hit test order.

DOMMatrix drawElementImage()

Why is this still returning a DOMMatrix? This return value won't be extensible to handle non-linear transforms in the future. If we want to return the transform, having a CanvasTransform object to hold the DOMMatrix would be better, since we can extend that with more information later.

Agreed, an extendable wrapper object would be better here. We can introduce a CanvasTransform interface which currently wraps a DOMMatrix that is nullable. @szager-chromium has been engaging with the WebGL community on this topic and will discuss this more. There may be an opportunity to remove the wrapper object if the non-linear hit testing is handled entirely on the canvas to handle global state, and we'll follow up in a separate issue for that.

Should it be possible to update the transform and not the hit test order or vice versa?

Yeah, I think it would be useful to be able to control these independently. We can split this into updateTransform and updatehitTestOrder.

Here is an updated proposal:

// A transform applied to an Element to map it to its canvas rendering.
[Exposed=(Window,Worker)]
interface CanvasTransform {
  constructor(optional DOMMatrixInit matrix);
  readonly attribute DOMMatrix? matrix;
};

partial interface Element {
  CanvasTransform getCanvasTransform();
  void setCanvasTransform(CanvasTransform transform);
};

dictionary DrawElementOptions {
  // The base transform applied to the element, used to compute its final canvas
  // transform and hit-test geometry. The browser automatically composes this
  // matrix with the translation and scaling derived from the draw call's
  // arguments (e.g., dx, dy, dwidth, dheight).
  //
  // Defaults to the canvas CTM for 2D contexts, and the identity matrix for
  // WebGL/WebGPU. This is primarily explicitly provided for WebGL/WebGPU (e.g.,
  // passing a model-view matrix). An error is thrown if a non-null
  // currentTransform is specified and updateTransform is false.
  DOMMatrixInit currentTransform;

  // True if the Element's canvas transform should be automatically updated.
  boolean updateTransform = true;
  
  // True if the Element's canvas hit test order should be automatically updated.
  boolean updateHitTestOrder = true;
};

// 2D
interface mixin CanvasDrawElementImage {
  void drawElementImage((Element or ElementImage) element,
                        unrestricted double dx, unrestricted double dy,
                        optional DrawElementOptions options = {});
  // Other drawElementImage overloads are similarly modified to have an
  // optional DrawElementOptions and return void.
}
CanvasRenderingContext2D includes CanvasDrawElementImage;
OffscreenCanvasRenderingContext2D includes CanvasDrawElementImage;

// 3D
// WebGLCopyElementImageConfig should be changed to inherit from DrawElementOptions
// GPUCopyElementImageDestination should be changed to inherit from DrawElementOptions

dictionary ElementsDrawnOnWorkerEventInit : EventInit {
  sequence<Element> elements = [];
};

[Exposed=Window]
interface ElementsDrawnOnWorkerEvent : Event {
  constructor(DOMString type, optional ElementsDrawnOnWorkerEventInit eventInitDict);
  readonly attribute FrozenArray<Element> elements;
};

partial interface HTMLCanvasElement {
  // Fired when elements are drawn on a worker thread. Drawn elements are added
  // to a pending notification queue, and a microtask is queued to batch these 
  // updates into a single task posted to the main thread.
  attribute EventHandler onelementsdrawnonworker;
};

[szager-chromium]

On the topic of non-linear hit testing: this was discussed during a meeting of the WebGL working group (minutes), and the idea of registering a coordinate re-mapping function was also raised by @ladyhavoc. Their idea, however, was to allow specifying a single coordinate re-mapping function for the canvas, rather than a separate re-mapping function for each drawn element.

In a 2D canvas each drawn element can be hit-tested separately in reverse drawing order to determine an event target, but drawing order can't be used in that way in a 3D scene. Instead, the currentTransform parameter to texElementImage2D can be used to arrange the drawn elements into (probably non-overlapping) CSS box space; the canvas-global coordinate re-mapping function will re-map a point in the projection of a 3D scene into CSS box space; and the normal hit testing process continues from there.

I'll try to nail down the details of this idea with the WebGL folks at a future WG meeting, but for now I don't think a per-element CanvasTransform.map is the right approach.