[Feature] Include x/y coordinates and pointerdown/pointermove/pointerup for any element during codegen

[JonMarbach]

Hi there; firstly I am amazed by Playwright - it's genius - thanks to everyone involved.

I see the previous enhancement that was committed which added x/y mouse coords to canvas elements (#9503), thinking that would meet my needs, but then I realized that we are actually using an img element like a canvas to display some off-screen-rendered content. I know our situation is pretty unique, but I saw other requests here asking for x/y coords in a div so I know I'm not the only one who would like x/y for any element. (Perhaps this could be done via a custom attribute on the element itself?)

Additionally, it would help us to record mouse/pointer events to capture pointer down/move/up. (Our application has in some places an interactive 3D editor.) I don't see support for that but maybe I've missed it.

(Apologies in advance if I'm doing anything wrong here in the submission - I am a newcomer.)

[mxschmitt]

Sounds like this is a very niece use-case. For canvas elements the Playwright codegen already generates x/y coordinates as you linked above. Also Playwright tries to generate role selectors, so click an element more from the user perspective "a button with text Submit" rather than by x and y coordinates.

You can see the recorder as a tool which works in most cases, but if the application you are testing does things in a very uncommon unique way, its probably better to write your tests manually.

[JonMarbach]

Thanks @mxschmitt. I'm not sure if you meant nice or niche in your response (maybe it's both), but really our use case is very similar to people who are using a canvas for custom display (users click and drag to move things around within that element), we just happen to be displaying the results in an img.

If the pointer event recording is kind of "out of scope" for Playwright that's understandable, however, it would help us a ton to just be able to get the XY of clicks on an arbitrary element (or, to be extremely minimal, to get xys on img elements as well as canvas.) I tried looking at the code for how canvas XY was added and it seems like it would be pretty trivial to make that a little broader - unfortunately I'm not in a position to make the change myself and submit a pr.

[mxschmitt]

Having x/y clicks on IMG elements sounds like a good feature request to me!

[asturur]

Hi everyone i m approaching playwright those days, i am one of the maintainers of https://github.com/fabricjs/fabric.js and i was looking forward writing a suite of tests for the library with this tool.
Recording of mouse move or pointer move with specific canvas coordinate would be a very appreciated feature indeed.

[abrowne2]

Would also love if this was feature!