[Feature] Mouse helper (tracking mouse position, clicks, drag & drops, selections)

[thernstig]

It would be nice to natively have support for a mouse helper. So if running in headful mode, or taking a screenshot, or video recording, one would see where the mouse is, text selections, drag & drops, and where it clicks during a failure.

@aslushnikov already had one in Puppeteer but it required installing it, but maybe it could be default-installed on the browserContext or page instead if wanted, or even for the browser. Having it out of the box would improve usability for all users. Especially during debugging.

puppeteer/puppeteer#4378 (comment)

[aslushnikov]

This will be addressed by #3526 (review)

[LanderBeeuwsaert]

Yes, in our test cases a lot of the time it's difficult to know at exactly what line playwright failed. Having visual confirmation; most importantly in the video if & where mouse clicks were performed will for some issues certainly help us to identify them. In cypress when you hover over an action in the action bar & the action is a click, you see the click on the screenshot for that moment, which is quite helpful.

[aslushnikov]

This should be now addressed with a much more advanced ui - please check out the new playwright inspector: https://playwright.dev/docs/inspector/

I'll close this bug now. We continue working on making inspector UI nicer and more helpful, so if you have any issues or suggestions for it, please file separate issues!

[thernstig]

@aslushnikov Maybe I am missing something, but how does this work if I execute a page.screenshot() as a custom handling when failures happen? Or if I use the record functionality, to see the mouse move?

I.e. I want this to use this in CI when a test case fails, I take a screenshot, and see where it failed. Without having to re-run with the inspector.

[LanderBeeuwsaert]

yeah, that would also be our preferred usage. We want to run it during our CI runs. We found out that +- 40% of the work is building the tests; but +-60% of the time we spend on hardening them when 1 in 20 runs a command fails because for that run it behaves a bit different because the CPU was slower or whatever. It's there that we would like to gain time.

[aslushnikov]

@thernstig @LanderBeeuwsaert good points! Re-opening then

[thernstig]

@aslushnikov great :) No idea of how you achieved this with the inspector, but hopefully you can re-use some of that code for also this scenario we described i.e. in CI.