UI Tests Code Coverage

[f-delahaye]

Any interest to have (basic) support for code coverage of UI Tests in Karate?

Unlike https://stackoverflow.com/questions/69297222/is-it-possible-to-get-code-coverage-with-karate-tests-for-java-code, this issue is about coverage of the Javascript code run by UI Tests.

AFAIK, nyc/istanbul is the standard tool for JS code coverage (what JaCoCo is for Java).

What Karate could do is to export the coverage collected by istanbul in files
What it probably can't do is:

• instrument the code
• and generate the report from the files

The former could be done by using one of the istanbul plugins (e.g. babel-istanbul-plugin, or vite-istanbul-plugin) or with the nyc CLI prior to the code being deployed.
The latter can be done with the nyc CLI

So, most of the work would have to be done outside of Karate - unless someone has an idea.

That said, I'm happy to propose a PR for the part that can be implemented in Karate if it can add value to the project.
I was thinking maybe a RuntimeHook could do the trick, with maybe a few parameters to enable/configure it?

[ptrthomas]

@f-delahaye I could be wrong - if you mean server-side code coverage when you run ui automation, haven't seen asked for. I have been asked if karate can do UI component tests, and I have never given much thought to it. perhaps we could bring up a browser as a "sandbox", load a page or inject a JS widget and do some things, and wonder how useful it is

we have been asked many times if there is a way to get UI performance when the backend is under a load test, refer https://stackoverflow.com/a/60101051/143475 - just mentioning this as perhaps related

[f-delahaye]

Thanks @ptrthomas

I actually meant coverage for the JavaScript code running in the browser.

Playwright (https://playwright.dev/docs/api/class-coverage) and Puppeteer (https://pptr.dev/api/puppeteer.coverage), among others, support it.
What i had in mind is only a few lines of code, though, and is nowhere near as advanced as their stuff .

I will try to create a mini example so you can have a better idea of what it'd look like.

[f-delahaye]

@ptrthomas
More explanations available here: https://github.com/f-delahaye/karate/tree/2703-ui-tests-coverage/examples/ui-coverage

Code itself has not been committed yet, waiting for your green light to start working on a better implementation.

But basically, running the rpm and mvn commands detailed in the README.md would generate this:

karate.get and karate.setHtml are fully covered by the tests, while karate.uncovered is not.

[ptrthomas]

@f-delahaye so this does not need any changes to karate core and is more of an example for others to refer to? in that case I'm fine with the new examples/ui-coverage. I would prefer it here actually (trying to move things over gradually), but you can decide: https://github.com/karatelabs/karate-examples/blob/main/README.md

[f-delahaye]

Hi @ptrthomas,

Sorry for the confusion, this would require some java code - basically a RuntimeHook with a couple of calls to the Driver to retrieve the coverage data, and a few File.write/createDirectories.

I could add the RuntimeHook in karate-examples,, or try to integrate it directly within karate-core with an extra configuration option.

The latter would be easier from a user perspective. But it would be just one piece of a process that includes npm commands outside of karate, as demonstrated in the example. So, is it really karate-core?