deno test printer feedback

[jespertheend]

I've been using deno as a test runner for some time now, and after having written about 900 tests, I've found a few issues with the way test results are printed. I'm not sure about any future plans for this, and I didn't want to create a separate ticket for every issue I had. So I figured I'd post something here first.

Finding the source of a failing test

When a test fails, the first thing you want to do is go to source code of the failing test in order to find out what exactly the test is about. Right now this is not very easy, you can do one of the following:

• Look for a path in the stack trace of the failing test, but sometimes stack traces are long and you won't be able to find the file you need right away.
• Scroll up to your test results and find the one that says 'FAILED' then look up a few lines until you see the path of the file.
• Use your ide to search for the name of the test in all your files, and hope that another file with a similar test name doesn't exist.

Possible solutions:

• Deno.test: shorter stack traces #13586 would certainlly help, but not eliminate the issue entirely
• In the list of test results, print the running n tests from file:///path/to/file.test.js with a different color. That way when you scroll up you can immediately distinguish tests that come from one specific file.
• Print the path of a failing test above the error. Currently the name of the test is already printed, but often this is not enough. Say you have a file calculator.test.js with test name "it adds two numbers together" and only the name is printed, it is not immediately clear that this test is from calculator.test.js rather than another file that might have a similar test.
• Print the path above the final list of failures. This could look similar to how the first list of test results look, where it prints the path only once if multiple tests in a single file fail.

It's hard to distinguish console.log() calls from test results

Right now if code calls console.log the result gets appended in between [test name] ... and ok without any new lines, and multiple calls print double new lines in between every call.
The same happens with Check https://example.com/file.js being printed when dynamically importing files.

For instance, this test, which dynamically imports blob urls:

Though I reckon this would be pretty nontrivial, it seems logging code is already pretty scattered accross the codebase (see #13001).

Hiding test results if they don't fail

If I remember correctly, the test runner used by the Chrome DevTools team simply shows a progress bar of all the tests and only failing tests are reported. I believe they use Mocha.

In a way I like it because it's a lot easier to find failing tests, since you don't have to scroll through all the test results. And if you have a huge amount of tests, where one of the first ones fails, you won't have to wait for the entire suite to finish because the error would get printed right away.

But on the other hand I kind of like the verbosity that Deno has right now. Seeing a whole bunch of green ok's is pretty satisfying.

A quick mockup

Here's a quick mockup of what some of these changes could look like.

I think it's a lot easier to visually parse like this. Not sure how feasible it would be to print error messages in red though. Since error messages might have custom color formatting. Maybe they should only be printed in red if the error message does not contain any ansi codes.

Recap

Quick recap of all the changes:

• print file paths in the results with a different color
• add file paths to the errors section
• add file paths to the failures section
• print error messages with a red color
• divide the tests in three sections: "results", "errors" and "failures" and print each section in yellow
• Better printing of console.log() calls in between the results (not shown in the mockup)

Feel free to let me know if these changes are desirable (or if only a subset of them are), I'd be glad to create a new ticket for each of these.

[kitsonk]

@bartlomieju has been looking into this. I am sure he will consider your feedback.

[bartlomieju]

@jespertheend thanks for opening this. As pointed out by Kit this is something I discussed with the rest of the team and want to improve to bring it closer to other popular tools like vitest.

The things I wanted to focus on in the first pass were:

• better visual separation of different test files (feat(test): add empty line between files #13930)
• improve final report, including failing tests details
• better handling of output from the the tests (console logs and direct writing to Deno.stdout/Deno.stderr)
• remove "internal" stack frames from errors (ie. don't show frames that code from Deno code)
• add "short" (or "dot" reporter)

In the second pass I want to use miette to provide nicer errors with source code context (this is a huge undertaking and requires severe changes to the test runner, but these changes are needed anyway to support timeouts in tests)

Thanks for the mockup, it's very useful. Feel free to open separate issues so we can discuss them more in depth.

[jespertheend]

Thanks! I went ahead and created:

• test: Print file paths with a separate color #14186
• test: add file paths to the errors section #14187
• test: add file paths to the failures section #14188
• test: print error messages with a red color #14189
• test: Better headers for the three deno test sections #14190
• test: console.log calls get printed in between the test results #14192

By the way, I used https://github.com/denoland/deno/files/8384024/testmockup.js.zip to generate the mockup, in case anyone wants to play around with it.