Implemented Watch mode filter by filename and by test name #1530

[mmulet]

Hi!
This pull request implements Watch mode filter by filename and by test name from issue #1530

Watch this short video for a demonstration of the feature:

Watch_Mode_Demo.mov

Overview

• Added new tests for this feature in test/watch-mode/basic-functionality.js
• Added documentation in the docs/recipes/watch-mode.md
• Feature works as requested in Watch mode filter by filename and by test name #1530. Namely:
• Press p to filter by a filename regex pattern.
• Press t to filter by a test name regex pattern.
• Press q to quit watch mode.
• Press Enter to trigger a test run.
• Reruns the filtered list of tests after applying filter (just like jest does)
• Use a to rerun all tests.

Summary of new feature

(taken from the docs)

Filter tests while watching

You may also filter tests while watching by using the cli. For example, after running

$ npx ava --watch

You will see a prompt like this :

Type `p` and press enter to filter by a filename regex pattern
	[Current filename filter is $pattern]
Type `t` and press enter to filter by a test name regex pattern
	[Current test filter is $pattern]

[Type `a` and press enter to run *all* tests]
(Type `r` and press enter to rerun tests ||
	Type \`r\` and press enter to rerun tests that match your filters)
Type `u` and press enter to update snapshots

command > 

So, to run only tests numbered like

• foo23434
• foo4343
• foo93823

You can type t and press enter, then type foo\d+ and press enter.
This will then run all tests that match that pattern.
Afterwards you can use the r command to run the matched tests again,
or a command to run all tests.

Code Overview

In the contributing guidelines, you say that you prefer shorter pull requests, but this is a bit long. So, to help speed things along, I'm going to provide an overview for each part that I added or changed.

1. The root of the feature starts in lib/watcher.js

I add a new variable called watchModeSkipTests
https://github.com/mmulet/ava/blob/78a3768fbe3360930351374f854f9cc7a46fda51/lib/watcher.js#L104

which is an instance of WatchModeSkipTests lib/watch-mode-skip-tests.js, this class handles the regex for the skipping, and most of the code will be passing this object around, populating its fields, etc.

There is one particular part of this file that I want to draw attention to.

#OnDataGenerator

Continue down the plan function until you come to the const _class;
This class handles the prompt for commands at watch mode. It does this by listening to stdin's data event in an asynchronous generator.

I switched to using an asynchronous generator rather than just plain old stdin.on('data') because some new commands (namely p, and t the ones where you enter regex), also need to listen to 'data'. By this I mean that they have to listen for data while processing the command. In other words, without the generator you would need an explicit state machine which can be hard to debug and keep track of.

To see what I mean, look at listenForCommand:

#data;
constructor() {
	this.#data = this.#onDataGenerator();
	this.#listenForCommand();
}
#onDataGenerator = async function * () {
...
}

#listenForCommand = async () => {
	for await (const data of this.#data) {
		await this.#onCommand(data);
	}
};

Note that the for await waits on #data, not on this.#onDataGenerator()! What this means is that nested inside the #onCommand function we can await more lines of input, like this:

#onCommand = async (data) => {
  switch(data) {
     case 't':
        promptTheUserForSomething()
        const dataFromUser = await this.#data.next();
        askTheUserAnotherQuestion();
        const dataFromUserAgain = await this.#data.next();
        ...

Then when you are done processing your command, ie, the this.#onCommand returns, you can run the next loop (by getting the next this.#data) and everything works as it should.

#listenForCommand = async () => {
        //This works just as you think it would,
        // each data is a new command, not
        // a prompt or something having to do with
        // inner state
	for await (const data of this.#data) {
		await this.#onCommand(data);
	}
};

And, that's how it works.
It's not too complicated, but maybe a bit unusual. So, I just wanted to document my choices and communicate clearly about why I made them.

The rest of the addition to the plan function is as you would expect, it asks the user for the command and prompts for regex filters if necessary.

Oh the places you'll go

As I mentioned before, the rest of the code is mostly passing around the watchModeSkipTests.

• From the plan function, the watchModeSkipTests, gets added to instructions which is yielded
• The yield ends up at the async for loop in start, which means it gets rolled up in the runtimeOptions variable. Which is the passed to api.run
• From inside api.run, the watchModeSkipTestsOrUndefined branches out to 2 different places

1. First is to the fork options, which passes it to the fork and in turn the worker. Note that the fork options is watchModeSkipTestsData because only the Data gets passed to the worker, not the class itself.
2. Then, it gets emitted as part of the 'run' event. This run event will be consumed by the reporter, more on that later.

Since we are branching, let's choose a path

To the worker

• So, after api.run, we can let the existing code pass all the options to the worker, it's only in [worker/base.js]
  (https://github.com/mmulet/ava/blob/78a3768fbe3360930351374f854f9cc7a46fda51/lib/worker/base.js#L75) that we need to pass the Data to the Runner
• Once we are in the runner, we've reached the skipTests 1 of 2 destinations
  It is inside the runner that we use the watchModeSkipTests to actually , well, skip tests. The logic is simple, on every chain check if we should skip the test, and if we should, we skip!

Another Side, Another Story

Meanwhile, we have a second task, not just skip a test from running, we also want to prevent its result from being shown at all (this is a filter, the whole idea is we have too many tests to sift through).

• Let's pick up from where we left off, being emitted by emit('run'), this brings us to the cli.js, all the arguments for run are in the plan variable which gets sent to the reporter with the reporter.startRun(plan).
• At startPlan, we've reached our destination, the reporter.
• Now, recall from our other destination (the runner) that we skip all tests that have been filtered out (don't match our regexes). So to prevent the output from being displayed, we look at all tests being marked [skip] and filter out those that are being skipped due to our filters

And there you have it, a complete tour of the changes!

About the tests

One thing you'll notice while reviewing the tests is that I added try catch blocks on each and every one . I did this because whenever a test would fail, the entire test would just hang and end in a timeout (presumably because it throws an error, and the this.done function never gets run).

I tried this with a simple example, and it also had the same result:

test('basic_test', withFixture('basic'), async (t, fixture) => {
	await fixture.watch({
		async 1({process}) {
			t.is(1,2); //should fail
			this.done();
		}
	});

Your contributing guidelines say not to include unrelated code changes in the pull request, so I'm considering this error as out of scope for now. But, now you know why the try catch blocks are there (the CLI will still report a t.$someFunc() failure as a failure, it just won't report exceptions as errors, throw $someError will be caught)

Last but not least

I'm going to submit this pull request for the bounty on https://oss.issuehunt.io/r/avajs/ava/issues/1530, but any extra tips (you
can use GitHub Sponsor page) would be very welcome!

---

IssueHunt Summary

Referenced issues

This pull request has been submitted to:

• #1530: Watch mode filter by filename and by test name

---

[novemberborn]

Hey @mmulet, this sounds really great! I'm a bit short on time but can hopefully have a look in the next few days.

[novemberborn]

High-level question, Jest uses regular expressions to select files and tests? I ask because our CLI options use glob patterns for files, and wildcard matching for individual tests. I'm wondering if it's more valuable to follow our CLI behavior, or to match Jest's interface.

And the regular expressions are matched agains filenames, rather than paths?

Also we refer to test "titles", not "names".

[novemberborn]

One thing you'll notice while reviewing the tests is that I added try catch blocks on each and every one . I did this because whenever a test would fail, the entire test would just hang and end in a timeout (presumably because it throws an error, and the this.done function never gets run).

Oh thanks for the reminder, I think I've suffered through that in the past. I'll push up a fix.

[novemberborn]

Fantastic start @mmulet. I've pushed up some small tweaks and included some more impactful suggestions below. What do you think?

[mmulet]

High-level question, Jest uses regular expressions to select files and tests? I ask because our CLI options use glob patterns for files, and wildcard matching for individual tests. I'm wondering if it's more valuable to follow our CLI behavior, or to match Jest's interface.

1. Confirmed yes, jest uses regex pattern. Doesn't make a difference to me whether is uses regex or glob.
   

And the regular expressions are matched agains filenames, rather than paths?

2. Just tested it in jest again, and jest matches against paths, not just filenames. I believe this the implementation I wrote for ava also matches against the whole path.

Also we refer to test "titles", not "names".

3. My bad.

[mmulet]

I believe I responded to all the suggestions. I'm out of time for now, but I should be able to work again on this Sunday night or Monday and implement these changes.

In the meantime, feel free to make any changes. Based on the suggestions you've made, we really seem to be on the same page here. Plus, I'm not the type to be possessive about code, so rewrite the whole thing if you want to.

[mmulet]

Summary of changes

This should be in the same order as the suggestions given above.

• Test "titles" not test names. In watch-mode-skip-tests
• Test "titles" not test names. In cli
• metadata.selected instead of skipped
• adjust the basic-functionality tests to reflect selected vs skipped
• initialize to selected: false in runner.js
• Skip loading entire file if we already know if we are skipping all the tests
• Remove shouldSkipThisTestBecauseItDoesNotMatchARegexSetAtTheWatchCLI wrapper
• Change tests fixture to just use t.pass() or t.fail() instead of t.is("bar", "bar")
• Instead of endMessage(...) : string, use endMessage(reporter,...) : void
• Change watchModeSkipTests to InteractiveFilter
• Change onDataGenerator to use split2
• Remove anonymous class, and replace with more variables
• Change watchModeSkiptTestsOrUndefined to InteractiveFilter (no undefined)
• Explicit api to convert WatchModeSkipTests to data (with getData)
• Made the InteractiveFilter regexes private.
• Remove replace(file|test)Regex and replace with setter
  Note, I actually didn't remove the replaceRegex function, because I added a new return value to it (filterChanged) that let's us know if the regex had any changes. This way, we don't have to rerun the tests if the regex didn't change.
• fileRegexOrNull => fileRegex (no null), same for titleRegex
• shorter invocation for this.#fileRegex?.test(file) !== true; replaced with !(this.#filepathRegex?.test(file) ?? true)

[mmulet]

This seemed like the best place to skip loading entire file if it was filtered out by the interactive filter, but would somewhere else be better?

[mmulet]

When I moved to metadata.selected rather than metadata.skipped, I had to delete the checks for this.checkSelectedByLineNumbers whenever it also checks metadata.selected.

All tests still pass, and it works fine. So, I am assuming that these checks are not needed.

[novemberborn]

@mmulet still looking at this; there's some edge cases that go pretty deep and I wanted to see how it'd work with globs and matches, too. Getting close to having that resolved.

Edge cases:

• Not running tests while prompting for a filter
• Not clearing the command prompt line when running tests due to changes, breaking the separator
• Correctly resetting previous failure counts when filtering over all possible test files (i.e. not from a file change)
• The interplay between the command line glob filter and the interactive file filter (regex or glob)
• The interplay between the interactive filter (regex or --match equivalent), --match and .only test modifiers (struggling with this at the moment)

I'll push up some changes when I can, for your feedback.

Thanks for pushing this forward!

[novemberborn]

@mmulet I'm about to push up some changes, please let me know what you think and thank you for bearing with me.

They fall into different categories:

• Editorial: renaming functions, moving code, changing the copy
• Feature removal: .only() is no longer special within the watcher, users should use the interactive filters instead
• Internal cleanup: change handling, reporter tweaks; implement --match using selected not exclusive flags
• Edge cases: don't run tests when files change when prompting for a filter; failure count resets

And then the big change: use glob patterns for file selection, and make this work by having the watcher handle all selection rather than api.js.

As well as a smaller change, using match patterns (akin to --match) for test selection, which just means passing down an additional pattern to what comes from the CLI.

I've yet to review test coverage, some edge cases could do with additional tests.

[mmulet]

Sorry for the delay, it might be a week or two until I get a chance to look at this.

[mmulet]

This is going to sound anticlimactic after 3 weeks, but "Looks great!".

• I like the glob patterns better than regex, it's more consistent
• I also like that we got rid of the InteractiveFilter class and moved everything to selected. It's much cleaner.

A few fixes

• I updated the docs to reflect the new glob patterns
• I found that if you do not have any patterns, but type m or g to enter a glob, then you leave the prompt blank, it will not return to the original prompt. The cause is oldGlobPattern === interactiveGlobPattern in watcher.js. I fixed it by always calling signalChanged() (moving it out of the else), but I'm not sure if this will have unintended consequences.

[novemberborn]

This is going to sound anticlimactic after 3 weeks, but "Looks great!".

Haha, no worries, glad to hear!

• I found that if you do not have any patterns, but type m or g to enter a glob, then you leave the prompt blank, it will not return to the original prompt.

Yes I did that on purpose actually, as a way to reset the pattern. With that insight, do you think it's a bug or a feature?

[mmulet]

Yes I did that on purpose actually, as a way to reset the pattern. With that insight, do you think it's a bug or a feature?

It does work to clear the pattern (and that's good), but I ran into the problem when I don't yet have any patterns (or I wish to keep the pattern the same).

Let's say I type m and enter the prompt for a new test filter, but I change my mind (or I meant to do file filter) and want to cancel and go back. It's not immediately clear to me how to do that. My first thought is to leave the prompt blank (because the tips are telling me to leave it blank to clear the pattern) and press return, but when I do that it just enters a line and I can't go back.

For this video I reverted the changes I just made to show what I just described:

ava.mov

I have no way to escape the filter prompt without entering a filter.

[novemberborn]

I have no way to escape the filter prompt without entering a filter.

OK that's definitely a bug.

Let's say I type m and enter the prompt for a new test filter, but I change my mind (or I meant to do file filter) and want to cancel and go back. It's not immediately clear to me how to do that.

I don't think we can do a cancel, other than by having you type a magic input. The TTY isn't in raw mode (which would let us recognize say an ESC press). That's because raw mode breaks Control+C interrupts.

[mmulet]

I have no way to escape the filter prompt without entering a filter.

OK that's definitely a bug.

Let's say I type m and enter the prompt for a new test filter, but I change my mind (or I meant to do file filter) and want to cancel and go back. It's not immediately clear to me how to do that.

I don't think we can do a cancel, other than by having you type a magic input. The TTY isn't in raw mode (which would let us recognize say an ESC press). That's because raw mode breaks Control+C interrupts.

The cancel isn't a big deal, so we can skip it if it's too much trouble. However, if you do want to support it, I think we might be able to use node's readline module for the prompt which already has ctrl+c handlers built in. readline might also fix that bug too.

[novemberborn]

Oh, readline, yes that should be entirely appropriate for this use case! Can't believe I forgot about that.

Looks like Ctrl+D can act as a "I have no input, don't erase my pattern" signal?

[mmulet]

Oh, readline, yes that should be entirely appropriate for this use case! Can't believe I forgot about that.

Looks like Ctrl+D can act as a "I have no input, don't erase my pattern" signal?

That would work but unfortunately Ctrl+D doesn't work on Windows.

Right now I added support for using Ctrl+C, such that Ctrl+C works as normal at the top of the prompt, but let's you use it as I have no input, don't erase my pattern while it is asking for a glob or match pattern.