Exit early when providing invalid rule ID to the CLI

[thibaudcolas]

Product: CLI

Expectation:

When I use the CLI with the -r option to specify which rules to run, and provide an invalid rule ID, I’d expect the CLI to exit early with a relevant error message.

Actual:

When providing an invalid rule ID, the CLI takes much longer to report a failure, and the failure message is a very generic "An error occurred while testing this page.".

Motivation:

Here’s a comparison of three scenarios:

# Takes 2-3s to run, returns results for all rules as expected.
time axe https://www.example.com/
# Takes about the same time to run, returns results for just the one rule.
time axe -r html-lang-valid https://www.example.com/
# With an invalid rule name, this takes 22s to run:
time axe -r html-valid-lang https://www.example.com/

I imagine there is a 20s timeout here. It feels unexpected to me that the CLI (or Axe core itself?) would have to wait for such a timeout when it’s not provided with any valid rules to run. Even if there was no early validation of the rule IDs provided, when the CLI runs, it should fail right away, with either a generic "there are no rules to run" or specific "invalid rule ID".

axe-core version: 4.4.3
@axe-core/cli: 4.4.4

- Node version: v16.16.0
- Platform:  Mac

[Zidious]

@thibaudcolas Apologies for the delay,

Thank you for reporting this.

There are a few ways we could handle this case:

1. If there is only one rule supplied and it is invalid, default to run all rules and display an warning message about the invalid rule
2. If supplying more than one rule and one of the rules supplied is invalid, go ahead and rule axe against the valid rules display a warning message about the invalid rule(s)

[thibaudcolas]

👍 no problem. I like both approaches, to me what matters most is to get feedback on the fact I used an invalid rule ID as early as possible.

This happened to me again just yesterday, when I forgot th-has-data-cells was plural. It’s not hard to fix but does take a bit of time to understand what is happening.

[straker]

Axe-core throws an error when passed an invalid rule to run, so we should surface that error to the user as soon as we get it from axe. We shouldn't try to correct the issue about the invalid rule and run axe without it.