Test runner matching every .ts and .js if glob is not provided

[marco-ippolito]

Version

Test on v23.6.0 and v22.10.0

Platform

All

Subsystem

test_runner

What steps will reproduce the bug?

Consider this folder structure:

└── test
    ├── fixtures
    │   ├── boom.js
    │   └── boom.ts
    └── index.test.js

When running node --test the test runner will execute ./test/fixtures/boom.js.
In v23 it will also execute ./test/fixtures/boom.ts, since --experimental-strip-types has been unflagged.

marcoippolito@marcos-MacBook-Pro test % node --test /Users/marcoippolito/Documents/projects/test/test/fixtures/boom.js:1 throw new Error('boom'); ^

Error: boom
at Object. (/Users/marcoippolito/Documents/projects/test/test/fixtures/boom.js:1:7)
at Module._compile (node:internal/modules/cjs/loader:1739:14)
at Object..js (node:internal/modules/cjs/loader:1904:10)
at Module.load (node:internal/modules/cjs/loader:1473:32)
at Function._load (node:internal/modules/cjs/loader:1285:12)
at TracingChannel.traceSync (node:diagnostics_channel:322:14)
at wrapModuleLoad (node:internal/modules/cjs/loader:234:24)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:151:5)
at node:internal/main/run_main_module:33:47

Node.js v23.6.0
✖ test/fixtures/boom.js (36.07275ms)
'test failed'

/Users/marcoippolito/Documents/projects/test/test/fixtures/boom.ts:1
throw new Error('boom TS');
^

Error: boom TS
at Object. (/Users/marcoippolito/Documents/projects/test/test/fixtures/boom.ts:1:7)
at Module._compile (node:internal/modules/cjs/loader:1739:14)
at Object.loadTS [as .ts] (node:internal/modules/cjs/loader:1831:10)
at Module.load (node:internal/modules/cjs/loader:1473:32)
at Function._load (node:internal/modules/cjs/loader:1285:12)
at TracingChannel.traceSync (node:diagnostics_channel:322:14)
at wrapModuleLoad (node:internal/modules/cjs/loader:234:24)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:151:5)
at node:internal/main/run_main_module:33:47

Node.js v23.6.0
✖ test/fixtures/boom.ts (62.136209ms)
'test failed'

✔ should return true (0.3725ms)
ℹ tests 3
ℹ suites 0
ℹ pass 1
ℹ fail 2
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 72.075583

✖ failing tests:

test at test/fixtures/boom.js:1:1
✖ test/fixtures/boom.js (36.07275ms)
'test failed'

test at test/fixtures/boom.ts:1:1
✖ test/fixtures/boom.ts (62.136209ms)
'test failed'

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior? Why is that the expected behavior?

Maybe this is intended behavior but I'd would expect matching .test.js.
I think it's an undesired side effect to execute everything.
I immagine breakages due to a lot of .ts fixtures being executed.

What do you see instead?

Everything is executed.

Additional information

I know changing this would be a breaking change, but I dont think it's sane as it is.

[targos]

It matches the files because they are under a test folder.
The default matching behavior is documented here: https://nodejs.org/docs/latest/api/test.html#running-tests-from-the-command-line

[pmarchini]

As correctly pointed out by @targos, this behaviour is intended.
I believe this is mostly a logic inherited from tapjs.
Checking other tools, this behaviour is different, and only **/*.{test,spec}.?(c|m)[jt]s?(x) are being matched.

IMHO, it is common to have test helpers/fixtures under the test directory.
For this reason, I believe we should consider changing this behaviour, even if it means introducing a breaking change.

[cjihrig]

Mocha also checks for a test/ directory. Independent from that, I would be -1 to changing this default. It's not just a breaking change, but an aggressive breaking change because people often run their tests across multiple versions of Node. The introduction of globbing (while a great change) has caused many headaches between Node 20 and 22. This change would not have nearly the same upside.

[marco-ippolito]

I think executing only .test.js/ts would be a better default behavior.

[AugustinMauroy]

I agree that a huge breaking change. But I'm also convinced that **/**.test.{cjs,mjs,js} or **/**-test.{cjs,mjs,js} is the healthiest for the ecosystem. Knowing that the test runner has not yet been given too much attention

[p-mcgowan]

Without changing the default behaviour, perhaps a flag --test-file-pattern=dist/test/*.spec.js instead?

Normally the positional args would be fine, but if using with npm test you cannot default it in the package script and still pass in flags afterwards:

"scripts": {
  "test": "node --test 'dist/test/*.spec.js'"
}

npm test -- --test-only 'dist/test/some-folder/*.spec.js'

In this case, the --test-only flag is ignored as it's after positionals.

Alternatively, allowing flags after positionals would also work, but the default pattern flag would be nice if you could additionally pass in a test name to run only that file instead of .only'ing it.

eg.
if test folder contains:

• a.spec.js
• b.spec.js
• setup.js
• something.js

running node --test --test-file-pattern='*.spec.js' should run a and b, whereas node --test --test-file-pattern='*.spec.js' test/a.spec.js would only run a.

Not sure how feasible that is, but something like that behaviour would be nice.

Related: #51384

[marco-ippolito]

As supected unflagging experimental-strip-types in v22 is a breaking change because it will execute .ts if glob is not provided. Ideas?
We just disable .ts by default for just v22? Id do it for other lines too along with .js (but it would be a breaking change)
.spec.ts and .test.ts are fine imho

[cjihrig]

.spec.ts and .test.ts are fine imho

I don't think there is a scenario where you can execute any additional files and be able to guarantee it won't break anyone. IMO the benefit to unflagging in the same way as v23 is worth it. I think users will be far more annoyed by the lack of "easy" TypeScript support in the test runner for the rest of v22's lifetime than they will be over possibly having to rename fixtures one time.

[cjihrig]

FWIW, we didn't backport globs in the test runner to v20 because there was a remote chance that the glob would match some existing pattern. In retrospect, I think that was a massive mistake (which we could technically still rectify), as it introduced hard to work around changes between v20 and v22. I don't think we should make the same mistake here.

[marco-ippolito]

Possible options are:

• 👍keep as it is, and unflag (some users will experience a breaking change) (example ./test/fixture/foo.ts will be executed)

• ❤️ remove the most problematic glob */test/**/*.{cts,mts,ts} from v23 too, this should remove a big chunk but still some user might experience breaking changes if they use .test.ts file (super niche case imho)

• 🚀 not unflag

• 🎉 disable .ts .test.ts etc... matching completely

[cjihrig]

There are other options too (ignore a test/fixtures directory, only process .spec.ts, etc.), but I think we need to pick a general direction before nailing down specifics like that.

[marco-ippolito]

I thin the main problem I heard users talkabout are:

• TypeScript projects with .test.ts files in the src folder, and once transpiled the execute the .test.js files. Now node --test that will execute both.

• ./test/fixtures/.ts were previously ignored now executed

Another issue is that there is no way to run the same command to exclude .ts files in v20 and v22 (Immagine a library supporting v20 and v22 running tests, v22 enables typescript and they have to disable it, they cannot use the same command for v22 and v20 because requires glob and they cant --no-experimental-strip-types)

[marco-ippolito]

Or backport glob to v20 there is a special semver minor in v20 coming could be a good time 😃