Cross-Runtime Testing APIs?

[karlhorky]

Since Node.js, Bun and Deno offer different APIs (node:test, bun:test, Deno.test) but have many similarities, would it make sense to have a set of specced cross-platform testing APIs?

Motivating Example

runtime: prefix, in the style of Node.js or Bun:

import { test } from 'runtime:test';

test('2 + 2', () => {
  expect(2 + 2).toBe(4);
});

Or, Deno-style global:

Runtime.test("assert works correctly", () => {
  assert(true);
  assertEquals(1, 1);
});

// Using existing global:
globalThis.test("assert works correctly", () => {
  assert(true);
  assertEquals(1, 1);
});

This was originally inspired by @nzakas's tweet here:

What I want: To write JavaScript tests once and be able to run them across Node.js, Bun, and Deno.

Problem: Bun and Deno have built-in test runners you have to import from to run tests. I use Mocha. This doesn't work.

Solution: ???

Source: https://twitter.com/slicknet/status/1762264774166085937

@CanadaHonk rightly mentions that, while there are similarities, there are probably challenges creating a spec:

most test framework APIs are pretty similar?

probably hard to spec due to sandbox/isolation/etc

Source: https://twitter.com/CanadaHonk/status/1762417370893516929

Alternatives Considered

Separate Package

Instead of something built into the standard library of multiple runtimes, use a separate package (either like Vitest, including testing features of its own, or a wrapper package).

Downsides:

1. Doesn't work out of the box or with zero dependencies
2. Downloading and maintaining another package in devDependencies

[voxpelli]

Why not use a test library instead then?

[ekwoka]

I think all the things you could definitely get them to agree on they already agree on.

It's the plugins and extensions that they might disagree on.

And you can use something like vitest.

While it's cool to have the runtime also have a test runner, it's not exactly in the scope of a runtime, is it? 🤔 Should package management also be prescribed?

[pi0]

If someone wanted to try something like this as a package, it is more than welcome to be part of unjs.io (cross linked to unjs/community#2) ❤️

Just reminded i got untest pkg with this idea 2 years ago and forgot to even push my progress since Vitest was born and it was amazing! I think main limitation is that we need to split the standard assertion library from runtime logic to make this happen.

[ljharb]

Since the ecosystem hasn’t ever come close to cohering on a standard pattern in libraries, i don’t think it makes sense to standardize one. Platforms choosing to offer a test framework despite this lack of coherence shouldn’t justify continued artificial forcing of patterns.

[pi0]

@ljharb Are you aware of any previous efforts that particular runtimes (Node, Bun and Deno) explicitly disagreed with a unified API for testing?

It is not an artificial need that today we cannot test a library against all WinterCG runtimes. (and it is growing, llrt made one for themselves, based on node-assert and jest-like runtime).

[pi0]

Also looking at the readme:

How are the APIs selected
By looking at the APIs that are already implemented and supported in common across Node.js, Deno, and Cloudflare Workers. If there were at least two implementations among those -- either already supported or in progress -- then it was added to the list.

I think this at least met the 2/3 criteria of this repo (or i might be totally wrong)

[ljharb]

@pi0 the issue isn’t disagreement, it’s artificially forcing into a standard something that never went through rigorous intentional design nor got extensive userland usage prior to implementation.

[pi0]

I always imagined API Standards are specifically being designed to be strict and purposed to be consistent and also imagined that WinterCG is a place to discuss about designing such (common) APIs to be consistent.

Would it still be something (in vision of @wintercg group) to be continued perhaps as a new proposal (not common-api but like proposal-testing-api)?

[Qard]

I don't think the test framework should be standardized, but I do think the lower-level machinery around it probably should be. Like a core assert(...) interface which could be wrapped to make the prettier variations like expect(...) with its chainability. I also think we should probably standardize a sandbox at the single test unit level and allow frameworks to compose and rearrange them as they see fit. Should probably also have a standardized functional output for that single unit of work with an interface that receives some event data about that run but not standardization for text output.

Test frameworks have varying structural opinions, but at the core of all of them is the need for a well-isolated sandbox which can intercept and analyze any failure from within. Standardizing that sandbox could be reasonable.

[mk-pmb]

Whatever we standardize here (if anything), please make sure it's easy to test (failing) async functions and promises (including rejection and timeout).

[ljharb]

@Qard that's not how they all work, but it's definitely worth reviewing what someone can come up with - I'd say that lacking ecosystem coherence, any approach that isn't compatible with the top N test frameworks (arbitrarily, 10?) isn't a good idea to move forward with.

[ekwoka]

I do see the benefit of standardizing the assert/expect behaviors/apis, even if other things like describe/it/test would be more up for grabs. As long as it leaves freedom for userland extension.

[Qard]

@ljharb Yep, agreed. We might not be able to make a tool that everyone will want to use, but as long as it satisfies most users it should be fine. Might even be several tools--probably would be, I would think assertions should be able to exist on their own from an error-catching execution sandbox.

A similar case is UrlPattern--not every routing framework will want to use that style, but for many it's enough. Putting aside if the design of that particular API is any good or not, I'm just using it as an example of an API solving a specific use case and not trying to be everything for everyone.

[nzakas]

From my point of view, what I would like is something along the lines of:

import { describe, it, expect } from "std:test";

Where each runtime could define std:test however it wants, or even let the developer define what std:test points to. As @ljharb points out, there's already plenty of alignment on how these things work, it's really just the matter of needing to switch between node:test, bun:test, and deno:test, or other options, that make the dream of writing your JS tests once and running in any runtime a problem.

[ljharb]

Even describe/it aren't universal among testing paradigms - that's just BDD, which isn't the only testing paradigm.