Skip to content

Tracker: get the full test matrix green (rusty_v8 + deno_core × 3 backends) #9

Description

@littledivy

Goal

Make every cell of the test matrix pass and ratchet the baselines up to match. Six cells: {rusty_v8, deno_core} × {jsc, sys-jsc, quickjs}. This is the umbrella tracker — fan out into focused PRs, each fixing a subsystem in src/jsc/ or src/quickjs/ and updating exactly one cell's baseline.

Scoreboard (as of this writing)

cell passing notes
jsc / rusty_v8 38 / 264 ~226 failing
sys-jsc / rusty_v8 38 / 264 ~226 failing
quickjs / rusty_v8 40 / 264 ~224 failing
jsc / deno_core just unblocked was 0/0 — harness bug, fixed in #7
sys-jsc / deno_core just unblocked enumerate with the loop below
quickjs / deno_core just unblocked enumerate with the loop below

Totals: 116 / 792 passing across the matrix.

The single most important insight

222 of the ~225 rusty_v8 failures fail on all three backends. The gaps are overwhelmingly engine-agnostic — missing or wrong behavior in the shared v8 C-ABI surface, not per-engine quirks. So fan out by subsystem, not by backend: implement object templates once and all three rusty_v8 cells move together. Verify the lift on each engine, then update each cell's baseline in its own PR.

Rules of engagement (do NOT violate — see CLAUDE.md)

  • Fix the backend (src/jsc/, src/quickjs/), never the tests. Vendored rusty_v8 tests and Deno's tests run as-is; editing them is cheating.
  • Ratchet only goes up. Never lower a baseline to dodge a regression — fix the regression.
  • One cell's baseline per PR (tests/status/baselines/<backend>/<suite>.txt) so parallel agents never collide on the same file. A subsystem fix that lifts all three engines → three PRs (or one PR touching one baseline + follow-ups), not one PR rewriting three baselines.
  • Never hand-edit tests/status/report.json / history.jsonl (CI owns them) or anything under vendor/ test dirs.
  • Backends are mutually exclusive features — don't enable two at once (they collide at link).

The loop

# cheapest first — rusty_v8 on quickjs, no WebKit build:
node tests/harness/run.mjs rusty_v8 quickjs
cat tests/status/.runs/quickjs__rusty_v8.json | jq -r '.failing[]'   # what to fix

# implement fix in src/quickjs/ (or src/jsc/), re-run, then record:
node tests/harness/run.mjs rusty_v8 quickjs --update    # rewrite baseline
node tests/harness/run.mjs rusty_v8 quickjs --check     # confirm ratchet holds

# JSC backends (macOS); deno_core needs a patched deno checkout — see CLAUDE.md.

A target that won't link scores 0 for its whole file — those are the biggest single wins (this is how #2 unlocked test_api). Check the build error for the missing v8__* symbol, define it, rebuild.

Fan-out: subsystem workstreams

Ordered roughly by failing-test count (counts are approximate and overlap; enumerate exact names from .runs/<cell>.json .failing). Each is an independent stream — pick one, fix across engines, ratchet per cell.

  • Object & function templates (~22 object + ~13 template + ~14 function) — e.g. context_from_object_template, context_with_object_template, function_template_intrinsic_data_property, compile_function. Highest leverage.
  • Strings & external strings (~15) — external_onebyte_string, external_onebyte_string_frees_external_memory, allow_code_generation_from_strings.
  • Context & extras binding (~12) — context_get_extras_binding_object, context_promise_hooks*.
  • Modules (~9) — module_evaluation, module_instantiation_failures1, module_snapshot.
  • Arrays / typed arrays / backing stores (~9) — backing_store*, typed-array paths.
  • Inspector / CDP (~8) — crdtp_e2e_with_v8_inspector, inspector_can_dispatch_method, inspector_async_task_smoke (builds on the crdtp shim from Unlock test_api + test_cppgc: implement crdtp (inspector) + cppgc C-ABI stubs #2/feat(quickjs): unlock rv8_test_api + test_cppgc (crdtp + cppgc + allocator refcount) #3).
  • Snapshots (~5) — snapshot_creator, snapshot_creator_context_embedder_data.
  • Promises & microtasks (~7+3) — promise_hook, context_promise_hooks_partial.
  • Externals & external references (~7) — external_references.
  • Exceptions / try-catch (~6) — bubbling_up_exception_in_function_call.
  • Weak handles & finalizers (~4) — weak_from_global, drop_weak_from_raw_in_finalizer.
  • WebAssembly (~4) — wasm_module_compilation, compiled_wasm_module (WAMR-backed on quickjs).
  • Heap / GC / cppgc (~5) — clear_kept_objects, GC-for-testing paths.
  • ICU (~4) — icu_* (needs the real ICU common data wired in, not the empty placeholder).
  • Dates, regexp, proxy, json, equality (long tail).
  • deno_core — newly unblocked. Run node tests/harness/run.mjs deno_core <backend> --deno-dir=../deno, enumerate failures, and add workstreams here.

Definition of done

  • Every cell's passing == total (or the maximal achievable set, with any genuinely-unsupportable tests documented).
  • All baselines updated to the passing set; --check green on every cell.
  • No regressions: the ratchet only moved up.

When a workstream lands, check its box and link the PR. Open follow-up issues for any subsystem that turns out to need deep work.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions