Skip to content

fix(test-results): parse the reports real runners actually emit - #283

Open
colek42 wants to merge 1 commit into
mainfrom
fix/test-results-real-reports
Open

fix(test-results): parse the reports real runners actually emit#283
colek42 wants to merge 1 commit into
mainfrom
fix/test-results-real-reports

Conversation

@colek42

@colek42 colek42 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

The bug

The test-results attestor could not read the JUnit output of node --test
the runner that ships with the language. A JS project using it produced no
test-results attestation at all
, so any policy requiring one was unsatisfiable
by that project, while the tests passed and a valid report sat on disk.

Found by running the attestor against a real project rather than against
fixtures. The failure surfaced as:

attestor test-results failed: no JUnit XML or CTRF JSON test report found in products

which points at product capture. Products were fine — product/v0.3 leaves
contained exactly junit.xml. The parser was rejecting it.

Three shapes, three reasons

Testcases directly under <testsuites>. Node emits
<testsuites><testcase/>…</testsuites> with no intervening <testsuite>. The
parser handled the nested form and a bare <testsuite> root, so the document was
correctly detected as JUnit and then refused with "contains no testsuite
elements". Root-level cases are now wrapped in one synthetic suite — left
unnamed, because the document did not name it and a fabricated name would appear
in the attestation as though the emitter had supplied it.

An unnamed <testsuite> root. The fallback required single.Name != "", so
whether a run counted at all depended on an optional attribute. Cases now
qualify it too.

ANSI escapes in failure text. Runners embed the failing assertion verbatim,
and Node writes the error's cause with colour codes still in it — 56 raw ESC
bytes in the report from a 2-failure run here. XML 1.0 forbids those, so the
document is not well-formed and no conforming parser will read it.

That last one made the failure path worse than a missing message: a policy asking
for passing tests was satisfied only by runs that passed, and refused runs
that failed with "no evidence" — the same answer it gives when no tests ran
at all. Those are different facts and a gate should not conflate them. Bytes XML
forbids are now dropped before unmarshalling; nothing is re-encoded and no
structure is repaired, so a document malformed for any other reason still fails.

Verified against a real project

A 9-test Node project, node --test with the JUnit reporter, evidence produced
by a cilock built from this branch:

run attestation policy
passing total 9, passed 9, failed 0 satisfied
real regression planted total 9, passed 7, failed 2 denied, naming both failing tests

The two it named are exactly the tests that catch the planted regression
(lexical instead of numeric version compare).

Tests

Six new unit tests: all three shapes, plus the two things that must still be
refused — an empty <testsuites>, and a structurally malformed document.
Widening what the parser accepts must not turn "no tests ran" into a passing
report. Each fails against the unfixed parser.

Existing tests in the package are unchanged and still pass.

The attestor could not read the JUnit output of `node --test`, the runner that
ships with the language. A JS project using it produced no test-results
attestation at all, so any policy requiring one was unsatisfiable by that
project — while the tests passed and a valid report sat on disk. Found by
running the attestor against a real project rather than against fixtures.

Three shapes were rejected, each for its own reason.

**Testcases directly under <testsuites>.** Node emits
`<testsuites><testcase/>...</testsuites>` with no intervening `<testsuite>`.
The parser handled the nested form and a bare `<testsuite>` root, so this
document was correctly DETECTED as JUnit and then refused with "JUnit document
contains no testsuite elements". Root-level cases are now wrapped in one
synthetic suite. It is left unnamed rather than given an invented name: the
document did not name it, and a fabricated name would appear in the attestation
as though the emitter had supplied it.

**An unnamed <testsuite> root.** The fallback required `single.Name != ""`, so
whether a run counted depended on an optional attribute. Cases now qualify it
too.

**Failure text containing ANSI escapes.** Runners embed the failing assertion
verbatim, and Node writes the error's `cause` with colour codes still in it —
56 raw ESC bytes in the report from a 2-failure run here. XML 1.0 forbids those,
so the document was not well-formed and no conforming parser would read it. That
made the failure path worse than the missing-message it looks like: a policy
asking for passing tests was satisfied only by runs that PASSED, and refused
runs that FAILED with "no evidence" — the same answer it gives when no tests ran
at all. Those are different facts and a gate should not conflate them. Bytes
XML forbids are now dropped before unmarshalling; nothing is re-encoded and no
structure is repaired, so a document malformed for any other reason still fails.

The empty-document guard is unchanged and still tested: widening the shapes we
accept must not turn "no tests ran" into a passing report.

Verified against a real project (9 tests, node --test, JUnit reporter):

  passing run  -> total 9, passed 9, failed 0
  failing run  -> total 9, passed 7, failed 2, naming both failing tests

Six new unit tests cover all three shapes plus the two things that must still be
refused — an empty <testsuites>, and a structurally malformed document. Each
fails against the unfixed parser.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants