Increase coverage NFv3 - results of SetNumberFormatDigitOptions

[romulocintra]

Uses Example given in tc39/ecma402#575 (review) to increment coverage in NFv3 tests and AO - SetNumberFormatDigitOptions.

Related with #3508

[romulocintra]

• Check if options has min and max SignificantDigits to assert on resolvedOptions

[Ms2ger]

This PR contains an implementation of the relevant part of the spec to compare against the user agent's implementation. The main question here is if this is an approach we'd like to land in test262. We could also consider landing it in staging for now, to at least have the increased coverage while we discuss.

[ptomato]

Unfortunately, I forgot to add this to the last maintainers meeting agenda. I've put it on the agenda for next time.

[jugglinmike]

Hi there, @romulocintra! We reviewed this patch during the Test262 maintainers' meeting yesterday. Coul you use a more declarative style? We'd prefer a set of meaningful cases with hard-coded expected results.

While your recreation of the proposal's algorithm is impressive, it becomes a sort of "reference implementation" if maintained here in Test262. We're not just interested in verifying the correctness of other runtimes; we also want to validate the design of the proposals themselves. In that sense, writing and reviewing a curated set of assertions (common cases, edge cases, etc.) is an important part of the design process--even if it doesn't inspire changes to the text.

If semantically-related "groups" of assertions emerge from this change, then all the better! Factoring those into dedicated tests will give implementors a better understanding of their bugs when/if failures occur (remember that a Test262 test can only fail once).

Thanks for the help!

[sffc]

Hi @jugglinmike -- This particular AO has a lot of corner cases that are difficult to cover fully from higher level assertions. It was very helpful when editorially refactoring this AO to write this test in order to ensure that we produce the same output given the same input. I think adding this test is useful in this specific case, even if this type of test isn't generally applicable in other cases.

[jugglinmike]

Hi there, @sffc! I've also written reference implementations to help validate others' work, and I've even used projects from third parties along these lines. While it may be tempting to enshrine those implementations as the de-facto source of truth, test authors and their reviewers would miss an opportunity to reaffirm the specified behavior. Abstractions like this patch's SetNumberFormatDigitOptions_v3 meaningfully distances editors and implementors such that accepting a patch is more akin to saying, "this code matches the proposal," and less like declaring, "this is how we wish implementations to behave."

Using an alternate implementation is a great way to identify edge cases, and once identified, we'd all benefit from tests which give some structure to those edges--grouping them as appropriate under titles/messages which describe their semantic significance. Such intentional curation does not have to mean settling for inferior coverage, though it almost certainly will lead to fewer assertions. (That also satisfies practical resource constraints--my machine takes 14 seconds to execute the 2 million assertions hidden within this test, indicating a degree of exhaustiveness which won't scale to the entire language.)