Investigate Tensor's Helpers.IsEqualWithTolerance helper

[stephentoub]

The System.Numerics.Tensors test suite has a Helpers.IsEqualWithTolerance helper method. This is letting through as equal some values which should not be considered equal, in particular when one of the values is infinity or negative infinity. This is one of the reasons we didn't notice the bug cited by #107838, because a value close to 0 and negative infinity were being considered equal. But when updating that helper to accomodate, other existing tests started failing. It's possible those are false positives, but it's also possible we have further edge case bugs to be fixed. We need to investigate.

[stephentoub]

@jeffhandley, this is what I was referring to earlier today offline.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-numerics-tensors
See info in area-owners.md if you want to be subscribed.

[jeffhandley]

Thanks, @stephentoub. We should investigate this during the .NET 9 GA milestone and verify correctness of test cases that were slipping through coverage before.

[tannergooding]

A valid implementation would be https://github.com/dotnet/runtime/blob/main/src/libraries/Common/tests/TestUtilities/System/AssertExtensions.cs#L818-L1037

This ensures that values such as inf and nan are checked exactly, while finite values are within an expected variance amount. It does so in a way that minimizes additional error and which can account for per input result differences.

[jeffhandley]

@tannergooding -- I'm punting this to 10.0.0. When we address it, if we find there were lurking correctness bugs, we can consider backporting the fixes to .NET 9 in servicing.