Open
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | eb02b202c281188e9187d3c292cc223107d35a25 |
| Compare | 6dc320ed049427c93a6d04c51494d1072c376d62 |
| Diff | Diff |
Regressions in System.Globalization.Tests.StringSearch
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IsPrefix_FirstHalf - Duration of single invocation | 206.66 ns | 229.92 ns | 1.11 | 0.01 | False | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 157.66 ns | 181.28 ns | 1.15 | 0.01 | False | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 157.44 ns | 181.38 ns | 1.15 | 0.01 | False | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 157.45 ns | 181.39 ns | 1.15 | 0.01 | False | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 206.70 ns | 229.91 ns | 1.11 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'Related Issues
Regressions
Improvements
Payloads
Histogram
Edge Detector Info
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, False))
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 229.92249249794838 > 216.9886323646198.
IsChangePoint: Marked as a change because one of 8/15/2022 5:23:15 PM, 8/17/2022 6:09:21 PM, 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -511.1758886379049 (T) = (0 -229.91438151755705) / Math.Sqrt((0.09403051199595515 / (46)) + (0.00013157600581682603 / (2))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (2) - 2, .025) and -0.11374256555709585 = (206.43404376176778 - 229.91438151755705) / 206.43404376176778 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, False))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 181.2754494076922 > 165.27447788978907.
IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -587.1745781288312 (T) = (0 -181.2439522304857) / Math.Sqrt((0.03207028389040226 / (49)) + (0.0019841443439575675 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1513574089450762 = (157.41762794278563 - 181.2439522304857) / 157.41762794278563 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 181.37783263034405 > 165.30027593373453.
IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -206.95138528770087 (T) = (0 -181.49162825910662) / Math.Sqrt((0.02835066301893864 / (49)) + (0.025898890250939163 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1529053477261196 = (157.42110019444647 - 181.49162825910662) / 157.42110019444647 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, False))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 181.38782301127938 > 165.30941049594205.
IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -1069.165213210168 (T) = (0 -181.38718382011413) / Math.Sqrt((0.024614597945961592 / (49)) + (8.171306914636648E-07 / (2))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (2) - 2, .025) and -0.1522914146690745 = (157.41433244315766 - 181.38718382011413) / 157.41433244315766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreCase, False))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 229.9117129107842 > 216.87844739024908.
IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -197.4211601971299 (T) = (0 -230.00988660654446) / Math.Sqrt((0.23150453470920937 / (50)) + (0.01927614907845868 / (2))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (2) - 2, .025) and -0.11423760004232615 = (206.42804245504473 - 230.00988660654446) / 206.42804245504473 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository