Conversation
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8225) and master.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 189.70 ± (189.65 - 190.27) ms | 206.90 ± (206.77 - 207.72) ms | +9.1% | ❌⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 193.34 ± (193.23 - 193.63) ms | 210.82 ± (210.58 - 211.34) ms | +9.0% | ❌⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1139.41 ± (1139.56 - 1146.95) ms | 1208.59 ± (1207.31 - 1213.22) ms | +6.1% | ❌⬆️ |
Full Metrics Comparison
FakeDbCommand
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 71.47 ± (71.49 - 71.85) ms | 72.07 ± (72.17 - 72.54) ms | +0.8% | ✅⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 75.52 ± (75.52 - 75.83) ms | 75.60 ± (75.56 - 75.88) ms | +0.1% | ✅⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1059.28 ± (1061.24 - 1066.75) ms | 1061.18 ± (1062.33 - 1067.86) ms | +0.2% | ✅⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 22.47 ± (22.43 - 22.50) ms | 22.31 ± (22.28 - 22.34) ms | -0.7% | ✅ |
| process.time_to_main_ms | 83.44 ± (83.23 - 83.65) ms | 83.35 ± (83.16 - 83.54) ms | -0.1% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.90 ± (10.90 - 10.91) MB | 10.90 ± (10.90 - 10.91) MB | +0.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 12 ± (12 - 12) | 12 ± (12 - 12) | +0.0% | ✅ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 22.45 ± (22.41 - 22.49) ms | 22.34 ± (22.30 - 22.37) ms | -0.5% | ✅ |
| process.time_to_main_ms | 85.06 ± (84.84 - 85.27) ms | 85.38 ± (85.17 - 85.60) ms | +0.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.87 ± (10.87 - 10.88) MB | 10.94 ± (10.93 - 10.94) MB | +0.6% | ✅⬆️ |
| runtime.dotnet.threads.count | 13 ± (13 - 13) | 13 ± (13 - 13) | +0.0% | ✅ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 224.72 ± (223.60 - 225.85) ms | 224.90 ± (223.80 - 226.00) ms | +0.1% | ✅⬆️ |
| process.time_to_main_ms | 518.20 ± (517.14 - 519.25) ms | 515.42 ± (514.32 - 516.52) ms | -0.5% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 47.77 ± (47.74 - 47.80) MB | 47.77 ± (47.73 - 47.81) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.3% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 21.08 ± (21.05 - 21.12) ms | 21.08 ± (21.05 - 21.11) ms | -0.0% | ✅ |
| process.time_to_main_ms | 72.62 ± (72.46 - 72.78) ms | 72.78 ± (72.61 - 72.96) ms | +0.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.60 ± (10.60 - 10.61) MB | 10.62 ± (10.62 - 10.62) MB | +0.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 20.90 ± (20.86 - 20.93) ms | 20.87 ± (20.83 - 20.91) ms | -0.1% | ✅ |
| process.time_to_main_ms | 73.09 ± (72.92 - 73.25) ms | 73.17 ± (72.99 - 73.34) ms | +0.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.72 ± (10.71 - 10.72) MB | 10.73 ± (10.73 - 10.73) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 388.37 ± (386.55 - 390.19) ms | 385.19 ± (383.20 - 387.18) ms | -0.8% | ✅ |
| process.time_to_main_ms | 516.60 ± (515.62 - 517.58) ms | 517.29 ± (516.41 - 518.18) ms | +0.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 49.89 ± (49.87 - 49.92) MB | 49.99 ± (49.96 - 50.01) MB | +0.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | -0.1% | ✅ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 19.16 ± (19.12 - 19.19) ms | 19.22 ± (19.18 - 19.25) ms | +0.3% | ✅⬆️ |
| process.time_to_main_ms | 71.47 ± (71.30 - 71.64) ms | 71.39 ± (71.23 - 71.56) ms | -0.1% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.67 ± (7.66 - 7.67) MB | 7.67 ± (7.66 - 7.68) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 19.18 ± (19.15 - 19.22) ms | 19.22 ± (19.18 - 19.26) ms | +0.2% | ✅⬆️ |
| process.time_to_main_ms | 72.54 ± (72.37 - 72.71) ms | 72.11 ± (72.00 - 72.21) ms | -0.6% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.72 ± (7.71 - 7.72) MB | 7.72 ± (7.71 - 7.72) MB | +0.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 306.20 ± (304.03 - 308.38) ms | 302.91 ± (300.74 - 305.08) ms | -1.1% | ✅ |
| process.time_to_main_ms | 478.08 ± (477.34 - 478.82) ms | 479.87 ± (479.20 - 480.53) ms | +0.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 36.98 ± (36.96 - 37.00) MB | 36.90 ± (36.87 - 36.93) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 27 ± (27 - 27) | 27 ± (27 - 27) | +0.1% | ✅⬆️ |
HttpMessageHandler
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 189.70 ± (189.65 - 190.27) ms | 206.90 ± (206.77 - 207.72) ms | +9.1% | ❌⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 193.34 ± (193.23 - 193.63) ms | 210.82 ± (210.58 - 211.34) ms | +9.0% | ❌⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1139.41 ± (1139.56 - 1146.95) ms | 1208.59 ± (1207.31 - 1213.22) ms | +6.1% | ❌⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 185.24 ± (185.01 - 185.47) ms | 204.73 ± (204.26 - 205.20) ms | +10.5% | ✅⬆️ |
| process.time_to_main_ms | 79.23 ± (79.06 - 79.40) ms | 88.64 ± (88.39 - 88.89) ms | +11.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.13 ± (16.10 - 16.15) MB | 15.96 ± (15.95 - 15.98) MB | -1.0% | ✅ |
| runtime.dotnet.threads.count | 20 ± (19 - 20) | 20 ± (20 - 20) | +1.1% | ✅⬆️ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 184.33 ± (184.18 - 184.47) ms | 203.28 ± (202.79 - 203.77) ms | +10.3% | ✅⬆️ |
| process.time_to_main_ms | 80.45 ± (80.35 - 80.54) ms | 89.13 ± (88.88 - 89.38) ms | +10.8% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.80 ± (15.66 - 15.93) MB | 16.00 ± (15.98 - 16.02) MB | +1.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 20 ± (20 - 20) | 21 ± (21 - 21) | +5.0% | ✅⬆️ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 388.94 ± (387.40 - 390.49) ms | 414.73 ± (413.47 - 415.99) ms | +6.6% | ✅⬆️ |
| process.time_to_main_ms | 499.49 ± (498.34 - 500.64) ms | 544.46 ± (543.23 - 545.68) ms | +9.0% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 57.45 ± (57.24 - 57.67) MB | 58.70 ± (58.66 - 58.75) MB | +2.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 30 ± (29 - 30) | 30 ± (30 - 30) | +1.7% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 189.37 ± (189.12 - 189.62) ms | 208.01 ± (207.59 - 208.43) ms | +9.8% | ✅⬆️ |
| process.time_to_main_ms | 69.01 ± (68.89 - 69.14) ms | 76.35 ± (76.09 - 76.61) ms | +10.6% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.18 ± (16.02 - 16.33) MB | 16.26 ± (16.23 - 16.29) MB | +0.5% | ✅⬆️ |
| runtime.dotnet.threads.count | 18 ± (18 - 18) | 20 ± (19 - 20) | +8.1% | ✅⬆️ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 188.39 ± (188.24 - 188.55) ms | 207.26 ± (206.76 - 207.77) ms | +10.0% | ✅⬆️ |
| process.time_to_main_ms | 69.80 ± (69.76 - 69.85) ms | 77.73 ± (77.49 - 77.98) ms | +11.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.79 ± (15.62 - 15.96) MB | 16.26 ± (16.24 - 16.29) MB | +3.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 19 ± (18 - 19) | 21 ± (20 - 21) | +10.8% | ✅⬆️ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 598.97 ± (596.47 - 601.48) ms | 597.57 ± (594.70 - 600.45) ms | -0.2% | ✅ |
| process.time_to_main_ms | 503.12 ± (502.25 - 503.98) ms | 543.60 ± (542.04 - 545.16) ms | +8.0% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 61.56 ± (61.46 - 61.66) MB | 61.60 ± (61.48 - 61.72) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 30 ± (30 - 30) | 31 ± (31 - 31) | +1.2% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 187.70 ± (187.47 - 187.93) ms | 205.21 ± (204.81 - 205.61) ms | +9.3% | ✅⬆️ |
| process.time_to_main_ms | 68.47 ± (68.33 - 68.61) ms | 75.87 ± (75.59 - 76.15) ms | +10.8% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.58 ± (11.47 - 11.69) MB | 11.65 ± (11.63 - 11.67) MB | +0.6% | ✅⬆️ |
| runtime.dotnet.threads.count | 17 ± (17 - 18) | 19 ± (19 - 19) | +8.6% | ✅⬆️ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 186.78 ± (186.58 - 186.97) ms | 204.97 ± (204.51 - 205.43) ms | +9.7% | ✅⬆️ |
| process.time_to_main_ms | 69.38 ± (69.33 - 69.44) ms | 77.16 ± (76.95 - 77.37) ms | +11.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.76 ± (11.69 - 11.84) MB | 11.68 ± (11.66 - 11.69) MB | -0.7% | ✅ |
| runtime.dotnet.threads.count | 19 ± (18 - 19) | 20 ± (20 - 20) | +7.2% | ✅⬆️ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 519.29 ± (516.15 - 522.43) ms | 524.81 ± (520.00 - 529.63) ms | +1.1% | ✅⬆️ |
| process.time_to_main_ms | 462.40 ± (461.78 - 463.01) ms | 503.12 ± (502.10 - 504.14) ms | +8.8% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 50.71 ± (50.68 - 50.74) MB | 50.69 ± (50.63 - 50.76) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 30 ± (30 - 30) | 30 ± (30 - 30) | +0.9% | ✅⬆️ |
Comparison explanation
Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
- Welch test with statistical test for significance of 5%
- Only results indicating a difference greater than 5% and 5 ms are considered.
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8225) - mean (72ms) : 70, 75
master - mean (72ms) : 69, 74
section Bailout
This PR (8225) - mean (76ms) : 74, 77
master - mean (76ms) : 74, 77
section CallTarget+Inlining+NGEN
This PR (8225) - mean (1,065ms) : 1025, 1105
master - mean (1,064ms) : 1025, 1103
FakeDbCommand (.NET Core 3.1)
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8225) - mean (113ms) : 108, 117
master - mean (113ms) : 109, 116
section Bailout
This PR (8225) - mean (114ms) : 111, 117
master - mean (114ms) : 111, 117
section CallTarget+Inlining+NGEN
This PR (8225) - mean (777ms) : 758, 796
master - mean (779ms) : 756, 801
FakeDbCommand (.NET 6)
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8225) - mean (100ms) : 97, 103
master - mean (100ms) : 97, 103
section Bailout
This PR (8225) - mean (100ms) : 98, 103
master - mean (100ms) : 97, 103
section CallTarget+Inlining+NGEN
This PR (8225) - mean (930ms) : 897, 964
master - mean (933ms) : 896, 971
FakeDbCommand (.NET 8)
gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8225) - mean (98ms) : 95, 101
master - mean (98ms) : 95, 101
section Bailout
This PR (8225) - mean (99ms) : 97, 100
master - mean (99ms) : 97, 102
section CallTarget+Inlining+NGEN
This PR (8225) - mean (812ms) : 780, 844
master - mean (815ms) : 777, 852
HttpMessageHandler (.NET Framework 4.8)
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8225) - mean (207ms) : 202, 213
master - mean (190ms) : 187, 193
section Bailout
This PR (8225) - mean (211ms) : crit, 207, 215
master - mean (193ms) : 192, 195
section CallTarget+Inlining+NGEN
This PR (8225) - mean (1,210ms) : crit, 1167, 1254
master - mean (1,143ms) : 1090, 1197
HttpMessageHandler (.NET Core 3.1)
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8225) - mean (303ms) : 295, 310
master - mean (272ms) : 268, 276
section Bailout
This PR (8225) - mean (301ms) : crit, 295, 308
master - mean (273ms) : 270, 275
section CallTarget+Inlining+NGEN
This PR (8225) - mean (997ms) : crit, 970, 1024
master - mean (919ms) : 899, 939
HttpMessageHandler (.NET 6)
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8225) - mean (293ms) : 287, 300
master - mean (266ms) : 263, 270
section Bailout
This PR (8225) - mean (294ms) : crit, 287, 301
master - mean (266ms) : 264, 268
section CallTarget+Inlining+NGEN
This PR (8225) - mean (1,173ms) : 1139, 1206
master - mean (1,134ms) : 1097, 1170
HttpMessageHandler (.NET 8)
gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8225) - mean (292ms) : 285, 299
master - mean (266ms) : 262, 269
section Bailout
This PR (8225) - mean (293ms) : crit, 285, 300
master - mean (265ms) : 263, 267
section CallTarget+Inlining+NGEN
This PR (8225) - mean (1,062ms) : 989, 1136
master - mean (1,009ms) : 944, 1074
BenchmarksBenchmark execution time: 2026-04-03 13:47:26 Comparing candidate commit 0230f16 in PR branch Found 7 performance improvements and 10 performance regressions! Performance is the same for 262 metrics, 9 unstable metrics.
|
Fixing tests and tooling
Summary of changes
Generate reference chain at type level
Reason for change
Last step to help pointing out memory leaks root cause (i.e. which root is responsible for keeping instances alive)
Implementation details
Listen to GC_BULK_ROOT_STATIC_VAR and GC_BULK_ROOT_EDGE to get the different kinds of root. Next, traverse the reference fields down to a leak (i.e. non reference object). The reference chain is kept at the type level to avoid consumming to much memory and serialized in .json format to be sent with profile data.
Test coverage
Unit tests and integration tests have been added.
Other details