chore: Update some dev-dependencies#2895
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2895 +/- ##
==========================================
- Coverage 93.43% 93.39% -0.04%
==========================================
Files 124 124
Lines 36178 36178
Branches 36178 36178
==========================================
- Hits 33802 33790 -12
- Misses 1527 1539 +12
Partials 849 849
|
|
Yeah, please update your license in a point release. |
|
Sorry, @mxinden, more changes. |
|
Blocked until |
|
| Branch | chore-update-dev-deps |
| Testbed | On-prem |
🚨 1 Alert
| Benchmark | Measure Units | View | Benchmark Result (Result Δ%) | Upper Boundary (Limit %) |
|---|---|---|---|---|
| neqo vs. neqo (reno, paced) | Latency milliseconds (ms) | 📈 plot 🚷 threshold 🚨 alert (🔔) | 96.17 ms(+2.99%)Baseline: 93.38 ms | 95.84 ms (100.35%) |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (reno, paced) | 📈 view plot 🚷 view threshold 🚨 view alert (🔔) | 96.17 ms(+2.99%)Baseline: 93.38 ms | 95.84 ms (100.35%) |
|
| Branch | chore-update-dev-deps |
| Testbed | On-prem |
🚨 1 Alert
| Benchmark | Measure Units | View | Benchmark Result (Result Δ%) | Upper Boundary (Limit %) |
|---|---|---|---|---|
| coalesce_acked_from_zero 1000+1 entries | Latency nanoseconds (ns) | 📈 plot 🚷 threshold 🚨 alert (🔔) | 96.72 ns(+7.31%)Baseline: 90.13 ns | 94.68 ns (102.16%) |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result nanoseconds (ns) (Result Δ%) | Upper Boundary nanoseconds (ns) (Limit %) |
|---|---|---|---|
| 1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client | 📈 view plot 🚷 view threshold | 209,530,000.00 ns(+1.26%)Baseline: 206,914,076.43 ns | 216,797,228.92 ns (96.65%) |
| 1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client | 📈 view plot 🚷 view threshold | 202,780,000.00 ns(+0.90%)Baseline: 200,967,579.62 ns | 211,651,027.73 ns (95.81%) |
| 1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client | 📈 view plot 🚷 view threshold | 38,655,000.00 ns(+24.86%)Baseline: 30,959,894.90 ns | 41,425,392.22 ns (93.31%) |
| 1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client | 📈 view plot 🚷 view threshold | 286,380,000.00 ns(-1.80%)Baseline: 291,632,229.30 ns | 304,174,590.98 ns (94.15%) |
| 1-streams/each-1000-bytes/simulated-time | 📈 view plot 🚷 view threshold | 118,980,000.00 ns(+0.29%)Baseline: 118,635,668.79 ns | 120,748,756.95 ns (98.54%) |
| 1-streams/each-1000-bytes/wallclock-time | 📈 view plot 🚷 view threshold | 588,390.00 ns(-0.82%)Baseline: 593,254.49 ns | 616,627.18 ns (95.42%) |
| 1000-streams/each-1-bytes/simulated-time | 📈 view plot 🚷 view threshold | 2,331,200,000.00 ns(-84.02%)Baseline: 14,592,589,171.97 ns | 19,792,052,100.27 ns (11.78%) |
| 1000-streams/each-1-bytes/wallclock-time | 📈 view plot 🚷 view threshold | 12,729,000.00 ns(-8.71%)Baseline: 13,943,009.55 ns | 14,976,715.33 ns (84.99%) |
| 1000-streams/each-1000-bytes/simulated-time | 📈 view plot 🚷 view threshold | 16,235,000,000.00 ns(-14.02%)Baseline: 18,882,582,802.55 ns | 19,971,355,323.91 ns (81.29%) |
| 1000-streams/each-1000-bytes/wallclock-time | 📈 view plot 🚷 view threshold | 50,762,000.00 ns(+0.26%)Baseline: 50,631,066.88 ns | 57,327,841.12 ns (88.55%) |
| RxStreamOrderer::inbound_frame() | 📈 view plot 🚷 view threshold | 109,630,000.00 ns(-0.06%)Baseline: 109,694,968.15 ns | 111,642,565.15 ns (98.20%) |
| coalesce_acked_from_zero 1+1 entries | 📈 view plot 🚷 view threshold | 89.38 ns(+0.67%)Baseline: 88.78 ns | 89.88 ns (99.43%) |
| coalesce_acked_from_zero 10+1 entries | 📈 view plot 🚷 view threshold | 105.38 ns(-0.63%)Baseline: 106.04 ns | 107.15 ns (98.35%) |
| coalesce_acked_from_zero 1000+1 entries | 📈 view plot 🚷 view threshold 🚨 view alert (🔔) | 96.72 ns(+7.31%)Baseline: 90.13 ns | 94.68 ns (102.16%) |
| coalesce_acked_from_zero 3+1 entries | 📈 view plot 🚷 view threshold | 106.46 ns(-0.10%)Baseline: 106.56 ns | 107.61 ns (98.93%) |
| decode 1048576 bytes, mask 3f | 📈 view plot 🚷 view threshold | 1,590,600.00 ns(-2.14%)Baseline: 1,625,430.57 ns | 1,785,535.37 ns (89.08%) |
| decode 1048576 bytes, mask 7f | 📈 view plot 🚷 view threshold | 5,058,500.00 ns(-0.13%)Baseline: 5,065,270.06 ns | 5,109,877.03 ns (98.99%) |
| decode 1048576 bytes, mask ff | 📈 view plot 🚷 view threshold | 3,032,400.00 ns(+0.11%)Baseline: 3,029,214.65 ns | 3,046,952.23 ns (99.52%) |
| decode 4096 bytes, mask 3f | 📈 view plot 🚷 view threshold | 5,545.00 ns(-24.75%)Baseline: 7,368.69 ns | 10,406.24 ns (53.29%) |
| decode 4096 bytes, mask 7f | 📈 view plot 🚷 view threshold | 19,419.00 ns(-1.94%)Baseline: 19,803.56 ns | 20,474.40 ns (94.85%) |
| decode 4096 bytes, mask ff | 📈 view plot 🚷 view threshold | 10,764.00 ns(-5.25%)Baseline: 11,360.36 ns | 12,548.09 ns (85.78%) |
| sent::Packets::take_ranges | 📈 view plot 🚷 view threshold | 4,521.90 ns(-4.36%)Baseline: 4,727.93 ns | 4,964.95 ns (91.08%) |
| transfer/pacing-false/same-seed/simulated-time/run | 📈 view plot 🚷 view threshold | 25,234,000,000.00 ns(-0.76%)Baseline: 25,427,708,333.33 ns | 26,062,547,516.87 ns (96.82%) |
| transfer/pacing-false/same-seed/wallclock-time/run | 📈 view plot 🚷 view threshold | 24,822,000.00 ns(-3.68%)Baseline: 25,770,669.87 ns | 27,086,732.91 ns (91.64%) |
| transfer/pacing-false/varying-seeds/simulated-time/run | 📈 view plot 🚷 view threshold | 25,204,000,000.00 ns(+0.12%)Baseline: 25,173,692,307.69 ns | 25,221,887,699.00 ns (99.93%) |
| transfer/pacing-false/varying-seeds/wallclock-time/run | 📈 view plot 🚷 view threshold | 24,767,000.00 ns(-4.12%)Baseline: 25,830,099.36 ns | 27,470,469.43 ns (90.16%) |
| transfer/pacing-true/same-seed/simulated-time/run | 📈 view plot 🚷 view threshold | 25,301,000,000.00 ns(-1.19%)Baseline: 25,604,955,128.21 ns | 25,845,138,755.69 ns (97.89%) |
| transfer/pacing-true/same-seed/wallclock-time/run | 📈 view plot 🚷 view threshold | 26,444,000.00 ns(-2.41%)Baseline: 27,098,355.77 ns | 28,638,661.98 ns (92.34%) |
| transfer/pacing-true/varying-seeds/simulated-time/run | 📈 view plot 🚷 view threshold | 25,037,000,000.00 ns(+0.17%)Baseline: 24,994,192,307.69 ns | 25,042,281,762.10 ns (99.98%) |
| transfer/pacing-true/varying-seeds/wallclock-time/run | 📈 view plot 🚷 view threshold | 25,811,000.00 ns(-1.98%)Baseline: 26,332,381.41 ns | 28,058,654.15 ns (91.99%) |
|
For the record, tokio-rs/tokio#7621 merged, which should unblock the |
ACK. I think we still need to wait until they cut a release. |
Signed-off-by: Lars Eggert <[email protected]>
There was a problem hiding this comment.
Pull Request Overview
Updates dev-dependencies across multiple neqo crates by upgrading the criterion benchmarking library from version 0.6 to 0.7, and adds a license exception for the libfuzzer-sys crate.
- Upgrades criterion from 0.6 to 0.7 across all neqo crates for benchmarking
- Adds NCSA license exception for libfuzzer-sys 0.4 in .deny.toml
Reviewed Changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| neqo-transport/Cargo.toml | Updates criterion dev-dependency to version 0.7 |
| neqo-http3/Cargo.toml | Updates criterion dev-dependency to version 0.7 |
| neqo-common/Cargo.toml | Updates criterion dev-dependency to version 0.7 |
| neqo-bin/Cargo.toml | Updates criterion dev-dependency to version 0.7 |
| .deny.toml | Adds NCSA license exception for libfuzzer-sys crate |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
| Branch | chore-update-dev-deps |
| Testbed | On-prem |
🚨 2 Alerts
| Iteration | Benchmark | Measure Units | View | Benchmark Result (Result Δ%) | Upper Boundary (Limit %) |
|---|---|---|---|---|---|
| 2 | neqo vs. google (cubic, paced) | Latency milliseconds (ms) | 📈 plot 🚷 threshold 🚨 alert (🔔) | 779.81 ms(+2.79%)Baseline: 758.67 ms | 771.03 ms (101.14%) |
| 9 | neqo vs. s2n (cubic, paced) | Latency milliseconds (ms) | 📈 plot 🚷 threshold 🚨 alert (🔔) | 224.68 ms(+1.61%)Baseline: 221.11 ms | 224.12 ms (100.25%) |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| google vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 281.50 ms(+1.23%)Baseline: 278.08 ms | 282.00 ms (99.82%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| msquic vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 222.31 ms(+12.28%)Baseline: 197.99 ms | 235.32 ms (94.47%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. google (cubic, paced) | 📈 view plot 🚷 view threshold 🚨 view alert (🔔) | 779.81 ms(+2.79%)Baseline: 758.67 ms | 771.03 ms (101.14%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. msquic (cubic, paced) | 📈 view plot 🚷 view threshold | 156.26 ms(-0.99%)Baseline: 157.83 ms | 160.77 ms (97.20%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (cubic) | 📈 view plot 🚷 view threshold | 94.57 ms(+3.62%)Baseline: 91.26 ms | 96.44 ms (98.07%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 93.89 ms(+1.32%)Baseline: 92.67 ms | 97.85 ms (95.95%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (reno) | 📈 view plot 🚷 view threshold | 92.83 ms(+1.64%)Baseline: 91.32 ms | 96.48 ms (96.21%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (reno, paced) | 📈 view plot 🚷 view threshold | 94.05 ms(+1.64%)Baseline: 92.53 ms | 97.44 ms (96.52%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. quiche (cubic, paced) | 📈 view plot 🚷 view threshold | 192.29 ms(-0.71%)Baseline: 193.66 ms | 197.02 ms (97.60%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. s2n (cubic, paced) | 📈 view plot 🚷 view threshold 🚨 view alert (🔔) | 224.68 ms(+1.61%)Baseline: 221.11 ms | 224.12 ms (100.25%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| quiche vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 154.09 ms(+0.72%)Baseline: 152.99 ms | 158.15 ms (97.44%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| s2n vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 172.91 ms(-0.61%)Baseline: 173.97 ms | 178.07 ms (97.10%) |
mxinden
left a comment
There was a problem hiding this comment.
We can tackle updating windows-sys in Firefox. Though I would like to plan this work well, given that it is a >2 weeks project. Is it worth it?
|
Let's wait until Fx deps have been uplifted. |
Signed-off-by: Lars Eggert <[email protected]>
Failed Interop TestsQUIC Interop Runner, client vs. server, differences relative to 207d309. neqo-latest as client
neqo-latest as server
All resultsSucceeded Interop TestsQUIC Interop Runner, client vs. server neqo-latest as client
neqo-latest as server
Unsupported Interop TestsQUIC Interop Runner, client vs. server neqo-latest as client
neqo-latest as server
|
Client/server transfer resultsPerformance differences relative to 207d309. Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.
Download data for |
Pull Request is not mergeable
Benchmark resultsPerformance differences relative to 207d309. 1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: 💔 Performance has regressed. time: [202.34 ms 202.78 ms 203.24 ms]
thrpt: [492.04 MiB/s 493.16 MiB/s 494.22 MiB/s]
change:
time: [+1.5351% +1.8522% +2.1941%] (p = 0.00 < 0.05)
thrpt: [−2.1470% −1.8185% −1.5119%]
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected. time: [284.52 ms 286.38 ms 288.31 ms]
thrpt: [34.685 Kelem/s 34.918 Kelem/s 35.146 Kelem/s]
change:
time: [−1.2149% −0.3127% +0.5373%] (p = 0.48 > 0.05)
thrpt: [−0.5344% +0.3137% +1.2299%]
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected. time: [38.481 ms 38.655 ms 38.851 ms]
thrpt: [25.740 B/s 25.870 B/s 25.987 B/s]
change:
time: [−0.7463% −0.1060% +0.5311%] (p = 0.75 > 0.05)
thrpt: [−0.5283% +0.1061% +0.7519%]
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: 💔 Performance has regressed. time: [209.22 ms 209.53 ms 209.91 ms]
thrpt: [476.39 MiB/s 477.25 MiB/s 477.97 MiB/s]
change:
time: [+3.2362% +3.4435% +3.6653%] (p = 0.00 < 0.05)
thrpt: [−3.5357% −3.3289% −3.1348%]
decode 4096 bytes, mask ff: No change in performance detected. time: [10.731 µs 10.764 µs 10.805 µs]
change: [−0.1967% +0.2092% +0.6887%] (p = 0.36 > 0.05)
decode 1048576 bytes, mask ff: No change in performance detected. time: [3.0229 ms 3.0324 ms 3.0437 ms]
change: [−0.5335% −0.0532% +0.4275%] (p = 0.85 > 0.05)
decode 4096 bytes, mask 7f: No change in performance detected. time: [19.364 µs 19.419 µs 19.479 µs]
change: [−1.1690% −0.3494% +0.2087%] (p = 0.39 > 0.05)
decode 1048576 bytes, mask 7f: No change in performance detected. time: [5.0467 ms 5.0585 ms 5.0715 ms]
change: [−0.8967% −0.2806% +0.2233%] (p = 0.37 > 0.05)
decode 4096 bytes, mask 3f: No change in performance detected. time: [5.5258 µs 5.5450 µs 5.5707 µs]
change: [−2.0827% −0.6920% +0.2939%] (p = 0.31 > 0.05)
decode 1048576 bytes, mask 3f: No change in performance detected. time: [1.5851 ms 1.5906 ms 1.5975 ms]
change: [−0.8844% −0.2802% +0.3256%] (p = 0.36 > 0.05)
1-streams/each-1000-bytes/wallclock-time: Change within noise threshold. time: [586.56 µs 588.39 µs 590.52 µs]
change: [+0.0376% +0.5694% +1.0926%] (p = 0.04 < 0.05)
1000-streams/each-1-bytes/wallclock-time: No change in performance detected. time: [12.689 ms 12.729 ms 12.770 ms]
change: [−0.0230% +0.4414% +0.8961%] (p = 0.06 > 0.05)
1000-streams/each-1000-bytes/wallclock-time: No change in performance detected. time: [50.613 ms 50.762 ms 50.942 ms]
change: [−0.0895% +0.2649% +0.6636%] (p = 0.18 > 0.05)
coalesce_acked_from_zero 1+1 entries: No change in performance detected. time: [89.093 ns 89.376 ns 89.656 ns]
change: [−0.4544% +0.1710% +0.9538%] (p = 0.67 > 0.05)
coalesce_acked_from_zero 3+1 entries: No change in performance detected. time: [106.14 ns 106.46 ns 106.80 ns]
change: [−0.9172% −0.2699% +0.3084%] (p = 0.41 > 0.05)
coalesce_acked_from_zero 10+1 entries: No change in performance detected. time: [105.23 ns 105.38 ns 105.64 ns]
change: [−0.7404% +0.8435% +3.2317%] (p = 0.61 > 0.05)
coalesce_acked_from_zero 1000+1 entries: No change in performance detected. time: [91.492 ns 96.723 ns 108.13 ns]
change: [−0.1746% +2.3559% +8.1407%] (p = 0.26 > 0.05)
RxStreamOrderer::inbound_frame(): Change within noise threshold. time: [109.45 ms 109.63 ms 109.91 ms]
change: [−0.7475% −0.5147% −0.2524%] (p = 0.00 < 0.05)
sent::Packets::take_ranges: :green_heart: Performance has improved. time: [4.4172 µs 4.5219 µs 4.6260 µs]
change: [−8.9389% −4.9730% −1.1980%] (p = 0.01 < 0.05)
transfer/pacing-false/varying-seeds/wallclock-time/run: Change within noise threshold. time: [24.723 ms 24.767 ms 24.815 ms]
change: [−2.4341% −2.1572% −1.9028%] (p = 0.00 < 0.05)
transfer/pacing-false/varying-seeds/simulated-time/run: No change in performance detected. time: [25.165 s 25.204 s 25.243 s]
thrpt: [162.26 KiB/s 162.51 KiB/s 162.76 KiB/s]
change:
time: [−0.2370% −0.0283% +0.1912%] (p = 0.79 > 0.05)
thrpt: [−0.1909% +0.0283% +0.2376%]
transfer/pacing-true/varying-seeds/wallclock-time/run: Change within noise threshold. time: [25.750 ms 25.811 ms 25.871 ms]
change: [+0.3162% +0.6643% +0.9833%] (p = 0.00 < 0.05)
transfer/pacing-true/varying-seeds/simulated-time/run: No change in performance detected. time: [24.999 s 25.037 s 25.073 s]
thrpt: [163.36 KiB/s 163.60 KiB/s 163.84 KiB/s]
change:
time: [−0.0212% +0.1864% +0.4043%] (p = 0.08 > 0.05)
thrpt: [−0.4027% −0.1861% +0.0212%]
transfer/pacing-false/same-seed/wallclock-time/run: Change within noise threshold. time: [24.794 ms 24.822 ms 24.865 ms]
change: [−2.0591% −1.8970% −1.6986%] (p = 0.00 < 0.05)
transfer/pacing-false/same-seed/simulated-time/run: No change in performance detected. time: [25.234 s 25.234 s 25.234 s]
thrpt: [162.32 KiB/s 162.32 KiB/s 162.32 KiB/s]
change:
time: [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
thrpt: [+0.0000% +0.0000% +0.0000%]
transfer/pacing-true/same-seed/wallclock-time/run: Change within noise threshold. time: [26.409 ms 26.444 ms 26.494 ms]
change: [−2.0428% −1.8832% −1.6609%] (p = 0.00 < 0.05)
transfer/pacing-true/same-seed/simulated-time/run: No change in performance detected. time: [25.301 s 25.301 s 25.301 s]
thrpt: [161.89 KiB/s 161.89 KiB/s 161.89 KiB/s]
change:
time: [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
thrpt: [+0.0000% +0.0000% +0.0000%]
Download data for |
No description provided.