ci: Don't depend on cached QNS results by larseggert · Pull Request #3123 · mozilla/neqo

larseggert · 2025-11-10T13:40:53Z

Because we have way too much cache pressure, main-branch QNS results are usually missing these days. So we don't see if a PR breaks something. Fix that by always running QNS on the main branch, too.

codecov · 2025-11-10T13:46:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.43%. Comparing base (f9cbd20) to head (0a19e68).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3123      +/-   ##
==========================================
- Coverage   93.50%   93.43%   -0.08%     
==========================================
  Files         125      125              
  Lines       36575    36575              
  Branches    36575    36575              
==========================================
- Hits        34199    34173      -26     
- Misses       1529     1556      +27     
+ Partials      847      846       -1

Components	Coverage Δ
neqo-common	`97.36% <ø> (ø)`
neqo-crypto	`83.17% <ø> (-0.49%)`	⬇️
neqo-http3	`93.30% <ø> (ø)`
neqo-qpack	`94.40% <ø> (ø)`
neqo-transport	`94.53% <ø> (-0.09%)`	⬇️
neqo-udp	`79.42% <ø> (ø)`
mtu	`85.26% <ø> (-0.19%)`	⬇️

github-actions · 2025-11-10T14:32:21Z

Bencher Report

Branch	ci-qns-no-cache
Testbed	On-prem

Click to view all benchmark results

Benchmark	Latency	Benchmark Result nanoseconds (ns) (Result Δ%)	Upper Boundary nanoseconds (ns) (Limit %)
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client	📈 view plot 🚷 view threshold	205,740,000.00 ns (-0.89%) Baseline: 207,588,509.32 ns	216,854,673.65 ns (94.87%)
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client	📈 view plot 🚷 view threshold	201,550,000.00 ns (-0.10%) Baseline: 201,752,732.92 ns	211,557,213.40 ns (95.27%)
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client	📈 view plot 🚷 view threshold	38,502,000.00 ns (+14.37%) Baseline: 33,663,948.24 ns	45,705,734.12 ns (84.24%)
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client	📈 view plot 🚷 view threshold	284,180,000.00 ns (-1.84%) Baseline: 289,512,484.47 ns	302,674,037.19 ns (93.89%)
1-streams/each-1000-bytes/simulated-time	📈 view plot 🚷 view threshold	119,240,000.00 ns (+0.38%) Baseline: 118,788,778.47 ns	120,565,290.64 ns (98.90%)
1-streams/each-1000-bytes/wallclock-time	📈 view plot 🚷 view threshold	581,040.00 ns (-1.57%) Baseline: 590,311.33 ns	611,715.89 ns (94.99%)
1000-streams/each-1-bytes/simulated-time	📈 view plot 🚷 view threshold	2,333,100,000.00 ns (-77.36%) Baseline: 10,302,978,260.87 ns	24,577,871,222.33 ns (9.49%)
1000-streams/each-1-bytes/wallclock-time	📈 view plot 🚷 view threshold	12,340,000.00 ns (-8.36%) Baseline: 13,465,884.06 ns	15,213,862.12 ns (81.11%)
1000-streams/each-1000-bytes/simulated-time	📈 view plot 🚷 view threshold	16,336,000,000.00 ns (-9.24%) Baseline: 17,999,931,677.02 ns	20,946,951,590.26 ns (77.99%)
1000-streams/each-1000-bytes/wallclock-time	📈 view plot 🚷 view threshold	49,444,000.00 ns (-2.28%) Baseline: 50,595,095.24 ns	56,052,278.68 ns (88.21%)
RxStreamOrderer::inbound_frame()	📈 view plot 🚷 view threshold	110,100,000.00 ns (+0.35%) Baseline: 109,714,761.90 ns	111,411,506.22 ns (98.82%)
coalesce_acked_from_zero 1+1 entries	📈 view plot 🚷 view threshold	89.26 ns (+0.23%) Baseline: 89.06 ns	90.35 ns (98.79%)
coalesce_acked_from_zero 10+1 entries	📈 view plot 🚷 view threshold	105.54 ns (-0.47%) Baseline: 106.04 ns	107.12 ns (98.53%)
coalesce_acked_from_zero 1000+1 entries	📈 view plot 🚷 view threshold	91.86 ns (+1.18%) Baseline: 90.79 ns	95.21 ns (96.48%)
coalesce_acked_from_zero 3+1 entries	📈 view plot 🚷 view threshold	106.26 ns (-0.26%) Baseline: 106.54 ns	107.55 ns (98.80%)
decode 1048576 bytes, mask 3f	📈 view plot 🚷 view threshold	1,759,400.00 ns (+5.17%) Baseline: 1,672,958.39 ns	1,877,540.24 ns (93.71%)
decode 1048576 bytes, mask 7f	📈 view plot 🚷 view threshold	5,037,200.00 ns (-0.46%) Baseline: 5,060,602.07 ns	5,106,555.18 ns (98.64%)
decode 1048576 bytes, mask ff	📈 view plot 🚷 view threshold	3,005,500.00 ns (-0.55%) Baseline: 3,022,109.11 ns	3,056,499.72 ns (98.33%)
decode 4096 bytes, mask 3f	📈 view plot 🚷 view threshold	6,234.50 ns (-11.44%) Baseline: 7,039.48 ns	9,851.05 ns (63.29%)
decode 4096 bytes, mask 7f	📈 view plot 🚷 view threshold	19,639.00 ns (-0.59%) Baseline: 19,755.60 ns	20,343.95 ns (96.53%)
decode 4096 bytes, mask ff	📈 view plot 🚷 view threshold	11,381.00 ns (+0.15%) Baseline: 11,364.18 ns	12,341.26 ns (92.22%)
sent::Packets::take_ranges	📈 view plot 🚷 view threshold	4,642.60 ns (-0.99%) Baseline: 4,689.26 ns	4,934.91 ns (94.08%)
transfer/pacing-false/same-seed/simulated-time/run	📈 view plot 🚷 view threshold	25,234,000,000.00 ns (-0.50%) Baseline: 25,359,648,648.65 ns	25,913,777,825.61 ns (97.38%)
transfer/pacing-false/same-seed/wallclock-time/run	📈 view plot 🚷 view threshold	25,020,000.00 ns (-1.62%) Baseline: 25,432,873.18 ns	26,993,629.87 ns (92.69%)
transfer/pacing-false/varying-seeds/simulated-time/run	📈 view plot 🚷 view threshold	25,170,000,000.00 ns (-0.04%) Baseline: 25,180,525,987.53 ns	25,232,484,282.87 ns (99.75%)
transfer/pacing-false/varying-seeds/wallclock-time/run	📈 view plot 🚷 view threshold	24,969,000.00 ns (-2.40%) Baseline: 25,583,112.27 ns	27,164,706.31 ns (91.92%)
transfer/pacing-true/same-seed/simulated-time/run	📈 view plot 🚷 view threshold	25,069,000,000.00 ns (-1.45%) Baseline: 25,438,351,351.35 ns	26,018,521,759.63 ns (96.35%)
transfer/pacing-true/same-seed/wallclock-time/run	📈 view plot 🚷 view threshold	25,213,000.00 ns (-5.47%) Baseline: 26,671,197.51 ns	28,615,147.99 ns (88.11%)
transfer/pacing-true/varying-seeds/simulated-time/run	📈 view plot 🚷 view threshold	25,006,000,000.00 ns (+0.04%) Baseline: 24,997,079,002.08 ns	25,045,685,243.90 ns (99.84%)
transfer/pacing-true/varying-seeds/wallclock-time/run	📈 view plot 🚷 view threshold	25,361,000.00 ns (-2.84%) Baseline: 26,103,436.59 ns	27,723,010.42 ns (91.48%)

🐰 View full continuous benchmarking report in Bencher

Needed for mozilla#3123. Will be a no-op until called from there, hence committing.

github-actions · 2025-11-11T07:45:49Z

Bencher Report

Branch	ci-qns-no-cache
Testbed	On-prem

Click to view all benchmark results

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
google vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	276.01 ms (-0.94%) Baseline: 278.63 ms	283.18 ms (97.47%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
msquic vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	195.99 ms (-3.40%) Baseline: 202.90 ms	242.26 ms (80.90%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. google (cubic, paced)	📈 view plot 🚷 view threshold	755.91 ms (-0.83%) Baseline: 762.23 ms	781.80 ms (96.69%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. msquic (cubic, paced)	📈 view plot 🚷 view threshold	157.62 ms (+0.05%) Baseline: 157.53 ms	160.17 ms (98.41%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. neqo (cubic)	📈 view plot 🚷 view threshold	98.83 ms (+6.63%) Baseline: 92.69 ms	99.14 ms (99.69%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	97.50 ms (+3.82%) Baseline: 93.91 ms	99.90 ms (97.60%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. neqo (reno)	📈 view plot 🚷 view threshold	96.06 ms (+3.69%) Baseline: 92.64 ms	98.97 ms (97.07%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. neqo (reno, paced)	📈 view plot 🚷 view threshold	98.26 ms (+4.68%) Baseline: 93.86 ms	100.01 ms (98.25%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. quiche (cubic, paced)	📈 view plot 🚷 view threshold	192.91 ms (-0.38%) Baseline: 193.65 ms	196.87 ms (97.99%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. s2n (cubic, paced)	📈 view plot 🚷 view threshold	221.73 ms (+0.24%) Baseline: 221.21 ms	224.18 ms (98.91%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
quiche vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	154.68 ms (+0.90%) Baseline: 153.31 ms	158.24 ms (97.75%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
s2n vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	174.94 ms (+0.74%) Baseline: 173.66 ms	177.74 ms (98.42%)

🐰 View full continuous benchmarking report in Bencher

Copilot

Pull Request Overview

This PR refactors the QNS (QUIC Network Simulator) CI workflow to eliminate dependency on cached results from the main branch. Previously, the workflow relied on cached QNS results that were frequently missing due to cache pressure, preventing proper detection of breaking changes in PRs. The solution builds and tests both the PR branch (neqo-pr) and the base branch (neqo-main) directly during PR runs, enabling immediate baseline comparisons.

Key Changes:

Removed cache-based baseline storage in favor of building both PR and main branch images on-demand for pull requests
Introduced a matrix strategy to build separate Docker images for PR and main branches
Modified result comparison logic to use freshly-built main branch results as baseline instead of cached data

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
`.github/workflows/qns.yml`	Complete workflow refactor: added branch determination job, converted docker-image to matrix build, modified implementations job to load both images, updated reporting to compare against main branch results instead of cache
`.github/actions/docker-build-qns/action.yml`	Removed unused composite action that was previously used for building QNS Docker images

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

codspeed-hq · 2025-11-11T15:31:24Z

CodSpeed Performance Report

Merging #3123 will degrade performances by 15.57%

_{Comparing larseggert:ci-qns-no-cache (09bffc5) with main (214099c)}

Summary

⚡ 2 improvements
❌ 1 regression
✅ 20 untouched

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Mode	Benchmark	`BASE`	`HEAD`	Change
⚡	Simulation	`client`	893.1 ms	830.5 ms	+7.54%
❌	Simulation	`client`	730.8 ms	865.5 ms	-15.57%
⚡	Simulation	`client`	15.9 ms	15.1 ms	+5.35%

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

Copilot

Pull Request Overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

github-actions · 2025-12-01T20:00:50Z

Benchmark results

Significant performance differences relative to be1e755.

1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: 💚 Performance has improved by -1.8906%.

       time:   [205.37 ms 205.74 ms 206.16 ms]
       thrpt:  [485.06 MiB/s 486.04 MiB/s 486.92 MiB/s]
change:
       time:   [-2.1606% -1.8906% -1.6125] (p = 0.00 < 0.05)
       thrpt:  [+1.6389% +1.9271% +2.2084]
       Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe

All results

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: No change in performance detected.

       time:   [201.06 ms 201.55 ms 202.09 ms]
       thrpt:  [494.82 MiB/s 496.16 MiB/s 497.35 MiB/s]
change:
       time:   [-0.3915% -0.0860% +0.2567] (p = 0.60 > 0.05)
       thrpt:  [-0.2561% +0.0861% +0.3930]
       No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe

1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: Change within noise threshold.

       time:   [282.41 ms 284.18 ms 285.97 ms]
       thrpt:  [34.969 Kelem/s 35.189 Kelem/s 35.410 Kelem/s]
change:
       time:   [-1.8863% -0.9555% -0.0307] (p = 0.04 < 0.05)
       thrpt:  [+0.0307% +0.9647% +1.9226]
       Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild

1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.

       time:   [38.343 ms 38.502 ms 38.679 ms]
       thrpt:  [25.854   B/s 25.973   B/s 26.081   B/s]
change:
       time:   [-0.5087% +0.0523% +0.6565] (p = 0.87 > 0.05)
       thrpt:  [-0.6522% -0.0523% +0.5113]
       No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe

1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: 💚 Performance has improved by -1.8906%.

       time:   [205.37 ms 205.74 ms 206.16 ms]
       thrpt:  [485.06 MiB/s 486.04 MiB/s 486.92 MiB/s]
change:
       time:   [-2.1606% -1.8906% -1.6125] (p = 0.00 < 0.05)
       thrpt:  [+1.6389% +1.9271% +2.2084]
       Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe

decode 4096 bytes, mask ff: No change in performance detected.

       time:   [11.337 µs 11.381 µs 11.431 µs]
       change: [-0.3800% +0.0934% +0.5869] (p = 0.70 > 0.05)
       No change in performance detected.
Found 23 outliers among 100 measurements (23.00%)
4 (4.00%) low severe
6 (6.00%) low mild
1 (1.00%) high mild
12 (12.00%) high severe

decode 1048576 bytes, mask ff: No change in performance detected.

       time:   [2.9950 ms 3.0055 ms 3.0178 ms]
       change: [-1.3820% -0.3200% +0.5106] (p = 0.57 > 0.05)
       No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
10 (10.00%) high severe

decode 4096 bytes, mask 7f: No change in performance detected.

       time:   [19.578 µs 19.639 µs 19.713 µs]
       change: [-0.3496% +0.1095% +0.6289] (p = 0.66 > 0.05)
       No change in performance detected.
Found 17 outliers among 100 measurements (17.00%)
1 (1.00%) high mild
16 (16.00%) high severe

decode 1048576 bytes, mask 7f: No change in performance detected.

       time:   [5.0263 ms 5.0372 ms 5.0489 ms]
       change: [-0.9363% -0.4229% +0.0299] (p = 0.09 > 0.05)
       No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
11 (11.00%) high severe

decode 4096 bytes, mask 3f: No change in performance detected.

       time:   [6.2100 µs 6.2345 µs 6.2671 µs]
       change: [-0.0338% +0.3081% +0.6614] (p = 0.09 > 0.05)
       No change in performance detected.
Found 12 outliers among 100 measurements (12.00%)
5 (5.00%) low mild
2 (2.00%) high mild
5 (5.00%) high severe

decode 1048576 bytes, mask 3f: No change in performance detected.

       time:   [1.7579 ms 1.7594 ms 1.7623 ms]
       change: [-0.3833% -0.0769% +0.1664] (p = 0.65 > 0.05)
       No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe

1-streams/each-1000-bytes/wallclock-time: No change in performance detected.

       time:   [579.15 µs 581.04 µs 583.21 µs]
       change: [-0.8947% -0.3570% +0.1686] (p = 0.19 > 0.05)
       No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
7 (7.00%) high severe

1-streams/each-1000-bytes/simulated-time: No change in performance detected.

       time:   [119.02 ms 119.24 ms 119.46 ms]
       thrpt:  [8.1747 KiB/s 8.1898 KiB/s 8.2050 KiB/s]
change:
       time:   [-0.0204% +0.2574% +0.5168] (p = 0.06 > 0.05)
       thrpt:  [-0.5142% -0.2567% +0.0204]
       No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
2 (2.00%) high mild

1000-streams/each-1-bytes/wallclock-time: No change in performance detected.

       time:   [12.299 ms 12.340 ms 12.383 ms]
       change: [-0.4047% +0.1031% +0.5719] (p = 0.69 > 0.05)
       No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild

1000-streams/each-1-bytes/simulated-time: No change in performance detected.

       time:   [2.3298 s 2.3331 s 2.3364 s]
       thrpt:  [428.01   B/s 428.62   B/s 429.23   B/s]
change:
       time:   [-0.1956% -0.0023% +0.1881] (p = 0.98 > 0.05)
       thrpt:  [-0.1878% +0.0023% +0.1959]
       No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

1000-streams/each-1000-bytes/wallclock-time: Change within noise threshold.

       time:   [49.327 ms 49.444 ms 49.566 ms]
       change: [-1.4714% -1.1717% -0.8418] (p = 0.00 < 0.05)
       Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

1000-streams/each-1000-bytes/simulated-time: No change in performance detected.

       time:   [16.085 s 16.336 s 16.589 s]
       thrpt:  [58.868 KiB/s 59.779 KiB/s 60.712 KiB/s]
change:
       time:   [-1.7643% +0.4507% +2.7061] (p = 0.69 > 0.05)
       thrpt:  [-2.6348% -0.4487% +1.7959]
       No change in performance detected.

coalesce_acked_from_zero 1+1 entries: No change in performance detected.

       time:   [88.949 ns 89.259 ns 89.578 ns]
       change: [-0.8386% -0.3949% +0.0342] (p = 0.08 > 0.05)
       No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe

coalesce_acked_from_zero 3+1 entries: No change in performance detected.

       time:   [105.92 ns 106.26 ns 106.60 ns]
       change: [-0.7473% -0.2373% +0.1738] (p = 0.35 > 0.05)
       No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
3 (3.00%) high mild
11 (11.00%) high severe

coalesce_acked_from_zero 10+1 entries: No change in performance detected.

       time:   [105.32 ns 105.54 ns 105.85 ns]
       change: [-0.3099% +0.2275% +0.9003] (p = 0.56 > 0.05)
       No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
2 (2.00%) low severe
3 (3.00%) low mild
1 (1.00%) high mild
5 (5.00%) high severe

coalesce_acked_from_zero 1000+1 entries: No change in performance detected.

       time:   [91.752 ns 91.859 ns 91.983 ns]
       change: [-0.6324% +0.0421% +0.6642] (p = 0.90 > 0.05)
       No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
5 (5.00%) high mild
8 (8.00%) high severe

RxStreamOrderer::inbound_frame(): No change in performance detected.

       time:   [110.01 ms 110.10 ms 110.20 ms]
       change: [-0.1850% +0.1344% +0.3643] (p = 0.40 > 0.05)
       No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) low mild
1 (1.00%) high mild
1 (1.00%) high severe

sent::Packets::take_ranges: No change in performance detected.

       time:   [4.5437 µs 4.6426 µs 4.7301 µs]
       change: [-2.2578% +1.6556% +5.4153] (p = 0.40 > 0.05)
       No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

transfer/pacing-false/varying-seeds/wallclock-time/run: No change in performance detected.

       time:   [24.921 ms 24.969 ms 25.028 ms]
       change: [-0.1993% +0.0570% +0.3362] (p = 0.68 > 0.05)
       No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) low mild
2 (2.00%) high mild
1 (1.00%) high severe

transfer/pacing-false/varying-seeds/simulated-time/run: No change in performance detected.

       time:   [25.133 s 25.170 s 25.207 s]
       thrpt:  [162.50 KiB/s 162.74 KiB/s 162.97 KiB/s]
change:
       time:   [-0.2915% -0.0776% +0.1559] (p = 0.49 > 0.05)
       thrpt:  [-0.1557% +0.0776% +0.2923]
       No change in performance detected.

transfer/pacing-true/varying-seeds/wallclock-time/run: Change within noise threshold.

       time:   [25.292 ms 25.361 ms 25.437 ms]
       change: [-1.4001% -1.0663% -0.7229] (p = 0.00 < 0.05)
       Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe

transfer/pacing-true/varying-seeds/simulated-time/run: No change in performance detected.

       time:   [24.969 s 25.006 s 25.043 s]
       thrpt:  [163.56 KiB/s 163.80 KiB/s 164.04 KiB/s]
change:
       time:   [-0.1068% +0.1164% +0.3436] (p = 0.31 > 0.05)
       thrpt:  [-0.3424% -0.1163% +0.1069]
       No change in performance detected.

transfer/pacing-false/same-seed/wallclock-time/run: Change within noise threshold.

       time:   [25.004 ms 25.020 ms 25.036 ms]
       change: [+1.1178% +1.3140% +1.4561] (p = 0.00 < 0.05)
       Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

transfer/pacing-false/same-seed/simulated-time/run: No change in performance detected.

       time:   [25.234 s 25.234 s 25.234 s]
       thrpt:  [162.32 KiB/s 162.32 KiB/s 162.32 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000]
       No change in performance detected.

transfer/pacing-true/same-seed/wallclock-time/run: Change within noise threshold.

       time:   [25.191 ms 25.213 ms 25.238 ms]
       change: [+0.3065% +0.4112% +0.5239] (p = 0.00 < 0.05)
       Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
1 (1.00%) high mild
1 (1.00%) high severe

transfer/pacing-true/same-seed/simulated-time/run: No change in performance detected.

       time:   [25.069 s 25.069 s 25.069 s]
       thrpt:  [163.39 KiB/s 163.39 KiB/s 163.39 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000]
       No change in performance detected.

Download data for profiler.firefox.com or download performance comparison data.

Because we have way too much cache pressure, `main`-branch QNS results are usually missing these days. So we don't see if a PR breaks something. Fix that by always running QNS on the `main` branch too, building both `neqo-pr` and `neqo-main` Docker images for PRs, and comparing the results directly instead of relying on cached baseline data.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

github-actions · 2025-12-10T09:00:44Z

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to main at f9cbd20.

neqo-pr as client	neqo-pr as server
neqo-pr vs. aioquic: A 🚀C1 neqo-pr vs. go-x-net: A BP BA neqo-pr vs. haproxy: A C1 BP BA neqo-pr vs. kwik: ⚠️C1 BP BA neqo-pr vs. linuxquic: A 🚀C1 ⚠️L1 neqo-pr vs. lsquic: L1 C1 neqo-pr vs. msquic: R Z A L1 C1 neqo-pr vs. mvfst: A L1 C1 ⚠️BA neqo-pr vs. nginx: A 🚀C1 BP BA neqo-pr vs. ngtcp2: A ⚠️L1 CM neqo-pr vs. picoquic: Z A L1 C1 neqo-pr vs. quic-go: A 🚀L1 C1 neqo-pr vs. quiche: A 🚀L1 ⚠️C1 BP BA neqo-pr vs. quinn: A 🚀C1 neqo-pr vs. s2n-quic: A L1 ⚠️C1 BA CM neqo-pr vs. tquic: S A ⚠️C1 BP BA neqo-pr vs. xquic: A 🚀C1 ⚠️L1	aioquic vs. neqo-pr: CM go-x-net vs. neqo-pr: CM kwik vs. neqo-pr: BP BA CM msquic vs. neqo-pr: CM mvfst vs. neqo-pr: Z A L1 C1 CM openssl vs. neqo-pr: LR M A CM quic-go vs. neqo-pr: CM quiche vs. neqo-pr: ⚠️L1 CM quinn vs. neqo-pr: V2 CM s2n-quic vs. neqo-pr: 🚀C1 CM tquic vs. neqo-pr: CM xquic vs. neqo-pr: M CM

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-pr as client

neqo-pr vs. aioquic: H DC LR C20 M S R Z 3 B U L1 L2 🚀C1 C2 6 V2 BP BA
neqo-pr vs. go-x-net: H DC LR M B U L2 C2 6
neqo-pr vs. haproxy: H DC LR C20 M S R Z 3 B U L1 L2 C2 6 V2
neqo-pr vs. kwik: H DC LR C20 M S R Z 3 B U A L1 L2 ⚠️C1 C2 6 V2
neqo-pr vs. linuxquic: H DC LR C20 M S R Z 3 B U E ⚠️L1 L2 🚀C1 C2 6 V2 BP BA CM
neqo-pr vs. lsquic: H DC LR C20 M S R Z 3 B U E A L2 C2 6 V2 BP BA CM
neqo-pr vs. msquic: H DC LR C20 M S B U L2 C2 6 V2 BP BA
neqo-pr vs. mvfst: H DC LR M R Z 3 B U L2 C2 6 BP ⚠️BA
neqo-pr vs. neqo: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
neqo-pr vs. nginx: H DC LR C20 M S R Z 3 B U L1 L2 🚀C1 C2 6
neqo-pr vs. ngtcp2: H DC LR C20 M S R Z 3 B U E ⚠️L1 L2 C1 C2 6 V2 BP BA
neqo-pr vs. picoquic: H DC LR C20 M S R 3 B U E L2 C2 6 V2 BP BA
neqo-pr vs. quic-go: H DC LR C20 M S R Z 3 B U 🚀L1 L2 C2 6 BP BA
neqo-pr vs. quiche: H DC LR C20 M S R Z 3 B U 🚀L1 L2 ⚠️C1 C2 6
neqo-pr vs. quinn: H DC LR C20 M S R Z 3 B U E L1 L2 🚀C1 C2 6 BP BA
neqo-pr vs. s2n-quic: H DC LR C20 M S R 3 B U E L2 ⚠️C1 C2 6 BP
neqo-pr vs. tquic: H DC LR C20 M R Z 3 B U L1 L2 ⚠️C1 C2 6
neqo-pr vs. xquic: H DC LR C20 M S R Z 3 B U ⚠️L1 L2 🚀C1 C2 6 BP BA

neqo-pr as server

aioquic vs. neqo-pr: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2 BP BA
chrome vs. neqo-pr: 3
go-x-net vs. neqo-pr: H DC LR M B U A L2 C2 6 BP BA
kwik vs. neqo-pr: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
linuxquic vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
lsquic vs. neqo-pr: H DC LR C20 M S R Z 3 B E A L1 L2 C1 C2 6 V2 BP BA CM
msquic vs. neqo-pr: H DC LR C20 M S R Z B U A L1 L2 C1 C2 6 V2 BP BA
mvfst vs. neqo-pr: H DC LR M 3 B L2 C2 6 BP BA
neqo vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
ngtcp2 vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
openssl vs. neqo-pr: H DC C20 S R 3 B L2 C2 6 BP BA
picoquic vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
quic-go vs. neqo-pr: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 BP BA
quiche vs. neqo-pr: H DC LR M S R Z 3 B A ⚠️L1 L2 C1 C2 6 BP BA
quinn vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 BP BA
s2n-quic vs. neqo-pr: H DC LR M S R 3 B E A L1 L2 🚀C1 C2 6 BP BA
tquic vs. neqo-pr: H DC LR M S R Z 3 B A L1 L2 C1 C2 6 BP BA
xquic vs. neqo-pr: H DC LR C20 S R Z 3 B U A L1 L2 C1 C2 6 BP BA

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-pr as client

neqo-pr vs. aioquic: E CM
neqo-pr vs. go-x-net: C20 S R Z 3 E L1 C1 V2 CM
neqo-pr vs. haproxy: E CM
neqo-pr vs. kwik: E CM
neqo-pr vs. msquic: 3 E CM
neqo-pr vs. mvfst: C20 S E V2 CM
neqo-pr vs. nginx: E V2 CM
neqo-pr vs. picoquic: CM
neqo-pr vs. quic-go: E V2 CM
neqo-pr vs. quiche: E V2 CM
neqo-pr vs. quinn: V2 CM
neqo-pr vs. s2n-quic: Z V2
neqo-pr vs. tquic: E V2 CM
neqo-pr vs. xquic: E V2 CM

neqo-pr as server

aioquic vs. neqo-pr: E
chrome vs. neqo-pr: H DC LR C20 M S R Z B U E A L1 L2 C1 C2 6 V2 BP BA CM
go-x-net vs. neqo-pr: C20 S R Z 3 E L1 C1 V2
kwik vs. neqo-pr: E
lsquic vs. neqo-pr: U
msquic vs. neqo-pr: 3 E
mvfst vs. neqo-pr: C20 S R U E V2
openssl vs. neqo-pr: Z U E L1 C1 V2
quic-go vs. neqo-pr: E V2
quiche vs. neqo-pr: C20 U E V2
s2n-quic vs. neqo-pr: C20 Z U V2
tquic vs. neqo-pr: C20 U E V2
xquic vs. neqo-pr: E V2

github-actions · 2025-12-10T09:08:43Z

Client/server transfer results

Performance differences relative to 8842dcd.

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server (params)	Mean ± σ	Min	Max	MiB/s ± σ	Δ `main`	Δ `main`
google-neqo-cubic	279.2 ± 3.8	271.1	294.5	114.6 ± 8.4	💔 1.2	0.4%
neqo-neqo-cubic	97.4 ± 4.9	90.6	121.6	328.6 ± 6.5	💔 1.8	1.9%
neqo-neqo-reno-nopacing	96.4 ± 4.2	89.3	106.3	332.0 ± 7.6	💔 1.6	1.6%
neqo-s2n-cubic	219.9 ± 4.0	212.2	230.3	145.5 ± 8.0	💚 -1.2	-0.5%

Table above only shows statistically significant changes. See all results below.

All results

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server (params)	Mean ± σ	Min	Max	MiB/s ± σ	Δ `main`	Δ `main`
google-google-nopacing	474.3 ± 4.1	466.2	484.3	67.5 ± 7.8
google-neqo-cubic	279.2 ± 3.8	271.1	294.5	114.6 ± 8.4	💔 1.2	0.4%
msquic-msquic-nopacing	201.0 ± 73.3	142.6	599.8	159.2 ± 0.4
msquic-neqo-cubic	223.4 ± 65.6	161.9	431.0	143.2 ± 0.5	-19.7	-8.1%
neqo-google-cubic	777.4 ± 4.4	768.6	788.9	41.2 ± 7.3	-0.9	-0.1%
neqo-msquic-cubic	157.7 ± 4.3	150.5	165.6	202.9 ± 7.4	0.5	0.3%
neqo-neqo-cubic	97.4 ± 4.9	90.6	121.6	328.6 ± 6.5	💔 1.8	1.9%
neqo-neqo-cubic-nopacing	95.5 ± 4.1	85.8	104.3	335.2 ± 7.8	-0.7	-0.7%
neqo-neqo-reno	98.0 ± 4.3	88.6	106.4	326.5 ± 7.4	1.0	1.1%
neqo-neqo-reno-nopacing	96.4 ± 4.2	89.3	106.3	332.0 ± 7.6	💔 1.6	1.6%
neqo-quiche-cubic	193.6 ± 4.6	187.6	205.5	165.3 ± 7.0	0.3	0.2%
neqo-s2n-cubic	219.9 ± 4.0	212.2	230.3	145.5 ± 8.0	💚 -1.2	-0.5%
quiche-neqo-cubic	152.7 ± 5.2	138.6	166.2	209.6 ± 6.2	0.1	0.1%
quiche-quiche-nopacing	145.5 ± 4.2	136.4	155.6	220.0 ± 7.6
s2n-neqo-cubic	173.5 ± 4.4	163.4	182.4	184.5 ± 7.3	-0.2	-0.1%
s2n-s2n-nopacing	243.4 ± 15.2	231.7	344.2	131.5 ± 2.1

Download data for profiler.firefox.com or download performance comparison data.

larseggert added a commit to larseggert/neqo that referenced this pull request Nov 10, 2025

ci: Add docker-build-qns action

3df6f2a

Needed for mozilla#3123. Will be a no-op until called from there, hence committing.

larseggert mentioned this pull request Nov 10, 2025

ci: Add docker-build-qns action #3124

Merged

larseggert force-pushed the ci-qns-no-cache branch 3 times, most recently from f43527c to 3e7e512 Compare November 10, 2025 15:12

larseggert marked this pull request as ready for review November 11, 2025 13:54

larseggert requested review from KershawChang, martinthomson and mxinden as code owners November 11, 2025 13:54

Copilot AI review requested due to automatic review settings November 11, 2025 13:54