feat(bbr3) Implemented bbr3's latest version based on the RFC

[Tipuch]

solves #1254
Based on https://www.ietf.org/archive/id/draft-ietf-ccwg-bbr-04.txt

tested with sudo tc qdisc add dev lo root netem delay 50ms 10ms distribution normal with a big file, this is a single connection
and it's BBR3 vs Cubic (the default) for quinn

1206KiB/s for BB3 vs 1049 KiB/s for Cubic so approximately 13% faster transfer speed (this is with a big file so over a long period of time).

Let me know if you need any additional information or if you want me to change the implementation in some ways.

[Ralith]

Thanks for working on this! I'm excited to have this feature, and those performance numbers look promising. It's hard to read the vertical scales, but I guess the second chart is the new path? The pacing rate units don't look quite right.

What conditions was this tested in? It'd be interesting to contrast e.g. loopback, LANs, wifi, cellular, long fat networks, satellite networks, etc. We don't need all of that to merge this, but it would help inform what our guidance to users on its use should be, and might motivate future changes in default configuration.

Is there a strong reason for us to keep the old BBR code around? It hasn't received any attention in years, so unless it serves a very strong purpose I think it would be better for this change to replace it outright, perhaps by just deleting it in a final commit.

As this is a large body of algorithmically nuanced code, review will be difficult. To start with, can you reorganize the PR as a series of incremental, linear commits? For example, please remove merge commits and squash together WIP, incomplete, or later-revised changes, and separate changes to existing Quinn interfaces from the introduction of new logic.

[Tipuch]

This was tested on local with simulated network latency (between a client and a server from the examples provided in quinn) 100 ms ping with 20 ms jitter.
The top one is Cubic and the bottom one is BBR3, as for the units I used the same units as the previous implementation of BBR but I guess I'll change that for Bytes/sec instead. I'll reorder and polish the commit history as well.
I'll try testing it in different conditions and post the graphs here as well.
As for the previous version of BBR I think we can replace it as it doesn't seem to me like it was ever in working condition (because some of the output parameters weren't being applied to the connection).

I'll edit the history here soon, change the units for pacing rate and post more graphs under different condition :)

[Tipuch]

Alright I have cleaned up the history of commits and adjusted the units for pacing_rate

I'll get to doing some more tests under different circumstances now.

[Tipuch]

I'm not sure what's the best way to share the graphs with you, I tried to get an average sample and no outliers (positive or negative) for cubic vs bbr3:
bbr3.html
bbr3_lo.html
bbr3_lo_tc.html
bbr3_wifi.html
cubic.html
cubic_lo.html
cubic_lo_tc.html
cubic_wifi.html

[Tipuch]

Using the perf server and client with default settings and within lo without limits and using tc qdisc add dev lo root netem delay 50ms 10ms distribution normal

bbr3:

cubic:

[Ralith]

Thanks for putting this together! The commit organization is great.

Because this is marked as experimental and replaces obsolete and equally experimental code, I'm not inclined to block this on an in-depth review of the actual BBR logic, which would otherwise take us much longer. That said, some documentation around it would dramatically improve our ability to notice implementation errors, and make it much easier for yourself or others to update and debug it in the future. Appropriate documentation might heavily cite or even excerpt the IETF draft.

[Tipuch]

I just noticed there are some inconsistencies with what the value of the pacing gain when in the draining state in the ietf draft, I'll update its value soon with additional justifications

[djc]

Please rebase and squash all your commits on your next push.

[Ralith]

(don't literally squash them all, just assemble into a logical progression like before)

[Tipuch]

I introduced a bug within the latest changes, so I will first find it and fix it and then clean up the commits.

[Tipuch]

I introduced a bug within the latest changes, so I will first find it and fix it and then clean up the commits.

fixed the bug, there were a couple errors in calculations but the main issue was with how inflight was set, ack_eliciting is the number of packets, not an amount of bytes, resulting in underestimating inflight by a lot

[Tipuch]

Alright cleaned up the commits :)

[happytrudy]

When can this submission be merged? I need to use the BBR congestion control algorithm.

[djc]

When can this submission be merged? I need to use the BBR congestion control algorithm.

This project is maintained by volunteers. If you want commercial support, feel free to email me.

[Fhokud]

Why not consider retaining both BBR and BB3 simultaneously? In certain scenarios, it still proves useful.

[djc]

Why not consider retaining both BBR and BB3 simultaneously? In certain scenarios, it still proves useful.

If you have concrete benchmarks/scenarios where there is a substantial advantage to the old implementations, we'll consider it.

[Fhokud]

If you have concrete benchmarks/scenarios where there is a substantial advantage to the old implementations, we'll consider it.

Test: A QUIC proxy (quinn 0.11 bbr) A QUIC proxy fork (quinn 0.12 Tipuch:bbrv3)

Code modifications only involve changing bbrconfig to bbr3config.

Tests conducted in real network environments:
Los Angeles 1Gbps to Guangzhou, China 1Gbps
and Hunan, China 1Gbps

Latency: 200 milliseconds
Jitter: Approximately 20 milliseconds

Interstate Gbps transmission often involves high latency, high jitter, and high packet loss.
bbr quic demonstrates significant advantages over bbr3 quic

Tested with 1/4/8/16 streams over a single QUIC connection
Averaged results from three runs

Complete data：
bbr-quinn.html
bbr3-quinn.html

TCP data included for reference（Oh my God）
bbr-tcp.html

[Itsusinn]

@Fhokud 's results seems strange. But this kind of downgrade problem is advantage of bbr3. It's more moral/fair for other internet traffic those don't use bbr.

[Tipuch]

I'll try and run some tests between the previous bbr version and the newest one, with real network conditions.

The results you're getting seem strange as if the BBR3 implementation did not scale with the number of streams.

[Tipuch]

I tested it with a VPS and the example server implementation in quinn single stream: 225ms ping, 20ms jitter.
I did several tries and it's sensibly the same with a slight advantage (5% speed up) for BBR3
approximately:

• 4279 KiB/s for BBR3
• 4142 KiB/s for BBR

you have to bear in mind that the previous BBR implementation for quinn was experimental and didn't adjust the pacing rate of the connection (which according to the original spec it should have).

BBR

BBR3

[Fhokud]

UDP 10M iperf3 log

iperf3 -c <<>> -p 443 -u -b 10M -t 60 -R -l 1200
Connecting to host <<>>, port 443
Reverse mode, remote host <<>> is sending
[  5] local 192.168.1.2 port 50967 connected to <<>> port 443
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.01   sec  1.20 MBytes  10.0 Mbits/sec  0.328 ms  38/1089 (3.5%)
[  5]   1.01-2.00   sec  1.10 MBytes  9.34 Mbits/sec  0.381 ms  67/1032 (6.5%)
[  5]   2.00-3.01   sec  1.16 MBytes  9.65 Mbits/sec  0.598 ms  42/1055 (4%)
[  5]   3.01-4.01   sec  1.11 MBytes  9.34 Mbits/sec  0.595 ms  57/1028 (5.5%)
[  5]   4.01-5.01   sec  1.16 MBytes  9.63 Mbits/sec  0.443 ms  45/1056 (4.3%)
[  5]   5.01-6.01   sec  1.12 MBytes  9.48 Mbits/sec  0.380 ms  53/1036 (5.1%)
[  5]   6.01-7.00   sec  1.14 MBytes  9.64 Mbits/sec  0.434 ms  38/1034 (3.7%)
[  5]   7.00-8.01   sec  1.15 MBytes  9.58 Mbits/sec  0.518 ms  45/1054 (4.3%)
[  5]   8.01-9.00   sec  1.13 MBytes  9.54 Mbits/sec  0.395 ms  47/1032 (4.6%)
[  5]   9.00-10.01  sec  1.14 MBytes  9.49 Mbits/sec  0.296 ms  52/1047 (5%)
[  5]  10.01-11.01  sec  1.14 MBytes  9.53 Mbits/sec  0.693 ms  53/1050 (5%)
[  5]  11.01-12.00  sec  1.13 MBytes  9.59 Mbits/sec  0.722 ms  38/1025 (3.7%)
[  5]  12.00-13.01  sec  1.13 MBytes  9.37 Mbits/sec  0.752 ms  66/1052 (6.3%)
[  5]  13.01-14.01  sec  1.14 MBytes  9.57 Mbits/sec  0.428 ms  48/1040 (4.6%)
[  5]  14.01-15.00  sec  1.11 MBytes  9.42 Mbits/sec  0.724 ms  58/1032 (5.6%)
[  5]  15.00-16.01  sec  1.15 MBytes  9.61 Mbits/sec  0.386 ms  41/1050 (3.9%)
[  5]  16.01-17.01  sec  1.14 MBytes  9.53 Mbits/sec  1.082 ms  50/1047 (4.8%)
[  5]  17.01-18.00  sec  1.12 MBytes  9.47 Mbits/sec  0.600 ms  54/1030 (5.2%)
[  5]  18.00-19.01  sec  1.15 MBytes  9.56 Mbits/sec  0.301 ms  46/1049 (4.4%)
[  5]  19.01-20.00  sec  1.15 MBytes  9.67 Mbits/sec  0.457 ms  35/1036 (3.4%)
[  5]  20.00-21.00  sec  1.14 MBytes  9.56 Mbits/sec  0.828 ms  42/1036 (4.1%)
[  5]  21.00-22.01  sec  1.16 MBytes  9.60 Mbits/sec  0.456 ms  45/1055 (4.3%)
[  5]  22.01-23.00  sec  1.14 MBytes  9.59 Mbits/sec  0.181 ms  44/1036 (4.2%)
[  5]  23.00-24.01  sec  1.12 MBytes  9.31 Mbits/sec  0.404 ms  70/1046 (6.7%)
[  5]  24.01-25.01  sec  1.14 MBytes  9.50 Mbits/sec  0.625 ms  55/1047 (5.3%)
[  5]  25.01-26.01  sec  1.13 MBytes  9.55 Mbits/sec  0.516 ms  43/1031 (4.2%)
[  5]  26.01-27.01  sec  1.10 MBytes  9.16 Mbits/sec  0.650 ms  88/1047 (8.4%)
[  5]  27.01-28.00  sec  1.13 MBytes  9.60 Mbits/sec  0.589 ms  43/1034 (4.2%)
[  5]  28.00-29.01  sec  1.13 MBytes  9.42 Mbits/sec  0.366 ms  60/1047 (5.7%)
[  5]  29.01-30.01  sec  1.15 MBytes  9.66 Mbits/sec  0.664 ms  35/1043 (3.4%)
[  5]  30.01-31.00  sec  1.13 MBytes  9.52 Mbits/sec  0.354 ms  49/1034 (4.7%)
[  5]  31.00-32.00  sec  1.14 MBytes  9.56 Mbits/sec  0.338 ms  46/1042 (4.4%)
[  5]  32.00-33.01  sec  1.15 MBytes  9.57 Mbits/sec  0.396 ms  45/1047 (4.3%)
[  5]  33.01-34.00  sec  1.13 MBytes  9.48 Mbits/sec  0.789 ms  56/1040 (5.4%)
[  5]  34.00-35.01  sec  1.16 MBytes  9.60 Mbits/sec  0.624 ms  41/1051 (3.9%)
[  5]  35.01-36.01  sec  1.12 MBytes  9.41 Mbits/sec  0.902 ms  61/1037 (5.9%)
[  5]  36.01-37.00  sec  1.15 MBytes  9.70 Mbits/sec  0.464 ms  33/1036 (3.2%)
[  5]  37.00-38.01  sec  1.14 MBytes  9.49 Mbits/sec  0.969 ms  51/1049 (4.9%)
[  5]  38.01-39.00  sec  1.12 MBytes  9.50 Mbits/sec  0.390 ms  51/1030 (5%)
[  5]  39.00-40.01  sec  1.15 MBytes  9.58 Mbits/sec  0.384 ms  45/1052 (4.3%)
[  5]  40.01-41.00  sec  1.13 MBytes  9.55 Mbits/sec  0.365 ms  47/1036 (4.5%)
[  5]  41.00-42.01  sec  1.15 MBytes  9.61 Mbits/sec  0.615 ms  39/1046 (3.7%)
[  5]  42.01-43.01  sec  1.14 MBytes  9.65 Mbits/sec  0.766 ms  37/1037 (3.6%)
[  5]  43.01-44.01  sec  1.15 MBytes  9.57 Mbits/sec  0.220 ms  46/1047 (4.4%)
[  5]  44.01-45.00  sec  1.14 MBytes  9.63 Mbits/sec  0.363 ms  41/1039 (3.9%)
[  5]  45.00-46.01  sec  1.15 MBytes  9.58 Mbits/sec  0.197 ms  39/1046 (3.7%)
[  5]  46.01-47.00  sec  1.12 MBytes  9.46 Mbits/sec  0.832 ms  56/1031 (5.4%)
[  5]  47.00-48.02  sec  1.16 MBytes  9.61 Mbits/sec  1.246 ms  46/1060 (4.3%)
[  5]  48.02-49.00  sec  1.08 MBytes  9.22 Mbits/sec  0.176 ms  77/1024 (7.5%)
[  5]  49.00-50.01  sec  1.16 MBytes  9.67 Mbits/sec  0.330 ms  38/1052 (3.6%)
[  5]  50.01-51.01  sec  1.13 MBytes  9.45 Mbits/sec  0.677 ms  55/1045 (5.3%)
[  5]  51.01-52.01  sec  1.14 MBytes  9.55 Mbits/sec  0.757 ms  47/1041 (4.5%)
[  5]  52.01-53.01  sec  1.14 MBytes  9.61 Mbits/sec  0.677 ms  40/1037 (3.9%)
[  5]  53.01-54.01  sec  1.17 MBytes  9.73 Mbits/sec  0.546 ms  31/1049 (3%)
[  5]  54.01-55.01  sec  1.12 MBytes  9.46 Mbits/sec  0.767 ms  52/1030 (5%)
[  5]  55.01-56.01  sec  1.13 MBytes  9.44 Mbits/sec  0.591 ms  60/1051 (5.7%)
[  5]  56.01-57.01  sec  1.10 MBytes  9.30 Mbits/sec  0.374 ms  62/1024 (6.1%)
[  5]  57.01-58.00  sec  1.14 MBytes  9.59 Mbits/sec  0.693 ms  43/1036 (4.2%)
[  5]  58.00-59.01  sec  1.14 MBytes  9.52 Mbits/sec  0.807 ms  48/1045 (4.6%)
[  5]  59.01-60.01  sec  1.15 MBytes  9.64 Mbits/sec  0.420 ms  39/1041 (3.7%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.28  sec  71.9 MBytes  10.0 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-60.01  sec  68.2 MBytes  9.54 Mbits/sec  0.420 ms  2919/62531 (4.7%)  receiver
  

UDP 50M iperf3 log

iperf3 -c <<>> -p 443 -u -b 50M -t 60 -R -l 1200
Connecting to host <<>>, port 443
Reverse mode, remote host 74.48.72.169 is sending
[  5] local 192.168.1.2 port 49209 connected to <<>> port 443
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.01   sec  5.66 MBytes  47.2 Mbits/sec  0.318 ms  521/5470 (9.5%)
[  5]   1.01-2.01   sec  5.82 MBytes  48.5 Mbits/sec  0.241 ms  167/5250 (3.2%)
[  5]   2.01-3.01   sec  5.65 MBytes  47.8 Mbits/sec  0.265 ms  259/5198 (5%)
[  5]   3.01-4.01   sec  5.79 MBytes  48.2 Mbits/sec  0.233 ms  168/5224 (3.2%)
[  5]   4.01-5.01   sec  5.69 MBytes  48.2 Mbits/sec  0.102 ms  222/5196 (4.3%)
[  5]   5.01-6.01   sec  5.80 MBytes  48.2 Mbits/sec  0.076 ms  201/5273 (3.8%)
[  5]   6.01-7.01   sec  5.58 MBytes  46.8 Mbits/sec  0.317 ms  202/5076 (4%)
[  5]   7.01-8.00   sec  5.75 MBytes  48.7 Mbits/sec  0.114 ms  260/5283 (4.9%)
[  5]   8.00-9.02   sec  5.79 MBytes  48.1 Mbits/sec  0.137 ms  192/5255 (3.7%)
[  5]   9.02-10.01  sec  5.64 MBytes  47.5 Mbits/sec  0.195 ms  247/5172 (4.8%)
[  5]  10.01-11.01  sec  5.73 MBytes  48.3 Mbits/sec  0.139 ms  151/5162 (2.9%)
[  5]  11.01-12.01  sec  5.69 MBytes  47.6 Mbits/sec  0.142 ms  291/5265 (5.5%)
[  5]  12.01-13.00  sec  5.71 MBytes  48.4 Mbits/sec  0.079 ms  192/5182 (3.7%)
[  5]  13.00-14.01  sec  5.69 MBytes  47.4 Mbits/sec  0.184 ms  261/5235 (5%)
[  5]  14.01-15.00  sec  5.72 MBytes  48.3 Mbits/sec  0.165 ms  198/5192 (3.8%)
[  5]  15.00-16.01  sec  5.71 MBytes  47.5 Mbits/sec  0.264 ms  190/5183 (3.7%)
[  5]  16.01-17.01  sec  5.67 MBytes  47.7 Mbits/sec  0.141 ms  296/5250 (5.6%)
[  5]  17.01-18.00  sec  5.70 MBytes  48.1 Mbits/sec  0.082 ms  165/5145 (3.2%)
[  5]  18.00-19.01  sec  5.68 MBytes  47.3 Mbits/sec  0.065 ms  327/5292 (6.2%)
[  5]  19.01-20.01  sec  5.72 MBytes  48.3 Mbits/sec  0.186 ms  141/5141 (2.7%)
[  5]  20.01-21.01  sec  5.72 MBytes  47.6 Mbits/sec  0.077 ms  278/5275 (5.3%)
[  5]  21.01-22.01  sec  5.73 MBytes  48.4 Mbits/sec  0.222 ms  168/5177 (3.2%)
[  5]  22.01-23.01  sec  5.72 MBytes  47.7 Mbits/sec  0.169 ms  237/5235 (4.5%)
[  5]  23.01-24.01  sec  5.71 MBytes  48.2 Mbits/sec  0.208 ms  163/5152 (3.2%)
[  5]  24.01-25.01  sec  5.62 MBytes  46.9 Mbits/sec  0.167 ms  341/5255 (6.5%)
[  5]  25.01-26.00  sec  5.78 MBytes  49.0 Mbits/sec  0.090 ms  132/5182 (2.5%)
[  5]  26.00-27.01  sec  5.64 MBytes  46.8 Mbits/sec  0.153 ms  309/5234 (5.9%)
[  5]  27.01-28.01  sec  5.74 MBytes  48.4 Mbits/sec  0.178 ms  184/5198 (3.5%)
[  5]  28.01-29.01  sec  5.67 MBytes  47.1 Mbits/sec  0.152 ms  263/5214 (5%)
[  5]  29.01-30.01  sec  5.72 MBytes  48.4 Mbits/sec  0.182 ms  187/5187 (3.6%)
[  5]  30.01-31.01  sec  5.76 MBytes  48.3 Mbits/sec  0.135 ms  180/5214 (3.5%)
[  5]  31.01-32.02  sec  5.72 MBytes  47.6 Mbits/sec  0.145 ms  254/5255 (4.8%)
[  5]  32.02-33.00  sec  5.70 MBytes  48.5 Mbits/sec  0.161 ms  141/5125 (2.8%)
[  5]  33.00-34.01  sec  5.67 MBytes  47.2 Mbits/sec  0.165 ms  294/5250 (5.6%)
[  5]  34.01-35.00  sec  5.72 MBytes  48.4 Mbits/sec  0.090 ms  191/5187 (3.7%)
[  5]  35.00-36.01  sec  5.64 MBytes  46.8 Mbits/sec  0.232 ms  288/5219 (5.5%)
[  5]  36.01-37.01  sec  5.80 MBytes  49.0 Mbits/sec  0.158 ms  162/5228 (3.1%)
[  5]  37.01-38.01  sec  5.63 MBytes  47.1 Mbits/sec  0.143 ms  259/5178 (5%)
[  5]  38.01-39.00  sec  5.71 MBytes  48.3 Mbits/sec  0.139 ms  225/5214 (4.3%)
[  5]  39.00-40.01  sec  5.74 MBytes  47.8 Mbits/sec  0.033 ms  225/5240 (4.3%)
[  5]  40.01-41.01  sec  5.61 MBytes  47.3 Mbits/sec  0.074 ms  262/5165 (5.1%)
[  5]  41.01-42.01  sec  5.81 MBytes  48.4 Mbits/sec  0.216 ms  183/5257 (3.5%)
[  5]  42.01-43.01  sec  5.57 MBytes  47.1 Mbits/sec  0.095 ms  308/5177 (5.9%)
[  5]  43.01-44.00  sec  5.79 MBytes  48.7 Mbits/sec  0.136 ms  144/5202 (2.8%)
[  5]  44.00-45.01  sec  5.59 MBytes  46.6 Mbits/sec  0.068 ms  342/5230 (6.5%)
[  5]  45.01-46.00  sec  5.72 MBytes  48.4 Mbits/sec  0.184 ms  176/5172 (3.4%)
[  5]  46.00-47.01  sec  5.62 MBytes  46.9 Mbits/sec  0.041 ms  315/5228 (6%)
[  5]  47.01-48.00  sec  5.73 MBytes  48.3 Mbits/sec  0.147 ms  189/5199 (3.6%)
[  5]  48.00-49.01  sec  5.70 MBytes  47.7 Mbits/sec  0.110 ms  227/5208 (4.4%)
[  5]  49.01-50.02  sec  5.72 MBytes  47.5 Mbits/sec  0.234 ms  262/5261 (5%)
[  5]  50.02-51.00  sec  5.67 MBytes  48.2 Mbits/sec  0.135 ms  183/5136 (3.6%)
[  5]  51.00-52.00  sec  5.62 MBytes  47.1 Mbits/sec  0.151 ms  297/5207 (5.7%)
[  5]  52.00-53.01  sec  5.85 MBytes  48.7 Mbits/sec  0.062 ms  149/5260 (2.8%)
[  5]  53.01-54.00  sec  5.53 MBytes  46.8 Mbits/sec  0.207 ms  328/5157 (6.4%)
[  5]  54.00-55.05  sec  6.09 MBytes  48.8 Mbits/sec  0.183 ms  146/5468 (2.7%)
[  5]  55.05-56.00  sec  5.29 MBytes  46.6 Mbits/sec  0.291 ms  323/4949 (6.5%)
[  5]  56.00-57.01  sec  5.83 MBytes  48.5 Mbits/sec  0.116 ms  179/5275 (3.4%)
[  5]  57.01-58.00  sec  5.62 MBytes  47.6 Mbits/sec  0.126 ms  232/5147 (4.5%)
[  5]  58.00-59.01  sec  5.82 MBytes  48.4 Mbits/sec  0.129 ms  183/5266 (3.5%)
[  5]  59.01-60.00  sec  5.62 MBytes  47.6 Mbits/sec  0.176 ms  227/5140 (4.4%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.30  sec   359 MBytes  50.0 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-60.00  sec   342 MBytes  47.8 Mbits/sec  0.176 ms  13787/312767 (4.4%)  receiver
  

UDP 100M iperf3 log

iperf3 -c <<>> -p 443 -u -b 100M -t 60 -R -l 1200
Connecting to host <<>>, port 443
Reverse mode, remote host <<>> is sending
[  5] local 192.168.1.2 port 49867 connected to <<>> port 443
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  11.6 MBytes  96.6 Mbits/sec  0.041 ms  873/10976 (8%)
[  5]   1.00-2.02   sec  11.4 MBytes  94.0 Mbits/sec  0.071 ms  589/10520 (5.6%)
[  5]   2.02-3.00   sec  11.2 MBytes  95.1 Mbits/sec  0.060 ms  556/10309 (5.4%)
[  5]   3.00-4.01   sec  11.3 MBytes  94.9 Mbits/sec  0.192 ms  381/10274 (3.7%)
[  5]   4.01-5.01   sec  11.4 MBytes  95.4 Mbits/sec  0.104 ms  638/10612 (6%)
[  5]   5.01-6.00   sec  11.3 MBytes  95.8 Mbits/sec  0.094 ms  380/10270 (3.7%)
[  5]   6.00-7.01   sec  11.2 MBytes  93.0 Mbits/sec  0.105 ms  728/10514 (6.9%)
[  5]   7.01-8.00   sec  11.5 MBytes  97.1 Mbits/sec  0.028 ms  327/10361 (3.2%)
[  5]   8.00-9.01   sec  11.3 MBytes  93.8 Mbits/sec  0.089 ms  678/10552 (6.4%)
[  5]   9.01-10.00  sec  11.4 MBytes  96.6 Mbits/sec  0.084 ms  310/10260 (3%)
[  5]  10.00-11.01  sec  11.3 MBytes  94.2 Mbits/sec  0.089 ms  628/10532 (6%)
[  5]  11.01-12.00  sec  11.5 MBytes  97.2 Mbits/sec  0.064 ms  343/10374 (3.3%)
[  5]  12.00-13.01  sec  11.3 MBytes  94.2 Mbits/sec  0.073 ms  587/10469 (5.6%)
[  5]  13.01-14.02  sec  11.6 MBytes  96.4 Mbits/sec  0.104 ms  355/10480 (3.4%)
[  5]  14.02-15.01  sec  11.2 MBytes  94.9 Mbits/sec  0.054 ms  557/10382 (5.4%)
[  5]  15.01-16.00  sec  11.5 MBytes  97.0 Mbits/sec  0.021 ms  307/10316 (3%)
[  5]  16.00-17.01  sec  11.2 MBytes  93.7 Mbits/sec  0.208 ms  552/10370 (5.3%)
[  5]  17.01-18.00  sec  11.5 MBytes  97.1 Mbits/sec  0.093 ms  437/10506 (4.2%)
[  5]  18.00-19.01  sec  11.4 MBytes  94.6 Mbits/sec  0.109 ms  535/10502 (5.1%)
[  5]  19.01-20.01  sec  11.4 MBytes  96.4 Mbits/sec  0.091 ms  387/10346 (3.7%)
[  5]  20.01-21.01  sec  11.4 MBytes  95.1 Mbits/sec  0.072 ms  495/10486 (4.7%)
[  5]  21.01-22.01  sec  11.2 MBytes  94.2 Mbits/sec  0.243 ms  559/10319 (5.4%)
[  5]  22.01-23.00  sec  11.4 MBytes  96.1 Mbits/sec  0.152 ms  385/10314 (3.7%)
[  5]  23.00-24.01  sec  11.4 MBytes  94.4 Mbits/sec  0.101 ms  625/10562 (5.9%)
[  5]  24.01-25.01  sec  11.5 MBytes  96.6 Mbits/sec  0.045 ms  385/10469 (3.7%)
[  5]  25.01-26.00  sec  10.8 MBytes  91.2 Mbits/sec  0.523 ms  630/10032 (6.3%)
[  5]  26.00-27.01  sec  11.5 MBytes  95.9 Mbits/sec  0.126 ms  556/10648 (5.2%)
[  5]  27.01-28.01  sec  11.3 MBytes  95.4 Mbits/sec  0.126 ms  475/10348 (4.6%)
[  5]  28.01-29.01  sec  11.4 MBytes  94.8 Mbits/sec  0.110 ms  610/10560 (5.8%)
[  5]  29.01-30.00  sec  5.21 MBytes  44.2 Mbits/sec  0.410 ms  364/4917 (7.4%)
[  5]  30.00-31.01  sec  9.87 MBytes  82.5 Mbits/sec  0.166 ms  371/8996 (4.1%)
[  5]  31.01-32.00  sec  16.6 MBytes   140 Mbits/sec  0.122 ms  775/15256 (5.1%)
[  5]  32.00-33.01  sec  13.4 MBytes   112 Mbits/sec  0.093 ms  793/12525 (6.3%)
[  5]  33.01-34.00  sec  11.5 MBytes  97.4 Mbits/sec  0.052 ms  270/10322 (2.6%)
[  5]  34.00-35.00  sec  11.1 MBytes  92.7 Mbits/sec  0.097 ms  639/10308 (6.2%)
[  5]  35.00-36.02  sec  11.5 MBytes  95.5 Mbits/sec  0.053 ms  609/10693 (5.7%)
[  5]  36.02-37.01  sec  11.0 MBytes  92.4 Mbits/sec  0.067 ms  778/10376 (7.5%)
[  5]  37.01-38.02  sec  11.6 MBytes  96.7 Mbits/sec  0.068 ms  295/10394 (2.8%)
[  5]  38.02-39.01  sec  11.1 MBytes  94.1 Mbits/sec  0.089 ms  650/10374 (6.3%)
[  5]  39.01-40.01  sec  11.3 MBytes  94.5 Mbits/sec  0.086 ms  594/10502 (5.7%)
[  5]  40.01-41.00  sec  11.2 MBytes  95.3 Mbits/sec  0.102 ms  469/10289 (4.6%)
[  5]  41.00-42.01  sec  11.6 MBytes  95.8 Mbits/sec  0.105 ms  257/10350 (2.5%)
[  5]  42.01-43.00  sec  11.6 MBytes  98.2 Mbits/sec  0.120 ms  378/10491 (3.6%)
[  5]  43.00-44.01  sec  11.6 MBytes  96.0 Mbits/sec  0.073 ms  417/10517 (4%)
[  5]  44.01-45.02  sec  11.6 MBytes  97.3 Mbits/sec  0.225 ms  273/10438 (2.6%)
[  5]  45.02-46.01  sec  11.4 MBytes  96.2 Mbits/sec  0.040 ms  428/10374 (4.1%)
[  5]  46.01-47.06  sec  11.8 MBytes  93.9 Mbits/sec  0.092 ms  600/10906 (5.5%)
[  5]  47.06-48.01  sec  11.0 MBytes  97.0 Mbits/sec  0.058 ms  367/9957 (3.7%)
[  5]  48.01-49.01  sec  11.4 MBytes  95.6 Mbits/sec  0.088 ms  385/10334 (3.7%)
[  5]  49.01-50.00  sec  11.5 MBytes  97.3 Mbits/sec  0.074 ms  316/10385 (3%)
[  5]  50.00-51.09  sec  12.2 MBytes  94.0 Mbits/sec  0.115 ms  598/11216 (5.3%)
[  5]  51.09-52.00  sec  10.7 MBytes  97.5 Mbits/sec  0.064 ms  335/9648 (3.5%)
[  5]  52.00-53.01  sec  11.5 MBytes  95.7 Mbits/sec  0.109 ms  412/10476 (3.9%)
[  5]  53.01-54.00  sec  11.4 MBytes  96.8 Mbits/sec  0.074 ms  383/10347 (3.7%)
[  5]  54.00-55.01  sec  11.5 MBytes  96.2 Mbits/sec  0.130 ms  335/10420 (3.2%)
[  5]  55.01-56.00  sec  11.6 MBytes  97.8 Mbits/sec  0.090 ms  253/10385 (2.4%)
[  5]  56.00-57.05  sec  11.7 MBytes  93.2 Mbits/sec  0.057 ms  778/11000 (7.1%)
[  5]  57.05-58.01  sec  10.7 MBytes  93.8 Mbits/sec  0.059 ms  585/9904 (5.9%)
[  5]  58.01-59.01  sec  11.6 MBytes  97.6 Mbits/sec  0.040 ms  270/10448 (2.6%)
[  5]  59.01-60.01  sec  11.4 MBytes  95.2 Mbits/sec  0.053 ms  363/10314 (3.5%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.32  sec   719 MBytes   100 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-60.01  sec   682 MBytes  95.4 Mbits/sec  0.053 ms  29208/625525 (4.7%)  receiver
  

Possibly, packet loss on this link fluctuates between 3% and 6%.
Did you account for packet loss during testing?

The results you're getting seem strange as if the BBR3 implementation did not scale with the number of streams.

Yes, that's strange.
I might need to look into how to obtain more detailed test data.

Perhaps BBR3 isn't suitable for this poor network connection.

[Ralith]

The old BBR implementation is unmaintained and has known issues. If its behavior happens to be convenient to you, you're free to maintain it downstream; quinn does not require congestion controllers to be built in. However, as others have noted, there may be drawbacks to a badly-behaved congestion controller which are less obvious than a simple uncontended benchmark result, so I recommend caution.

[Ralith]

Thanks, looking pretty good now! As usual, please squash together changes to earlier commits in the same PR.

[Itsusinn]

Why not consider retaining both BBR and BB3 simultaneously? In certain scenarios, it still proves useful.

I created a crate from baseline files https://github.com/rust-proxy/quinn-congestions which provides BBR.(Maybe also brutal in future)