Upload speed significantly lower than download speed when running NDT server on AWS ECS

[kshitij-cyara]

Summary

I am running an NDT server using the official docker image. During testing I consistently observe that download speeds match the expected bandwidth, but upload speeds are significantly lower.

I am trying to understand whether this behavior is expected due to network conditions, TCP tuning, or something related to the NDT server setup. Apart from the ecs setup, i also tried to deploy the same on a plain ec2 instance (c6in.large) but the result was same in that case as well.

---

Environment

Cloud Provider: AWS
Region: ap-south-2
Compute: EC2 instance running in an ECS cluster
Instance Type: c6in.large

OS: Amazon Linux 2023
Container Runtime: Docker
Orchestration: ECS (EC2 launch type)

---

Current Architecture

Client
↓
Network Load Balancer
↓
ECS Service
↓
ECS Task (Docker container running NDT server)
↓
EC2 instance

Key configuration details:

• ECS network mode: host

• Container image: custom build of the NDT server pushed to ECR

• NLB forwards traffic directly to the ECS service

• Target group type: instance

• NDT server exposed on port 443

---

Network tuning configuration

The instance has the following TCP settings configured:

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_adv_win_scale = -2
net.ipv4.tcp_notsent_lowat = 131072

TCP congestion control is set to BBR.

---

Upload throughput is consistently lower than download throughput.

---

Goal

I am trying to achieve symmetric upload and download throughput when the client network supports it.

Any suggestions or best practices for running NDT servers in cloud environments would be greatly appreciated.

[bassosimone]

That's an interesting question, thank you! Here's what I historically remember. I tested the early 2019 implementation cloud to cloud using two GCE zones (IIRC Helsinki and Munich?) and the output was symmetric and ~5.5 Gbit/s. I do remember it was very close to iperf results, so I was pretty satisfied.

There are several bottlenecks you might be hitting including network bottlenecks, network stack bottlenecks, TLS bottlenecks and Go code bottlenecks. I will check my past notes and get back at you with more information. It would be useful, if possible to share publicly, for you to elaborate on (a) what is the average speed output you get and (b) to know more about how do you know that the available bandwidth in both directions is actually X.

Also, regarding this:

net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728

Can you elaborate on why you chose these values? IIRC, rmem_max and wmem_max should have no impact when you configure tcp_rmem and tcp_wmem but my memory may fail me, so knowing you rationale would be a good refresher for me. Also, I see that rmem and wmem are not symmetrical in the initial value. Was this deliberate?

Thanks!

[kshitij-cyara]

@bassosimone Regarding, the variance between the download and upload speed. I am attaching some screenshots down below.

Questions:

1. net.core.rmem_max and net.core.wmem_max were increased to 256MB to allow applications such as iperf3 to request larger buffers if needed. My understanding was that these values act as the upper bound that the kernel will allow for socket buffers.
2. This was not a deliberate optimization for the workload. I kept the default initial values provided by the system for the middle parameters and only increased the maximum values to allow larger autotuned buffers when necessary. These values are mainly hit and trial for me to maybe optimize the speed further.

Removing them, making them same had no effect on the speed what so ever.

[bassosimone]

@kshitij-cyara thanks for explaining your rationale regarding the parameters. I see you are using the web client. I also understand that you are using iperf3 to figure out the ground truth. I think these are useful pieces of information and I can work with them to try and suggest ways to understand what is going on.

Would you be willing to compile and use https://github.com/m-lab/ndt7-client-go instead? This client allows us to (a) bypass the browser network stack and (a) optionally bypass TLS. All in all, if your reference bandwidth is iperf3, my thinking is that we should try to approximate what iperf3 does in order to reach as much as possible an apples-vs-apples comparison and then we can circle back to attributing bottlenecks, document them, and see whether we (M-Lab) can improve our tools.

The README.md explains how to compile the client. To run a test you would then use this command line pattern:

./ndt7-client -scheme $scheme -server $ipAddr

Use wss (secure Web Socket) as the $scheme to use TLS; use ws (WebSocket) to skip TLS.

The ipAddr variable should be the IP address of your server.

You can also use -download=false -upload=true (or combinations of those parameters) to decide whether you want to run the download test, the upload test, or both of them.

The default output is human-friendly but you can use -format=json if you prefer JSON.

Thank you!

[kshitij-cyara]

@bassosimone Thanks for the reply.
I tried the same with the go client as you have suggested. The results for them are as below:
The result to the plain ec2 instance from the web client from browser.

The result from the go client to the ec2 instance.

The result from the go client to the ecs cluster setup.

The results from the go client is pretty good, but is there a possibility of finding why the results from the web client in the browser is so different from the go client one. In my code i am directly using ndt/v7/download and ndt/v7/upload through websocket connections and getting the result in that way and they are similar to the web client one, meaning the upload speed is very less than whats expected. Is there any debugging steps i can perform to find out the exact reason.

[kshitij-cyara]

@bassosimone Reminder for the above comment. Awaiting your reply.

[bassosimone]

Thanks! This asymmetry you showed from the browser tests make me wonder whether this could be an instance of m-lab/ndt7-js#83. Are you perhaps using Firefox? Here's a TL;DR of what I know. On Windows Firefox upload performance is significantly lower than on Linux. My understanding is that the browser multi-process architecture gets somehow in the way asymmetrically across operating systems. (The JavaScript code runs in the renderer process and needs to talk to the network I/O process, which requires to marshal/unmarshal messages.

If your tests fit into this specific category of Firefox on Windows, I know that the issue has been reported upstream to Mozilla. Instead, if this is a different web browser, what is the browser engine? What is the operating system?

[kshitij-cyara]

@bassosimone Thanks for replying:
The tests we have performed using browser are:

1. Chrome from windows 11 machine.
2. Chrome from fedora 43 machine.

The same i have shared with you.

I checked the results from firefox too, it was significantly lower, lower than what i have shared with you. From there i actually saw the the issue you have attached and moved on from firefox browser.

My main concern is through code (i am creating a chrome extension), in code i am using ndt/v7/download and ndt/v7/upload urls and through websocket connections and getting the result in that way and they are similar to the test with chrome browser on windows machine.

As in the end, i want that through code atleast, i want the speed equal to what i am getting through go client library, both upload and download speed.

[bassosimone]

Understood, thanks for explaining the context to me. I think the limitations you are seeing probably reflect limitations of the browser technology. It makes sense that the Chrome extension shows the same performance of Chrome. For measuring Gigabit internet connections using web protocols, it seems the browser is a limitation. Future work on NDT will try to address these performance issues but I think this will take some quarters to materialize.

[kshitij-cyara]

@bassosimone Thanks for replying.
So, Right now, the way we are connecting the websocket (in our chrome extension codebase) and calculating the speeds using MLab self hosted server have its limitations. Can you suggest anything else which may help in our case of creating a chrome extension(in NodeJS) which needs to calculate the internet speed.
Thanks for helping anyways.

[bassosimone]

Is it possible for you to bundle the ndt7-client-go client with the extension and to execute it? If that is possible, then this would be the most efficient way of integrating a ndt7 client. The client itself is written in Go meaning that it can be cross compiled for Windows, Linux, macOS. I know of projects that do this but they are not Chrome Extensions, rather they are Electron applications.

[kshitij-cyara]

@bassosimone Thanks for helping me.
I am pretty sure, its not possible but we are still looking into it. Apart from this, how can we add some kind of authentication in our self hosted server like in my current setup. As if our self hosted server is private, we want to limit people from using it. Any kind of authentication is enough, even a static token. Is there anyway of adding this without affecting the speed much. Whats the recommended approach for this ?

Again thanks for replying.