GPX uploads fail with 503 for users on slow connections

[Rub21]

A user on the forum reported that could not upload two GPX files (1.1 MB and 1.5 MB): https://community.openstreetmap.org/t/gpx-upload-trouble/144548

The error was:

Error 503 backend write error
Varnish cache server

The files were valid, and the same files uploaded fine later on a better connection. The problem seems not the file or the metadata, it seems the upload speed.

The Apache frontend config in chef sets a body read timeout with a 120 second cap (apache.frontend.erb#L45):

RequestReadTimeout handshake=20-40,MinRate=500 header=20-40,MinRate=500 body=20-120,MinRate=500

So the whole upload has to finish in 120 seconds , and a 1.5 MB file needs at least ~13 kB/s. On slow mobile or rural connections this fails, Apache drops the request, Fastly returns the 503, and the user gets no useful message.

Today, an alternative to avoid this issue could be compress the file to .gpx.gz before upload, but I feel like most users don't know this, and I think a small hint on the upload form, or a clearer error message suggesting compression, would help.

[Rub21]

Some ideas that could help.

• Increase the timeout in chef, for example body=20,MinRate=500 (no hard cap) or a higher cap like body=20-600. but the downside it that slow uploads hold Apache workers open much longer.
• Compress the file in the browser before upload, using the CompressionStream API to send it as .gpx.gz. The server already accepts this format, so no backend change. Uploads get 4-5x smaller. This also fits well with Compress plain GPX uploads with gzip openstreetmap-website#7124
• Upload directly the file to object storage with a presigned URL, so the file skips the Apache timeout, rails would still import the points from S3 in the background job, same as today.

[tomhughes]

I don't believe your understanding of the apache config is correct. I'd need to look it up to check but I didn't think there was any overall timeout so long as the client maintained some minimal flow rate.

Regardless that config is an extremely important protection that was added as a result of a real world attack.

[tomhughes]

Also this is an operational issue, not a web site issue, so I will move this.

[tomhughes]

So I think you might be right that 120s is the maximum that the overall timeout will reach, and we probably could increase that, but I@m not sure that is the real problem here.

To start with the total number of request body timeouts I can find in our logs for the last months is basically in single figures - it's really very rare indeed.

On top of that such timeouts return 408 and I'm not sure why fastly would change that to the 503 we see here.

[Rub21]

I did some tests to reproduce this issue. I uploaded a GPX file (1.5 MB) at different speeds using curl --limit-rate, first against production and then against the dev server (master.apis.dev.openstreetmap.org), that is not behind Fastly.

Production:

rate	result	failed at	body sent
200 kB/s	200 OK	-	100%
100 kB/s	200 OK (15.6s total)	-	100%
20 kB/s	503 backend write error	23.4s	31%
5 kB/s	503 backend write error	44.5s	15%

In the dev server

The file(1.5 MB) at 20 kB/s uploaded fine, 200 OK after 74s. At 5 kB/s also worked, 200 OK after 296s. That second one took more than 120s, so the dev server clearly does not have the same Apache limits than production, this comparison only shows that the backend itself can handle slow uploads without problems.

In Production

@tomhughes you are right that Apache is not the problem, the failures happen at 23s and 44s, far from the 120s cap, and the response is always the Fastly "backend write error" page (Error 54113), never a 408. It also matches what you saw in the logs, Apache almost never reach the body timeout because Fastly gives up first.

So it look like a timeout is in Fastly side while it streams the request body to the backend.

[Firefishy]

@Rub21 would you mind re-testing?

I think it is related to Fastly buffering connection and it hitting a first_byte_timeout=15s timeout

I have just pushed the following live as a potential workaround:

if (req.request == "POST" && (req.url ~ "^/traces" || req.url ~ "^/api/")) {
  set bereq.first_byte_timeout = 130s;  # exceed Apache's body=20-120 cap
}