[Bug]: Error 499 while syncing thousands of new or modified files

[kiv57]

⚠️ Before submitting, please verify the following: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
• I agree to follow Nextcloud's Code of Conduct

Bug description

From the desktop client I added a local folder for syncing, containing about 50 000 files (photos).

After a few tens of seconds, the client shows a bunch of [\FilePath] : Network error : 499, then syncing speed falls to zero.

I can restart the sync : a hundred more files are synchronised before the errors appear again.

Steps to reproduce

1. From Windows desktop client, add a local folder to sync on server
2. Start syncing
3. Wait for about 100 files to sync and see errors on client.

Expected behavior

Smooth syncing of thousands of files.

Which files are affected by this bug

All local files (.jpg) waiting for syncing

Operating system

Windows

Which version of the operating system you are running.

Windows 11

Package

Official Windows MSI

Nextcloud Server version

30.0.2

Nextcloud Desktop Client version

3.14.3

Is this bug present after an update or on a fresh install?

Updated from a minor version (ex. 3.4.2 to 3.4.4)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

• Default internal user-backend
• LDAP/ Active Directory
• SSO - SAML
• Other

Nextcloud Server logs

Hundreds of "File" is locked :

{"reqId":"l7HW7MVwWoeMkNZ5u5hG","level":3,"time":"2024-11-20T08:34:51+00:00","remoteAddr":"5.48.170.58","user":"vir","app":"no app in context","method":"POST","url":"/remote.php/dav/bulk","message":"\"Photos/2011/photos a reclasser 11642.jpg\" is locked","userAgent":"Mozilla/5.0 (Windows) mirall/3.14.3stable-Win64 (build 20241031) (Nextcloud, windows-10.0.26100 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"30.0.2.2","data":{"path":"/Photos/2011/photos a reclasser 11642.jpg"},"id":"673da21c06ae4"}
{"reqId":"l7HW7MVwWoeMkNZ5u5hG","level":3,"time":"2024-11-20T08:34:51+00:00","remoteAddr":"5.48.170.58","user":"vir","app":"no app in context","method":"POST","url":"/remote.php/dav/bulk","message":"\"Photos/2011/photos a reclasser 11641.jpg\" is locked","userAgent":"Mozilla/5.0 (Windows) mirall/3.14.3stable-Win64 (build 20241031) (Nextcloud, windows-10.0.26100 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"30.0.2.2","data":{"path":"/Photos/2011/photos a reclasser 11641.jpg"},"id":"673da21c06aed"}
{"reqId":"gyKijf5CLunGJGAZm6po","level":3,"time":"2024-11-20T08:34:51+00:00","remoteAddr":"5.48.170.58","user":"vir","app":"no app in context","method":"POST","url":"/remote.php/dav/bulk","message":"\"Photos/2011/photos a reclasser 11641.jpg\" is locked","userAgent":"Mozilla/5.0 (Windows) mirall/3.14.3stable-Win64 (build 20241031) (Nextcloud, windows-10.0.26100 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"30.0.2.2","data":{"path":"/Photos/2011/photos a reclasser 11641.jpg"},"id":"673da21c06af2"}
...

Additional info

Server setup :

• Debian Bookworm
• php-fpm 8.3
• mariadb 11.5.2 (I left the 11.4 LTM version, maybe that was a mistake)

PHP config :

upload_max_filesize = 16G
post_max_size = 16G    
memory_limit = 16G   
max_file_uploads = 100
output_buffering = Off
max_input_time = 3600
max_execution_time = 3600
session.gc_maxlifetime = 3600
default_socket_timeout = 600 # changed from 60 with no change on bug

Edit : I tried to add OWNCLOUD_MAX_PARALLEL=1 to the client config file without success.

[Darnokiv]

I do have the same problem, but on different platforms as well (win10, win11, ubuntu24). The error started as 499, but changed to 299 by now. The symptoms are exactly the same, only the networkerror number shown changed. There are a few forum entries regarding this error from a few years ago, but there never was a concrete solution. Really appreciate @kiv57 opening the issue!

For me the syncing start to fail after the first 100 files as well, then it jumps to syncing file 200 and after this 400, 600, by then i usually restart the sync. Hope this helps identifiying the bug! 😃

Edit: The error just changed back to 499

[kiv57]

I think I found a solution (also read my post in the help nextcloud forum).

Client log showed a 504 Gateway Time-out from the server, followed by exactly 100 files in error 499.

I tried several php and nginx settings, it appears to work after adding the following in my /etc/nginx/sites-available/cloud config file in the container, in the location ~ \.php(?:$|/) section :

fastcgi_send_timeout 600;
fastcgi_connect_timeout 600;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

Note that my NC instance runs in an incus container behind an nginx reverse proxy. I also added some of those parameters on the host side.
After (tens of) hours of testing, I must admit that I haven't kept track of all my files changes.

Correct me if I'm wrong : I naively understand that the NC desktop client uploads the first 100 files one by one, then by packs of 100.
The timeout occurs if the sync of a 100 pack files takes too long.

Edit : I will wait a bit to be sure that the bug is solved for me and Darnokiv before closing the bug.

[Darnokiv]

Thanks for the quick answer. Did you changed the config in the nextcloud container? My nextcloud runs as an app on truenas scale and I never got file editing inside a container working. I will try tomorrow, but right now I'm not sure what I would have to change exactly (within the nextcloud instance, the separate nginx reverse proxy on the server or the client). And since I don't have that much time free at the moment, this may take a while...

But thanks a lot for the help! I would really like to solve the problem in the long run.

[kiv57]

I changed the mentioned settings in the host first (running the reverse proxy), then in the container.

The bug only disappeared when I added the lines to the container's nginx config, but I think the reverse proxy would also close the connection with default settings of 60 seconds. So you should change both sides if you can.

[SK-Crow]

I do have the same problem, but on different platforms as well (win10, win11, ubuntu24). The error started as 499, but changed to 299 by now. The symptoms are exactly the same, only the networkerror number shown changed. There are a few forum entries regarding this error from a few years ago, but there never was a concrete solution. Really appreciate @kiv57 opening the issue!

For me the syncing start to fail after the first 100 files as well, then it jumps to syncing file 200 and after this 400, 600, by then i usually restart the sync. Hope this helps identifiying the bug! 😃

Edit: The error just changed back to 499

Did you manage to fix this? I'm getting a 299 error with the exact same scenario. I am on Freebsd on Truenas Core though. It'll fail and fail and fail until suddenly it starts uploading again. Happens on multi accounts from different countries, so it's clearly my setup. Only started a few months ago around the time that they broke the dark theme on the desktop client.

[Darnokiv]

Sadly, I had no time to find a solution to this. Finding a way to change the .config files in the TrueNAS app was the hurdle had took to long.
My workaround for big files was to pause the synchronization and upload it via the browser. After that I started the synchronization again. For my everyday use I can gladly ignore the error.

I'm sorry I couldn't help but I would be very happy if you give me an update when you find a solution, thanks!

[kiv57]

@Darnokiv my fix above worked for me. Since then I've had several occasions to upload large amounts of files (photos), without problems.

I don't know if it's a problem to have to push the php parameters to such a level - maybe that's another discussion to have before marking the bug as resolved.

[Darnokiv]

Sorry @kiv57 , I meant that I was not able to test your solution yet because I was not able to find a way to modify the .php file through the console in truenas (which has no text editor).

[kiv57]

Thank you for your clarifications.

So we can conclude that this bug still need a "all use cases fix", as installations like yours do not allow to change the appropriate server settings.

[narbux19]

I'm having the same issue: desktop client will upload the first 100 files, then gives Network Error 499 for the next 100 files after a while. This 'a while' is actually 60s, which is the timeout specified in the nginx config. Unsurprisingly, I also found the 504 Gateway Time-out in the Client logs. Suggests to me that something is going wrong combining the next 100 files into a 'bulk' package, causing the timeout?

Like @Darnokiv I'm on Truenas Scale (25.10.1) running Nextcloud in a Docker container. Editing the nginx configuration, even if you manage to do it (this post appears to have a solution), would only temporarily solve the issue given nextcloud's frequent updates.

For whoever is running into this same issue I just wanted to post my workaround: you can switch off bulk uploading by adding the line 'bulkupload.enabled' => false to your config.php file, which in Scale is located in <nextcloud_appdata_dataset>/appdata/config/.

Nextcloud version 3.0.5_1.6.1
Client version 3.15.3 on Win10

[gerroon]

I have the same issue on NC Apache Docker 30.0.5, Win 10 Desktop 3.15.3/ Sending initial 1000s of files is very slow like coupkle KBs/sec and there are a lot of error 499 on the client. Has anyone found a real solution?

[IlyaVassyutovich]

My setup

• Nextcloud: docker container with optimized* and tweaked docker image based upon nextcloud:28.0.1-fpm.
• Nginx: docker container based upon nginx:1.25.3-alpine. Unfortunately, I've completely forgot how I got nginx.conf, but it looks almost like configuration from NC's docs.
• Traefik: vanilla image traefik:v2.10.4, only necessary configuration of middlewares specific to NC
• HAProxy: 2.8.3 on pfSense, TLS termination happens here, otherwise nothing relevant to the issue at hand

* Only minor tweaks here, no breaking and/or big changes were made

Same issue here and it's driving me mad. I've tried setting up fastcgi_send_timeout and other params as per @kiv57 's suggestion to no avail (not with 60s, not with 600s).

Also tried to disable parallelism in client as was suggested here. Also to no avail.

This is terrible and makes my whole setup completely unusable.

UPD: Trying @narbux19 's workaround with disabling bulk upload.

UPD: Disabling bulk upload is indeed a workaround that helped to make it at least work. Sync is extremely slow now, but it's better than no sync at all. I guess I'll toggle bulk upload when I need to sync big changes back and forth. Thank you @narbux19 💜

[gerroon]

@IlyaVassyutovich bulk upload trick didn't do that much of help to me, especially with the time outs.