Nextcloud desktop removed all files from server

[rbu]

Expected behaviour

Nextcloud Desktop should not remove files from the server that are not gone from the machine.

Actual behaviour

Two days ago, Nextcloud desktop removed all my files from the server, including folders shared with others. It also removed myself from shares (to me) of others.

The files on the local system stayed intact. However, recovering from this took me significant amounts of time, sweat and conflict solving. Plus, all of the folder timestamps on the server are now gone.

Steps to reproduce

I hope I cannot reproduce this.

I'm seeing frequent crashes of the Nextcloud client. There's high memory usage that I attribute this to (#124), sometimes up to 1.5 GB of RAM when it's been running for a while.

Client configuration

Client version: 2.5.3

Operating system: Fedora 30

OS language: English

Qt version used by client package (Linux only, see also Settings dialog): 5.12.4

Client package (From Nextcloud or distro) (Linux only): nextcloud-client-2.5.3-1.fc30

Server configuration

Nextcloud version: 16.0.4.1

Logs

Access log:

91.XX.XX.XX - rbu [18/Sep/2019:13:57:29 +0000] "DELETE /remote.php/dav/files/rbu/projects HTTP/1.1" 204 0 "-" "Mozilla/5.0 (Linux) mirall/2.5.3git (Nextcloud)" "91.XX.XX.XX"
...
91.XX.XX.XX - rbu [18/Sep/2019:13:57:36 +0000] "DELETE /remote.php/dav/files/rbu/pictures HTTP/1.1" 204 0 "-" "Mozilla/5.0 (Linux) mirall/2.5.3git (Nextcloud)" "91.XX.XX.XX"

[rbu]

This just happened to me AGAIN. It started removing everything in one subfolder and then crashed after removing a few thousand files from it. I have a complete (non debug) log of this process in the client. The first line that starts the delete is this:

[OCC::ActivityListModel::removeActivityFromActivityList         Activity/Notification/Error successfully dismissed:  "Could not remove folder '/home/rbu/Documents/X/2005' "
[OCC::ActivityListModel::removeActivityFromActivityList         Trying to remove Activity/Notification/Error from view... 
[OCC::ActivityListModel::removeActivityFromActivityList         Activity/Notification/Error successfully removed from the list.
[OCC::ActivityListModel::removeActivityFromActivityList         Updating Activity/Notification/Error view.
[_csync_merge_algorithm_visitor         INSTRUCTION_IGNORE             client dir:  X/.Y
[csync_reconcile        Reconciliation for local replica took  0.035 seconds visiting  33230  files.
[_csync_merge_algorithm_visitor         INSTRUCTION_REMOVE             server file: A/B/C
[_csync_merge_algorithm_visitor         INSTRUCTION_REMOVE             server file: A/B/D.xml
[_csync_merge_algorithm_visitor         INSTRUCTION_REMOVE             server file: A/B/E.js

Since the log file contains tons of private file names, I can't easily upload it here unfortunately.

[rbu]

This just happened to me a THIRD time within one week. It removed ALL the files from the server and crashed syncing the changes down.

I've disabled the client for now, as it's simply running amok on my files. I have a log file for this incident and the previous comment's incident of partial deletion that I'm happy to answer questions with.

[DominiqueFuchs]

Hi @rbu , really sorry for your experience with the client. However, to find the root cause for this it´s absolutely crucial to be able to reproduce this over here (if I just set up the client on my standard fedora box, connect it to my fresh NC16 docker instance with a user having a random tree of online folders nothing unexpected happens).

I understand you do not want to expose file names by posting a complete log. Few things though:

1. Could you use the exact same machine (/fedora installation) and use the affected client with a different testuser and throw in some dummy files? May be very helpful to see if the same issue appears after some time.
2. Any infos about the server instance differing from a standard installation? Any specific apps activated?
3. Which filesystem are you using on your client machine? XFS, ext4, btrfs?
4. Specific addendum to Update README #2: is E2E activated on the server (if yes - is it activated on your folders?)

[rbu]

Hey @DominiqueFuchs, thanks for the response. Regarding the log, if you have specific questions regarding the log or a way to share it privately with an individual, I could do that.

1. That's a good idea, however I probably won't have the time to do this in the next 10 days (on vacation next week)
2. The server is privately hosted (on Docker/Fedora). I'll include a list of enabled plugins [1]. There's a (small) "Samba" remote share enabled in my files (which is the only thing that did not get deleted during any of those incidents).
3. The client is on btrfs, same is the server. There's about 40k files in over 1k directories in my personal folder.
4. Not activated in the client or server.

Edit: I'd like to add one thing that I found odd. I remember seeing error messages about "too many open files" (in red, part of the sync status bar) before/during one of these incidents happening. I wasn't sure if it's too many network connections, open files or maybe it failed to set up inotify listeners? Note that my client system allows a large number of inotify listeners: fs.inotify.max_user_watches=524288
I found a lot of lines like this in the log files (which could explain why the "Scans" samba share did not get deleted):

[_csync_merge_algorithm_visitor 	INSTRUCTION_REMOVE             server file: Scans/2015/document2015-08-19-131546.pdf
[OCC::SyncEngine::checkForPermission 	checkForPermission: RESTORING "Scans/2015/document2015-08-19-131546.pdf"
[OCC::PropagateItemJob::scheduleSelfOrChild 	Starting INSTRUCTION_NEW propagation of "Scans/2015/document2015-08-19-131546.pdf" by OCC::PropagateDownloadFile(0x558b531cffd0)
[OCC::PropagateItemJob::done 	Could not complete propagation of "Scans/2015/document2015-08-19-131546.pdf" by OCC::PropagateDownloadFile(0x558b531cffd0) with status 2 and error: "Not allowed to remove, restoring; Restoration Failed: Too many open files"
[OCC::ActivityWidget::slotItemCompleted 	Item  "Scans/2015/document2015-08-19-131546.pdf"  retrieved resulted in  "Not allowed to remove, restoring; Restoration Failed: Too many open files"
[OCC::ActivityWidget::slotItemCompleted 	Item  "Scans/2015/document2015-08-19-131546.pdf"  retrieved resulted in error  "Not allowed to remove, restoring; Restoration Failed: Too many open files"

I saw the first incident maybe 3 days after upgrading the fedora package from 2.5.2-1.fc30 to 2.5.3-1.fc30. I have since my last comment downgraded to 2.5.2-1.fc30 and haven't seen this problem again.

[1] List of enabled plugins

• Accessibility
• Activity
• Auditing / Logging
• Bookmarks
• Brute-force settings
• Calendar
• Collaborative tags
• Comments
• Contacts
• Deleted files
• External storage support
• Federation
• File sharing
• First run wizard
• Gallery
• Log Reader
• Monitoring
• Nextcloud announcements
• Notes
• Notifications
• Password policy
• PDF viewer
• Plain text editor
• Privacy
• Recommendations
• Right click
• Share by mail
• Support
• Tasks
• Theming
• Update notification
• Usage survey
• Versions
• Video player
• Viewer
• Group folders
• Optical character recognition
• Default encryption module
• Full text search
• Full text search - Elasticsearch Platform
• Full text search - Files
• Full text search - Files - Tesseract OCR
• LDAP user and group backend
• Social sharing via email
• Talk

[rbu]

This has not happened in the 3 weeks since I downgraded from 2.5.3-1.fc30 to 2.5.2-1.fc30. I guess I'll just stay on that version for the foreseeable future 🤷‍♂️

[rbu]

I thought I'd give this another chance and BOOM... all files are gone... again, on 2.6.2. That's after about 2 weeks of usage. I restored and had another few days of it not deleting all files.

[luspi]

Just wanted to say that this happened to me today, I caught the desktop client in the middle of deleting all files on the server INCLUDING files that I do not have locally but only on the server. The activity log shows no errors but lists a selection of files that were supposedly deleted . Thankfully I'm backing my server up regularly...

Desktop client: 2.6.4
Nextcloud: 18.0.1
Server OS: Ubuntu 18.04

[Bun-Bun]

This also happened to me today. During the night while I was asleep all the files on my server were deleted. Server was also full because the files were also being re uploaded (thus in the main storage and trashbin) by the client that deleted the files from the server. Local files on that client appear to be ok. Local files on all other connected clients were deleted.

I am in the process of cleaning this up and restoring from a server backup, but would really like to figure out how to prevent this in the future as it has taken most of my day.

Confirmed that it was the one client by looking in the servers access-logs

xxx.xxx.xxx.xxx - Bun-Bun [14/Mar/2020:04:19:16 -0600] "DELETE /remote.php/dav/files/Bun-Bun/wallpapers HTTP/1.1" 204 - "-" "Mozilla/5.0 (Windows) mirall/2.6.2stable-Win64 (build 20191224) (Nextcloud)"

I only have one client of that version installed on my home IP. One of those lines for each of the items in the root of my nextcloud folder.

Server OS: Centos 7
Desktop client: Windows 2.6.2 (Win 10 Pro 1803)
Nextcloud: 14

[Matthiasfranck]

The samen happened to me today. Has someone found a solution to solve this? Ik would like to use Nextcloud again, but with such as risk of loosing data this is not possible.

[Bun-Bun]

My solution was to install Nextcloud-2.3.3.1 client and configure it to not check for updates.

That is the last version of the client before they started screwing everything up with it.

[skjnldsv]

Happened to me again today.
Got a DELETE request for all files/folders in root.

2020-04-23T06:21:35.283574205Z 192.168.1.110 - skjnldsv [23/Apr/2020:06:21:35 +0000] "DELETE /remote.php/dav/files/skjnldsv/Wallet HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Linux) mirall/2.6.4git (Nextcloud)" "192.168.1.110"
2020-04-23T06:21:35.312710439Z 192.168.1.110 - skjnldsv [23/Apr/2020:06:21:35 +0000] "DELETE /remote.php/dav/files/skjnldsv/Wallpapers HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Linux) mirall/2.6.4git (Nextcloud)" "192.168.1.110"
2020-04-23T06:21:35.365646789Z 192.168.1.110 - skjnldsv [23/Apr/2020:06:21:35 +0000] "DELETE /remote.php/dav/files/skjnldsv/Ressources HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Linux) mirall/2.6.4git (Nextcloud)" "192.168.1.110"
2020-04-23T06:21:35.707730232Z 192.168.1.110 - skjnldsv [23/Apr/2020:06:21:35 +0000] "DELETE /remote.php/dav/files/skjnldsv/Recipes HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Linux) mirall/2.6.4git (Nextcloud)" "192.168.1.110"
2020-04-23T06:21:36.191836604Z 192.168.1.110 - skjnldsv [23/Apr/2020:06:21:36 +0000] "DELETE /remote.php/dav/files/skjnldsv/Projets HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Linux) mirall/2.6.4git (Nextcloud)" "192.168.1.110"
... etc etc