.couch files becomes .couch.locked and unable to restore databases

[ahmedsaifreza]

all the files become .locked and unable to restore the database. Got the following errors:

This is a single node server

[notice] 2022-06-01T00:00:56.405884Z [email protected] <0.421.0> -------- chttpd_auth_cache changes listener died because the _users database does not exist. Create the database to silence this notice. [error] 2022-06-01T00:00:56.405935Z [email protected] emulator -------- Error in process <0.5498.2815> on node '[email protected]' with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,400}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,375}]},{mem3_shards,load_s>

[nickva]

CouchDB doesn't create .locked files. I am not sure why that is happening. Since you mentioned "restore", since CouchDB doesn't have a built-in backup/restore feature, perhaps it's the software used to restore database files?

[ahmedsaifreza]

Thanks for the response. Not using any software for backup or restore. This .locked issue happened out of no where on my product instance. Lost all the data since couldn’t do anything with the .couch.locked files. Now setting another server to replicate every hour from the production DB. Any suggestions on how to stop data loss due to corruption of .couch file in the future? Thanks in advance. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Nick Vatamaniuc ***@***.***> Sent: Wednesday, June 1, 2022 8:56:36 PM To: apache/couchdb ***@***.***> Cc: Ahmed Saif Reza ***@***.***>; Author ***@***.***> Subject: Re: [apache/couchdb] .couch files becomes .couch.locked and unable to restore databases (Discussion #4045) CouchDB doesn't create .locked files. I am not sure why that is happening. Since you mentioned "restore", since CouchDB doesn't have a built-in backup/restore feature, perhaps it's the software used to restore database files? — Reply to this email directly, view it on GitHub<#4045 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJXRUZAUYXKQFTRFFC5JU3DVM522JANCNFSM5XQAMNMQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[nickva]

If you're not aware of any application you run which creates those files, there is a good chance it's from a ransomware virus https://fileinfo.com/extension/locked

You can check if the criminals left a note, like readme txt file of some sort, in the home or root directory, which would confirm that theory.

[ozgursel]

[paragasu]

my couchdb server also got affected by cerber ransomware last week. Do you actually able to download the decrypter after make the payment? Did you manage to restore your db?

[ozgursel]

Yes, I did.

[nickva]

@ozgursel you can make sure that the shard files in the _dbs db match up with the shard map. https://docs.couchdb.com/en/3.2.2/cluster/index.html

You can make a new database named differently, (say newdb1) with the same Q sharding factor and see what the _dbs shard doc looks like. Then re-create the clustered db doc for your db to match.

[ozgursel]

I saw but didnt try yet. Is there any way the get docs from single .couch file? @nickva

[paragasu]

Are you using cluster setup?

[ozgursel]

It doesn't matter actually.
I figured out to restore DB, thanks to @nickva.
@paragasu but you have to decrypt your files first then I will guide you to restore if you want.
and desperately I can say decryptor comes with a unique key I guess, anyway I send you from mail

[nickva]

@ozgursel you can read the docs from either the :5986 localhost port (a node-local port) or from the :5984/_node/_local/$shardname paths

[ozgursel]

@nickva Is it a possible 0-day vulnerability on CouchDB?

[nickva]

@ozgursel It's possible. There was a recent vulnerability fix for CouchDB https://docs.couchdb.org/en/3.2.2/cve/2022-24706.html make sure to upgrade your instances. It could also be another vulnerability which is used for the attack so it's hard say definitely.

[rnewson]

As a general (and obvious) note you should not pay ransomware authors but restore from a backup on to a clean, secured replacement server. obviously that is not always an option.

If you have recovered your shard files (shards/000-FFF/blah123.couch, etc) you can recreate the _dbs database manually. The format is not documented but it's fairly simple, an example is below. shard_suffix is an erlang string of the .12345 part of the shard filenames (a list of integers where the ints are ascii values).

If you uncover any evidence to suggest your served was compromised via couchdb we are very interested to hear any and all details.

{
  "_id": "dbname",
  "shard_suffix": [
    46,
    49,
    54,
    53,
    52,
    54,
    51,
    53,
    48,
    54,
    56
  ],
  "by_node": {
    "[email protected]": [
      "00000000-7fffffff",
      "80000000-ffffffff"
    ],
    "[email protected]": [
      "00000000-7fffffff",
      "80000000-ffffffff"
    ],
    "[email protected]": [
      "00000000-7fffffff",
      "80000000-ffffffff"
    ]
  },
  "by_range": {
    "00000000-7fffffff": [
      "[email protected]",
      "[email protected]",
      "[email protected]"
    ],
    "80000000-ffffffff": [
      "[email protected]",
      "[email protected]",
      "[email protected]"
    ]
  }
}

[ozgursel]

then put this json using

curl -X PUT http://username:password@yourhost:yourport/_node/_local/_dbs/yourdbname -d '{theonelinejson}'

command

[paragasu]

Hi @nickva can explain more on how to come up with the shard_suffix value?

[rnewson]

It would be great if @ahmedsaifreza, @paragasu and @ozgursel could tell us which version of CouchDB they are each running, so we can eliminate previously patched security vulnerabilities from consideration.

[paragasu]

I am running couchdb 2.3.1

[rnewson]

thank you.

[paragasu]

I am actually running couchdb 3.1.1

[rnewson]

oh and Operating System and version would help too.

[paragasu]

Debian 10

[rnewson]

thanks

[ozgursel]

[ozgursel]

https://www.countercraftsec.com/blog/active-exploitation-of-apache-couchdb-cve-2022-24706/
:(

[rnewson]

sounds like it, yes. You'll need to rebuild the server from trusted sources, it has clearly been compromised and you can no longer trust anything on it to be what it appears.

In addition to using the latest version of CouchDB's installer, you should install a firewall (iptables based is easiest) and only allow inbound connections to port 5984 (or 6984 if you are using TLS, which we recommend).