Per-user Database document access control

[wa0nder]

Edit - added a little clarification to the example below

Determine which documents and document fields are returned by a GET request applying a process virtually identical to 'validate_doc_update' functions.

Summary

Disclaimer: I have already seen the various opened issues about 'per document access' here, here and here and this isn't the same as those, at least as far as I can tell.

What I have in mind is to apply a validate function to not only PUT requests as done currently but also to GET requests to allow individual db users to completely control which documents and which part of those documents they want to share.

Desired Behaviour

When a GET request is made, CouchDB looks for a 'validate_doc_request' (VDR) function (similarly to a 'validate_doc_update' function) and, if it exists, the requested document is passed to it. Whatever the VDR returns due to filtering/restriction code is what is sent to the requesting client.

Possible Solution

When a PUT request with a document is made, a 'validate_doc_update' (VDU) function is called to apply restrictions, filtering, etc. In the same way, when a GET request for a specific document is made, run a 'validate_doc_request' (VDR) function.

CouchDB would grab the requested document and send it to the VDR which would be able to apply filtering, restrictions, conditional statements, security object considerations, etc. and return only a subset (or empty) of that document along with the '_id' and '_rev' fields.

All returned fields would contain a path from root to 'position in document' in order to keep track of where data subsets originated in the document.

After a user modifies the returned document and sends it back, CouchDB would run a 'merge subset with parent document' function before sending to VDU for validation. For security reasons, if a returned value's 'path' cannot be matched in the original doc then the merging of that value will be ignored. That's all.

Here's a quick example to demonstrate.

1. Sample document in DB:

{
  _id: 'testDoc',
  _rev: ....,
  field1: {
    field1_a: {
      name: 'bo'
    }
  },
  field2: ...
}

2. Let's pretend that after some VDR runs it spits out:

{
  _id: 'testDoc',
  _rev: ....,
  field1_a: {
    path: 'field1/',
    value: { name: 'bo' }
  }
}

3. After user modification client-side some fields are added:

{
  _id: 'testDoc',
  _rev: ....,
  field1_a: {
    path: 'field1/',
    value: { name: 'jane', field1_a_1: [...] }
  }
  field_wrong: {
    path: 'field4/',
    value: [34, 5, ...]
  }
}

4. After submission back to CouchDB and the merge function is run:

{
  _id: 'testDoc',
  _rev: ....,
  field1: {
    field1_a: {
      name: 'jane',
      field1_a_1: [...]
    }
  },
  field2: ...
}

*note - field_wrong is ignored because there is no matching path in original document

5. Now the VDU will be sent the original doc as 'oldDoc' and this merged doc as 'newDoc'. Everything else proceeds as normal.

That's it. Please let me know if I need to clarify further. Also, if this is infeasible for "behind the scenes" Erlang restrictions and/or CouchDB architectural reasons let me know and I can remove this proposal. Thanks!

Additional context

Despite no longer being recommended from a development perspective, I'm still interested in the two-tier stack idea of CouchDB and what sorts of applications can be created by making the most of the built-in server capabilities.

[janl]

Heya, thanks for taking the time to think this through and to write it up.

The major issue with this approach is that reverse-VDU would be prohibitively slow for general operation. We are already in talks of redesigning the current VDU system to improve validated-write speed, so adding another slow path does not sound like a good idea :)

In addition, this approach ignores other features of CouchDB, like _all_docs and _changes, as well as _revs_diff, all of which are essential to CouchDB use, so it is not practical for a full solution.

If you have any feedback on https://lists.apache.org/thread.html/rf75e0b8266c678bcfdc2e5655317c716f5d317e9cb2f75b326820311%40%3Cdev.couchdb.apache.org%3E that’d be most appreciated, as this is the closest to being shipped.

[wa0nder]

Thanks for responding! It seems that you're the only? one working on this type of feature so, really, thanks for your time. I will also take a look at what you've already submitted for sure.

A couple of quick questions since I haven't hacked on the internals of CouchDB yet:

1. What makes a validation path in general slow? For security reasons validation has to happen in some shape or form before data hits a database for any database. Is this a particular problem in CouchDB because some people use it simply as a "dumb" database with a separate application server layer and others use it with the two-tier stack in mind and there's a need to attempt to optimize for both use cases with minimal tradeoff?

2. Also, I double checked the links for _all_docs, _changes and _revs_diff and I don't understand (again, I fully admit I'm not familiar with CouchDB internals) how a VDR would be incompatible with those features. Specifically,

• a. _all_docs endpoint request would either be disabled if a VDR is present or only allowed if the security object shows the request is made by the db owner.

• b. _changes is only concerned with documents that made it past the VDU to be PUT in the database, right? Since adding a VDR requires a 'merge' step on the VDU side of things, this would mesh perfectly with the changes feed since the 'merge' step, by functionality, has to track precisely the changes a user submits back to the db.

• c. _revs_diff is concerned with revision numbers and a VDR or a 'merge' step doesn't affect revisioning at all.

I'm also not sure if the title is worded unclearly so to clarify, this suggestion is meant to work with the per-user db pattern.

thanks

[janl]

1. What makes a validation path in general slow?

• serialisation in JSON
• passing to a separate OS process (couchjs)

2. a. We have considered this as inadequate, as folks would want to be able to use PouchDB/Replication with this, which requires _changes.

b. for _all_docs and _changes, imagine a database with 1 million documents, and 5 docs per user. How would you efficiently get the 5 docs, either in _all_docs (_id) or _changes (seq) order? the answer can’t be, load 1M docs, and through 999995 away, based on the VDR.

b. I’m not sure your your “merge” approach helps keep documents apart from untrusted users.

c._revs_diff returns revision ids, which are content addressable hashes for document JSON bodies. You don’t want to leak that you might have known a specific version of a document, so ACL will have to be applied.

this suggestion is meant to work with the per-user db pattern.

The current per-doc-access-control idea is so that folks can avoid using db-per-user.

[wa0nder]

b. for _all_docs and _changes, imagine a database with 1 million documents, and 5 docs per user. How would you efficiently get the 5 docs, either in _all_docs (_id) or _changes (seq) order? the answer can’t be, load 1M docs, and through 999995 away, based on the VDR.

hmm, I think we're talking past each other a bit here. The assumptions behind the idea hinges on the 'db-per-user' pattern so if you are trying to get away from that then I can see the disconnect where my proposal doesn't really make sense.

Assuming for use-cases where db-per-user is fine and/or desired, my proposal means very little needs to change in terms of CouchDB's existing architecture because it leverages an existing pattern (the VDU concept). Of course, if the team is trying to get away from or optimize VDU's then I can see the problem there.

b. I’m not sure your your “merge” approach helps keep documents apart from untrusted users.

The 'merge' step doesn't keep documents apart from users, the VDR/VDU does that. The 'merge' step is to allow subsets of documents to be re-integrated into the original document since the VDR makes it possible to limit how much of a document gets out to a user.

c._revs_diff returns revision ids, which are content addressable hashes for document JSON bodies. You don’t want to leak that you might have known a specific version of a document, so ACL will have to be applied.

As with '_all_docs', only allow '_revs_diff' to be returned to the owner of the db.

In any case, I did read this in your RFC:

handling many small databases, say >10000 (depending on hardware) can become a challenge, if most of them are active concurrently. It forces dbs to be set to q=1, migrating off q!=1 requires downtime, 10k bidirectional replications are going to need A LOT of CPU and RAM. sharing documents among two or more users requires the creation of yet more databases.

I was thinking of applying the 'db-per-user' pattern to allow many users to have their own sandbox in which to do what they liked but this sounds like 'db-per-user' is not a good idea from a technical standpoint. Are there plans to retire 'db-per-user' in the future?

[janl]

there is no plan to retire db-per-user, but any improvements to CouchDB will be adding things like the per-doc-access feature that helps folks not to use db-per-user.

[wa0nder]

ok gotcha, that's good to know. Thanks again for your help

[wa0nder]

@janl

I took a look at the RFC and noticed these two lines:

Non-goals for now:
...

• sharing individual documents between multiple users or groups.

users can only create documents with their own username in _access.

It is the first issue that I'm attempting to solve without a separate application layer and why I made this discussion. I am interested to see how, as you said in the RFC, "the design of this iteration aims to allow turning these non-goals into actual goals later". Without digging deeper into CouchDB's internals I don't feel qualified yet to give further technical feedback/critique.

Thanks again for your efforts.

[janl]

keeping group sharing out of this is for implementation simplicity (this is tricky enough as-is), and on CouchDB 2/3.x this will have limits that CouchDB 4 won’t have, which make the runtime characteristics of group sharing easier. per-doc-access as designed today allows to be extended in CouchDB 4 and beyond to support group sharing.

[SinanGabel]

(1) Question: can a user also be a role?

I understand that it would work like this i.e. having a key:value inside each document:
{..., _access: ["shirley"], ...}
and thus "shirley" would have access to that document.

But will it also be possible to use a role for the access criteria?

(2) Quetion: In terms of speed and performance will it make any difference if one utilizes the partitioned database structure i.e. places all documents of a particular user in a specific partition?

Thank you in advance.