Improvements to API, storage methods, schema flexibility

[weex]

To suppore more clients including iOS and Android, more efficient storage and retrieval or documents, a more flexible document schema and document encryption we want to make some changes to Causeway. This issue is proposal and discussion of such changes. It doesn't have to be in-depth but we just want to document the principles we're following.

[alexanderdushkin]

As I initiated this whole thing, let me mention what I'd like to change:

• make the API RESTful, e.g. /jobs instead of /query?query=jobs;
• make the API return what a user needs exactly, and not raw data, e.g. give out a list of active jobs and not a list of all jobs;
• edit/delete data in buckets with proper authentication mechanisms;
• think of a more efficient signature checking mechanism (been discussing this with weex);
• wrap the whole thing up into NGINX + Docker, enable caching to make it more production-style.

Additionally, I really see it working on a document-based storage engine like MongoDB. I understand many of you are familiar with SQL-like databases, but a document-oriented DB would be easier to set up and work with in code, considering that Akrmn is doing document JSONifying now. The entire Rein is based on signed document exchange, so Mongo just feels right here.

However, as I mentioned while discussing in private with weex, any database engine would work.

[piratelinux]

Not sure what happened to the work on document sharing. But I think it would be good if causeway servers could share documents with each other. Clients just have to share what servers they are using and then the servers can share the content between each other, though it can be optional what servers clients reveal.

I think Mongo DB is good, I used it recently for some other projects, so it just depends if you feel like transitioning to it. It does feel more natural with json, and should be more efficient than SQL if you have deep trees of data in json format.

edit: Also could keep the DB we have now and add the mongodb just for the json documents, so you just store the id of the json document in the sql db, and use the mongodb for getting more details on it.

Also, there should be a quick way to know if one database of jobs on one server is different than that on the other server, maybe do a merkle tree of posted jobs, so you can easily check.

[weex]

Since I'm running the two servers and can see load isn't an issue for now, I'm onboard with api changes but Mongo, Docker and nginx I'll leave for anyone else who wants to run a server.

Besides my own practical concerns I want to ensure that servers don't need to be trusted any more than necessary. This means if there are api calls that bundle data for efficiency, there still remains a way to get the more complete raw data.

[alexanderdushkin]

@weex, sorry for the late reply. Obviously, there will be an API method to get raw data.

Firstly, the built-in Flask server is not suitable for production use according to the developer himself. As far as I know, Flask also does not support SSL, which we need for the iOS app. We do need NGINX for Causeway, one way or the other.

If we use NGINX and want all the software to be deployed with just one command on a new server, Docker is a must. It's a way to make the boilerplate configuration activity be done automatically. I see @gutsal-arsen has already submitted a pull request about Docker so I guess using it is not an issue.

What about database engines, I can't argue. PostgreSQL/SQLite 3 is your choice and I should comply or build my own fork which doesn't make much sense if I'd be building the standard version anyway.

[weex]

If you can make it easy for me to run, have it all come back up if the server restarts and with no stability issues I'm fine running the Docker, nginx, mongodb combo.

[alexanderdushkin]

Yes, obviously. I proposed this combo with stability and high availability in mind.