Initially Pryv.io was built on top of MongoDB with separated collection per user. This initial design allowed to isolate peruser data on the file system.
This design has a drawback as MongoDB was consuming a fixed amount of RAM per collection and the with growing sets of users (over 40'000 per node) 16Gb was needed. In v1.6.0 an option to merge the Mongo's collection was added, resulting in an average RAM requirement of 4Gb for 100'000 users.
From v1.7.0 Sqlite has been investigated in order to provide back the ability to isolate peruser data on the file system. The motivation is to provide full control over the user's data in order to facilitate and prove the "right to be forgotten."
From v1.8.0 a Sqlite version for Event has been provided on top of the datastore abstraction.
Since v1.9.2 Pryv.io can be deployed in "full-cloud" setup without relying on the file system. This can be done by configuring all storage modules to use MongoDB. For the attachments and S3 implementation is in development.
For future v1.9.3 Pryv.io will be also capable in being "full local" with only SQLite databases.
In v2 the default baseStorage engine is PostgreSQL (storages.base.engine: postgresql in config/default-config.yml). MongoDB remains fully supported — set storages.base.engine: mongodb in override-config.yml, or run the test harness with just test-mongo instead of just test.
base code: components/storage/src/userLocalDirectory.js
Localization of user data on the host file system, usually in var-pryv/users then a directory path is constructed using the 3 last characters of the userId and the userId.
Exemple with userId c123456789abc: var-pryv/users/c/b/a/c123456789abc/
In this directory, the attachments and any user attributed data and sqlite db should be stored.
base code: components/storage/src/userLocalIndex.js
This database is a per-server index to map userId and userName. In the future it could be extended to allow user aliases.
- With SQLite (default) the db file can be usually found at
var-pryv/user-index.db - With MongoDB the collection is
id4nameand stored in the main host databasepryv-node
Settings to activate MongoDB instead of SQLite: storageUserIndex:engine = 'mongodb'
base code: components/storage/src/userAccountStorage*.js *: Mongo or Sqlite
This database contains the password and passwords history of the user.
- With SQLite (default) it can be found in the "User local directory" named as
account-1.0.0.sqlite. - With MongoDB the collection is
passwordsand stored in the main host databasepryv-node
Settings to activate MongoDB instead of SQLite: storageUserAccount:engine = 'mongodb'
base code: components/platform
This database contains all indexed and unique fields for users such as emails and custom systems streams data, plus the user→core mapping in multi-core deployments.
Since v2 the platform DB is always rqlite (distributed SQLite). bin/master.js spawns and supervises an embedded rqlited in single-core mode (one node) and in multi-core mode (each core runs its own node, joined into one Raft cluster via DNS discovery on lsc.{dns.domain}).
- Data lives in
var-pryv/rqlite-data/(Raft log + SQLite snapshot) - HTTP API:
http://localhost:4001(default) - Other engines (mongodb, postgresql) still ship
PlatformDBimplementations for conformance tests, but rqlite is the only engine that can be selected at runtime viastorages.platform.engine
base code: components/storage/src/localDataStore and localDataStoreSQLite
Main storage for events , streams & attachments this implementation follows the modular API of datastore abstraction.
- Fully implemented with MongoDB
- Only events are implemented with SQLite - Expecting full SQLite implementation in v1.9.3
base code: components/storage/src/user
Only implemented for MongoDB - Expecting full SQLite implementation in v1.9.3
- test B2I7 is failing when testing
storagewithfull-mongoas indexes for password is not yet created. Runjust test-full-mongo storageto reproduce