Support for cross-database links

[lvca]

SUMMARY

Status: Design
Progress: Design 50% -> Development & Unit Tests 0% -> Integration Tests 0%
Currently Assigned: @lvca
ETA: 6 days

---

With ArcadeDB links are expressed with RIDs. A Rid is composed by 2 numbers: #<bucket-id>:<offset-in-bucket>. If we create a new Cross RID (CRID from now on) by adding the database id, it would be like this:

#<database-id>:<bucket-id>:<offset-in-bucket>

The database-id is a number. This means the server must keep track of database-name/number in a map.

Once this is available, the user should be able to load records from other databases. This is perfect for multi-tenant applications, where each tenant has its own database, but the common data can reside on a central database.

[lvca]

Quick update on this feature.

This could be considered a federated database.

The map database-id/path should be per database in the server. It could be like the security file, stored in a directory of the server's config. This allows declaring which databases a database can cross-link and not assuming just any databases configured in the server. It could be something like:

federated-database.json:

{
  "databases": {
    "0": {
      "url": "file://usa-florida",
      "user": "external",
      "password": "XXXX"
    },
    "100": {
      "url": "remote://server-Italy:2480",
      "user": "external",
      "password": "YYYY"
    }
  }
}

Whereas the key in the databases map contains the id as a key and the configuration as a value with the URL of the database (local or remote) and the credentials used to login on to the external database.

[lvca]

@frank-dkvan there is no high request behind this feature, right now it's a low priority unless we find a sponsor for it. What's your use case? Please tell us more about how you're using ArcadeDB!

[frank-dkvan]

Hi @ivca, the use case is actually we want to support partitioned data storage on many distributed data servers since the data volume is very huge. currently I only see ArcadeDB supports distributed data replication for HA. But I don't see ONE graph data can be distributed partitioned and stored on different data server nodes,i.e. partitioned data A stored on Data Server A with Data Replication Server B and C, partitioned data B stored on Data Server D with Data Replication Server A and B, etc... that's why I assume the mechanism "#::" might be a workaround to support this case and eager to see it :-). waiting for ur suggestion. thanks!

[frank-dkvan]

@lvca What I need is actually data sharding features, like MongoDB data sharding. so that larger data can be balanced and distributed in different hardware nodes...

[frank-dkvan]

hi @lvca do we have a plan to support data sharding?

[lvca]

@frank-dkvan you're right, by using cross-db links it would mean to allow sharding of data by using multiple databases. This was the original idea.

[brandonzylstra]

I assume this is still not done, since a search mainly leads to this page—is that assumption correct?

[lvca]

Correct, it's not implemented yet. We have a couple of users that are cross linking resources across databases by storing the RID as a string "#2:10:3", where 2 is the 2nd database in a configured list and 10:3 the actual RID in the target database.

They basically decompose the RID by separating the first part, and executing a lookup on that database. It could be an easy workaround for now in case it's needed