Add Etcd as a data store backend #742
Conversation
Thank you Alvin for the TiKV implementation Co-Authored-By: Alvin Peters <[email protected]>
|
|
| @@ -71,6 +71,8 @@ impl DistributedBlob { | |||
| Store::MySQL(store) => store.get_blob(key, read_range).await, | |||
| #[cfg(feature = "rocks")] | |||
| Store::RocksDb(store) => store.get_blob(key, read_range).await, | |||
| #[cfg(feature = "etcd")] | |||
| Store::Etcd(_) => unimplemented!(), | |||
There was a problem hiding this comment.
how can I declare that my backend will not be able to handle blobs and avoid such lines ?
There was a problem hiding this comment.
@mdecimus could you help me please?
There was a problem hiding this comment.
You should implement etcd as a lookup store, not a data store. Check the Redis implementation for guidelines.
There was a problem hiding this comment.
Well, I want to store data but not blobs:
- Data store
- Full-text store
- Lookup store
https://stalw.art/docs/get-started
Should I still implement blobs even if this is not recommended with etcd ?
I really want that this implementation checks all boxes on the https://stalw.art/docs/get-started page but not blobs. As I want Garage/S3 to store them.
There was a problem hiding this comment.
Well, I want to store data but not blobs:
That is not possible with the current design. A data store has to offer also blob and lookup functionalities in order to be functional.
I don't have experience with etcd but according to their website it was designed as a store for settings and metadata. For this reason it should be added as a lookup store rather than a data store.
There was a problem hiding this comment.
Okay, but my true question is if I do it as a lookup store I can not store data into it
no FTS
And I will have basically done the same as Redis and this is not solving my infrastructure problem
There was a problem hiding this comment.
What is your use case exactly? And why are you choosing Etcd over FoundationDB or PostgreSQL? It makes sense as a lookup store but I wouldn't use it for FTS or blob storage as it is not the right tool for the job.
There was a problem hiding this comment.
my use case is finding the right softwares that scale across multiple data centers. Multi master multi read. The worst setup you could imagine.
Garage works perfectly out of the box for S3, great. The have a KV api but not what we need for this project.
Etcd seems to be built to work in the same way, but only with implementation I can be sure.
PostgresSQL is terrible to configure for multiple datacenter. A nightmare.
FoundationDB should have worked but did not and was nonsense to setup across multiple data centers.
I use LDAP for accounts, perfect no need to have write access. There is no writes to be made anyway on my setup. And for so little writes managing conflicts would be easy.
Multiple data centers with multi Read and write is a complicated setup I know ^^
The average round trip is 60ms in the worst case.
Can you elaborate more about why it would not be suitable for FTS?
Does it store large data?
There was a problem hiding this comment.
Does it store large data?
Not if you use an external blob store. If you are sure that etcd can store large amounts of data go ahead, but it has to implement the blob store methods even if they are not used.
Also, if this featured is merged it won't be distributed by default in order to keep the binary size small since etcd is not a popular choice for storing indexes.
Ref: #634
See:
etcd-serverandetcd-clienton Debian/UbuntuGoals: provide a backend to store everything I need to store
Blob store(etcd is not made for large data)https://stalw.art/docs/get-started