[blog] Qdrant Version 1.14 #1529
Conversation
✅ Deploy Preview for condescending-goldwasser-91acf0 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
|
||
| **Ending our Reliance on RocksDB** | ||
|
|
||
| The **mutable ID tracker no longer relies on RocksDB**. This continues our journey toward minimal external dependencies. With our custom-built [**Gridstore**](/articles/gridstore-key-value-storage/), you can expect fewer random compactions and more predictable disk usage. |
There was a problem hiding this comment.
The new mutable ID tracker does not rely on RocksDB.
It's another kind of custom storage, which is very specialized to its purpose - the mutable ID tracker.
How it works is actually very simple: we have point mappings, and we store all changes to these mappings in a file simply by appending each change as an entry.
If we add a point, we create a mapping, thus we append a 'create point mapping for point x' entry to the file. If we delete a mapping, a 'delete point mapping for point x' is appended to the file. On load we simply walk over all change entries and reconstruct the mappings in memory.
Of course, this could grow forever if there are a lot of changes.
But, this integrates very well with our optimizers. The optimizers ensure that if the ID tracker grows large, it is picked up for optimization. In that case it is transformed into the immutable ID tracker, which benefits from other optimizations. That's why this simple mechanism in the mutable ID tracker is very effective here.
Co-authored-by: Tim Visée <[email protected]>
Co-authored-by: Tim Visée <[email protected]>
Co-authored-by: Kacper Łukawski <[email protected]>
Co-authored-by: Kacper Łukawski <[email protected]>
Co-authored-by: Kacper Łukawski <[email protected]>
Co-authored-by: Kacper Łukawski <[email protected]>
Rendered