Better persistence and compaction for Ahnlich-DB to override #9 #255
Description
Persistence is a second class citizen for in-memory databases but still I hate our current model of persistence (it just serves the quick and dirty I built it for initally). The obvious issues are
- We reserialize the entire database at fixed intervals
- The snapshot size grows linearly with the store size
- Due to the first reason, as the store size grows, we end up spending a longer time to be able to serialize the entire snapshot
If we are going to go for a more popular concept of an Append Only File, we could shave down some of these annoyances. We log every mutating command e.g (CREATE, SET, DELETE, DROP) in order of occurence and then we can "replay" these on startup of the database. This would also let us do things like periodic compaction and we can easil
The first issue that sticks out in my mind to solve would be order of occurence... given that we use a lock-free concurrent data structure (papaya) underneath, there doesn't seem to be an ideal view of the "order" of events in which an operation happened