RocksDB Storage Migrations

[RomanBrodetski]

https://www.notion.so/matterlabs/Versioning-and-compatibility-18-08-proposal-24ea48363f2380e69cd1de4e6b399f84

[EmilLuta]

I assume in progress, there's no PR linked here.

[EmilLuta]

No info.

[EmilLuta]

Work to be done:

1. 10% - latest version of rust crates
2. 30% - ergonomics for library (simplify interface)
3. 60% - integrate it into zksync-os

For 1, almost done, pending PR this week.

[slowli]

Regarding selecting a storage codec (e.g., bincode, Protobuf, RLP etc.) before the storage is migrated, there are following considerations:

• Conceptually, we want to choose whether it would be acceptable to drop the databases after migration and rebuild them from scratch, or to implement data migrations. (Obviously, the first approach would be easier to implement from the migration perspective.) Note that we effectively use the first approach rn when upgrading protocol versions. With the first approach, the choice of current codecs is not very impactful.
• Preliminarily, we would use Protobuf as the main codec (likely with the code-first approach via prost proc macros, as opposed to schema definitions + protoc compilation). However, the matterdb library does support multiple codecs, so it's feasible e.g. to support both Protobuf + bincode (which AFAIU is used for most data currently).
• Important to understand that regardless of the codec used, data migration is likely will never be a no-op, because matterdb enforces more complicated storage layout, with keys prefixed by a collection index. Still, if the codec is preserved, migration becomes simpler (moving values without re-encoding them).

[EmilLuta]

First part is done -- PR here.

@popzxc will move the repo to MatterLabs domain.

@slowli is currently working on part 2.

[popzxc]

Repo has been moved a while ago: github.com/matter-labs/matterdb/

[slowli]

Opened the initial API simplification PR: matter-labs/matterdb#1

[EmilLuta]

I will follow up with a code review.