What has changed?

[l-monninger]

Summary

Using this discussion to capture commentary on what has changed in the aptos-core state database since we forked.

movement forked at the following commit: aptos-labs/aptos-core@56f4002

maptos-core forked at the following commit: aptos-labs/aptos-core@74f9c0e

[l-monninger]

aptos-db commits of interest, pt. 1

🟢 = unlikely source of conflicts or incorrectness
🟦 = may cause conflicts without proper upgrade path but should not affect correctness
❌ = likely affects correctness

• 🟢 Hot state (CachedStateView) config update: just seems to expose cache invalidation parameters. No DB representation alterations.
• 🟢 Hot state max size: no DB representation alterations.
• 🟢 Hot state eviction: no DB representation alterations.
• 🟢 JMT verification of leaf count before committing: just for integration with cache invalidation it seems.
• 🟢 JMT deletions: also seems to be motivated by cache invalidation.
• 🟦 New shard metrics: this doesn't affect fundamental DB ops. But, we will likely need to figure out whether there is a particular upgrade path.
• 🟢 Introduces hot state: uses a DashMap for in memory storage; doesn't look like any clear logical dependencies on persistent format.
• 🟦 Usage of cache in speculation API: this looks all in in memory, but understanding the assumptions behind the unreachable statements is probably worth a closer look.

[l-monninger]

aptos-db commits of interest, pt. 2

🟢 = unlikely source of conflicts or incorrectness
🟦 = may cause conflicts without proper upgrade path but should not affect correctness
❌ = likely affects correctness

• 🟦 Sparse (in-memory) Merkle Tree no longer doubles as speculative state: my guess is this should work without much of an upgrade path, but perhaps there's something earlier in the commit history.
• 🟦 Indexer batch processing includes instance: I think this only changes how the indexer would behave.
• 🟢 State Merkle truncation: doesn't change representation just usage. I don't think there are any obvious added assumptions being made here.
• ❌ Move DoStateCheckpoint off critical path: widespread changes to ColumnFamilies--particular schema DB. Affect Merkle DB and regular DB it seems. There's a small chance this is sharding only, but I'm still poking around.
• 🟦 Write block info schemas if sharding features enabled: sharding only.
• 🟦 Indexed DB Event V2 translation: doesn't seem to affect core DB, but this is of course based on a new Event structure.
• 🟢 State sharing between StateStore and BufferedStated: doesn't seem to affect representation.
• 🟢 State view updates for sharding: doesn't seem to affect representation.

[l-monninger]

schemadb commits of interest, pt. 1

🟢 = unlikely source of conflicts or incorrectness
🟦 = may cause conflicts without proper upgrade path but should not affect correctness
❌ = likely affects correctness

• ❌ Migrate event and transaction schemas: move a of read operations over to the indexer reader and makes updates to the indexer db schema. Still unclear whether this move out into a new DB side-steps migrating table info, i.e., whether we can just introduce this. Also still unclear whether fetching behind migration version is well-supported at this commit.

[l-monninger]

#33 (comment) shows that the new StateView is empty. I'm wondering if this has something to do with diffs like he below, where the table_info_reader is swapped for the indexer_reader:

aptos-labs/aptos-core@872b134#diff-a72b752f9c58eb76c8deee7e9f457835dea0a651362dd9176f2fda7df4f0958c

[l-monninger]

From this diff, it doesn't look like they're using the indexer to populate the whatever now acts as the StateView for the executor. But, it seems elsewhere like many client operations still run through StateView which now uses the indexer_reader:

aptos-labs/aptos-core@872b134#diff-4d3211585dd6312db70a9dfba692a8a499e275affae1bff0c2a81229f8041464

[l-monninger]

The Mystery of AIP-97

AIP-97 references a migration guide for sharding and seemingly several other upgrades. However, there are several peculiarities.

1. The migration guide seems to suggest the format change can be accomplished by setting the sharding_enabled: true parameter. But, then then actual "migration" described seems to be a sync from a peer. That is, it seems more like a proper migration was performed on one node from which others were asked to sync.
2. AIP-97 says additional migration guides would be included in the 1.18.0. However, they are not: https://github.com/aptos-labs/aptos-core/releases/tag/aptos-node-v1.18.0-rc
3. AIP-97 is not in fact referenced in any of the releases: https://github.com/aptos-labs/aptos-core/releases?q=AIP-97&expanded=true

I'd offer a few possible explanations:

1. AIP-97 was never, in fact, adopted. After all, it is still marked draft.
2. A simpler approach to the migration was determined to be embeddable with the full node, i.e., a migration is not necessary.
3. Sharding has still not landed on mainnet, I believe, as the Shardines stack is being built out. It may be that this was suggested before a more comprehensive plan was provided.

[l-monninger]

Today, I've been trying a few of things relevant to the above:

1. Bisect to where the the state DB last works properly.
2. While bisecting looking for somewhere where just starting with sharding_enabled: true actually works.
3. Passing in different configs for opening with the indexer_reader (to me this is the most likely culprit for StateKey simply being missing).