Prevent database corruption on ungraceful shutdown

[preston-evans98]

Background

Currently, ungraceful shutdowns during DB commit can leave the database in an inconsistent state.

Design

To prevent issues, we need to make minor changes to the rollup's startup sequence, the commit-flag enum, and the commit flow.

enum CommitFlag { 
   Successful { height: u64 },
   ComittingKernelNomt { height: u64 },
   ComittingUserNomt { height: u64 },
   ComittingRocksDB{ height: u64 },
}

On startup, check commit flag
1. If status is successful, start from the height +1
2. If ComittingKernelNomt, check whether the commit finished by comparing the NOMT root to the root stored in the live state table. If they match, no rollback is required. Otherwise, roll back kernel NOMT. Start from height
3. If ComittingUserNomt in progress, check whether the commit finished by comparing the NOMT root to the root stored in the live state table. If the roots matched, the commit didn't succeed and we're safe. Start from height
1. If the roots do not match, this means NOMT is out of sync with the flat state. To recover we can either
1. Enable rollback for user NOMT (not ideal)
2. Skip the consistency check (and update, which would be a no-op anyway) on the next commit
3. Note that witness generation will be impossible until the next commit. SO, when we enable proving, commit the witness before updating the NOMT instance.

On commit:

1. Commit ledger and accessory state. These can't be used read block execution - so they should be safe even if they're not fully consistent; they'll get overwritten with the same data when we process the block again (TODO: Verify this is true of accessory state; might be used in sequencer for state root calcs etc.)
2. Update flag -> ComittingKernelNomt { height: u64 },
3. Commit kernel NOMT.
   1. NOMT claims to provide crash fault tolerance through a WAL, so this is atomic
4. Update flag to ComittingUserNomt
5. Commit user NOMT
6. Update flag to CommittingRocksDB
7. Commit all kernel/user DBs as a unit (remove "other" - it only contains a singleton which can be put in the main DBs;
   1. Commit archival first. If this fails, it'll get overwritten with identical values on the next run
   2. Commit all flat values atomically.
8. Update flag to Successful

[bkolad]

Points for Discussion

1. Making FlatDb writes atomic (see point 7 above)

a) Ensure all relevant RocksDB settings are correctly configured (e.g., WriteOptions::disableWAL = false, etc).
b) If separate_archival_state is set to false, then the user, kernel, and other states will be stored in the same RocksDB (todo remove separate_archival_state)
c) RocksDB writes are atomic only if all updates are issued in a single db->Write() call using one batch. This is currently not the case.

At the moment, we perform five separate write calls, for example:

// kernel & user
self.archival_db.write_opt(...)
self.live_db.write_opt(...)
// other
self.other.write_schemas(&state.other)

To guarantee atomicity, all updates must be squashed into a single batch (e.g., one WriteBatch / SchemaBatch) and committed with one self.db.write(...) call

2. The CommitStatus flag. Pseudocode:

match status{
   CommitStatus::CommittingKernelNomt & different_roots => commit unsucesfull continue
   CommitStatus::CommittingKernelNomt & same_roots=> kernel.rollback(1)

   CommitStatus::CommittingUserNomt & different_roots => kernel.rollback(1)
   CommitStatus::CommittingUserNomt & same_roots => user.rollback(1) & kernel.rollback(1)

    CommitStatus::CommittingArchivalUserAndKernel => user.rollback(1) & kernel.rollback(1) (<- we don't need this?)
    CommitStatus::CommittingLiveUserAndKernel => {
        1. Get all the keys from PRUNING table that ware modified in the previous version
       2. Het all the values from archival
       3. Apply the on live_db

        OR 

       ===> user.rollback(1) & kernel.rollback(1)
     }



   CommitStatus::Success => continue
}

TODO handle the case where we crash just after CommittingLiveUserAndKernel it succeeds but Success wasn't saved.
Use VersionedTableMetadataKey::CommittedVersion to detect it.

3. Testing methodology

[bkolad]

Make sure the ledger db can override the data in CommittingLiveUserAndKernel case
Same for accessory state (we don't read it on sequencer startup - check the latest state root, )