Performance issue archiving DataColumnSidecars

[twoeths]

Describe the bug

On one of lido nodes, archiving blocks could take up to 1.5 minutes

here is 1h interval

I think the spike mainly happened due to gc

Expected behavior

as short as possible

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.38.0

[twoeths]

Some logs found for that archived_block incident

Jan-26 14:54:42.986[chain]         verbose: Archiving of finalized blocks complete totalArchived=32, finalizedEpoch=423475
Jan-26 14:54:42.986[chain]         verbose: dataColumnSidecars prune slotRange=[13420160-13420165, 13420167-13420168, 13420170-13420185, 13420188-13420191], numOfSlots=28, totalNumOfSidecars=3584
Jan-26 14:53:46.087[chain]         verbose: Migrated dataColumnSidecars from hot DB to cold DB migratedEntries=2688
Jan-26 14:53:11.118[chain]         verbose: Migrated blobSidecars from hot DB to cold DB migratedEntries=0
Jan-26 14:53:11.111[chain]         verbose: Migrated blocks from hot DB to cold DB fromSlot=13551200, toSlot=13551169, size=32

it took 35s to migrate DataColumnSidecars from unfinalized db to finalized db
and almost 1m for leveldb to prune finalized DataColumnSidecars

[matthewkeil]

We can potentially solve this in a few ways. I was throwing around the idea of a compound key so that we can use a single repository for both before and after finalization. Or the idea of using an index to convert slot to root. That way there is no moving of date required. The only thing we would need to do is prune non-cononical forks once finalization happens.

The simple solution would be to create a slot->root index for finalized slots. The hot db method will be used for both hot and cold lookups. current implementation is root_columnIndex. To allow slot-wise lookups for range requests convert slot to root and then use same method as for hot db.

If we come up with a more complex indexing method we can optimize for the cold db lookups as for hot db we have all the information we need to pull columns no matter what compound key we use. (root, slot and column index) are all known between fork choice and reqresp request. some options for keys are

• slot_columnIndex_root
• slot_root_columnIndex

Both of these would allow gt/lt iterators for slot but the first will allow specific columns to be pulled first. In reality for by_range column requests will likely be sparse relative to what is in the db so likely we will need to pull the whole slot and just serve the indices that were requested. this is not great but it was how we were doing it before when we had the columns all concat'd and we were slicing out the ones we wanted. so not a great option. but perhaps there are other compound keys schemas that we can come up with which will be better for finalized requests.