[Bug] Production startup time takes multiple minutes

[vicsn]

🐛 Bug Report

Recent benchmarks show that startup time for production ledgers with thousands of deployments can take multiple minutes on solid m7i.4xlarge machines. Specific data will be added to this issue by @meddle0x53

Loading the ledger happens via Ledger::load and then VM::from. Given the small surface area and heavy dependence on I/O, a few simple timer! and lap! additions should be sufficient to learn what is taking so much time. Likely contenders are:

• Serial calling of load_deployment, which creates Stacks - the VM's abstraction around deployed programs. If Stack loading is a bottleneck, this will mean that lazy loading them (as proposed by Only keep track of recently used stacks in memory. #2591 ) will create a large performance issue.
  • Perhaps we can load deployments without any imports in parallel
  • Perhaps we can extract parallelizable components from load_deployment
• Initializing the ConsensusStore