Problem
Currently, when the service starts up or syncs blocks from S3, it loads each shard one at a time (sequentially). For blocks with 12 shards, this means waiting for shard 1 to finish, then shard 2, then shard 3, and so on. This is slow and doesn't use available network bandwidth efficiently.
On instances with high network capacity, the sequential loading creates a bottleneck - the network sits mostly idle while we slowly load one shard at a time.
Solution
Add a new flag--block.syncer.shard-concurrency that allows loading multiple shards at the same time (in parallel). This keeps the network busy and speeds up the sync process significantly.
Benefits
- Faster service startup time.
- Faster block syncs when new data arrives.
- Better utilization of available network bandwidth.
- Configurable to match instance capabilities.
Problem
Currently, when the service starts up or syncs blocks from S3, it loads each shard one at a time (sequentially). For blocks with 12 shards, this means waiting for shard 1 to finish, then shard 2, then shard 3, and so on. This is slow and doesn't use available network bandwidth efficiently.
On instances with high network capacity, the sequential loading creates a bottleneck - the network sits mostly idle while we slowly load one shard at a time.
Solution
Add a new flag
--block.syncer.shard-concurrencythat allows loading multiple shards at the same time (in parallel). This keeps the network busy and speeds up the sync process significantly.Benefits