Parent

• Parent issue: Milestone 1 - Shardlake Post-Prototype Roadmap #5

Goal

Allow index builds to run in parallel across shards and workers.

Detailed tasks

• 9.1 Parallel shard building
  • Use rayon or async tasks to build shards concurrently.
• 9.2 Distributed build mode
  • Support splitting the dataset across workers.
• 9.3 Merge index shards
  • Combine distributed build outputs into final artifacts.
• 9.4 Build scheduler
  • Add CLI support via shardlake build-index --parallel.

Definition of done

• Local parallel builds are supported.
• Distributed build outputs can be merged deterministically.
• The CLI exposes a parallel build path.

Child issue breakdown

• 9.1 - Parallelize shard build execution #75 9.1 - Parallelize shard build execution
• 9.2 - Add distributed build worker mode #76 9.2 - Add distributed build worker mode
• 9.3 - Merge distributed build outputs #77 9.3 - Merge distributed build outputs
• 9.4 - Add parallel build scheduler and CLI support #78 9.4 - Add parallel build scheduler and CLI support

Dependency summary

• 9.3 - Merge distributed build outputs #77 depends on 9.2 - Add distributed build worker mode #76.
• 9.4 - Add parallel build scheduler and CLI support #78 depends on 9.1 - Parallelize shard build execution #75 and 9.3 - Merge distributed build outputs #77.

Dependency graph

#75   #76
  \     |
   \   #77
    \ /
    #78