This issue is to introduce basic optimistic indexing logic to the feature branch, so that follow up PRs can improve upon this logic (potentially allowing the work to be parallelized).

The basic implementation doesn't have to solve all known issues (race conditions, dependency checks, pruning, etc.), those will be solved later on the feature branch.