FR: Support distributed implementations of index traits

[bts]

Is your feature request related to a problem? Please describe.

At the moment, there is not an easy way to create distributed/networked implementations of storage traits like Index, MutableIndex, ReadonlyIndex, and ChangeIdIndex because at least some methods on each trait are currently infallible. There's no way to represent and handle I/O errors that might occur in a distributed setting. Without severe workarounds, index implementations would be forced to keep entire indices in memory. The Backend, OpStore, and OpHeadsStore storage traits already support distributed implementations, so this feature would bring the *Index traits in line with those three.

Describe the solution you'd like

Convert the trait methods which could suffer I/O failures to return Result<T> instead of T. Some methods on Index already do this, but we would need to convert the remaining methods across these four traits.

Though we will want to convert most methods on these traits to return Result<T>, we tentatively would not do this for ReadonlyIndex::start_modification() at first. That's because the consequences are quite a bit more involved than for the other methods, due to causing WorkspaceCommandHelper::start_transaction() to become fallible. From what I can tell, I think most distributed implementations of the storage traits would probably be able to get by without making this method fallible by using an interior mutability pattern.

Describe alternatives you've considered

There are workarounds like communicating errors out of band, but this does not seem practical for trait implementers. Additionally we already have the ability surface I/O failures on some existing Index methods, and all non-index storage traits.

Additional context

I previously opened an initial discussion about this effort in a draft PR at #7790, but I've opened this issue so that we will have a more permanent artifact to reference from commit messages.