new feature: hedged request support for read-only operations

[dentiny]

Feature Description

Hi team, I've suffered tail latency in my production environment, I'm wondering if you think it makes sense to support hedged request to cut down tail latency.
Reference: https://research.google/pubs/the-tail-at-scale/

Problem and Solution

Having hedged request, which issues duplicate request (potentially to another serving host, or using another TCP connection) will improve tail latency.
Different from the timeout configurations, with hedged request there're actually multiple on-the-flight requests; the client (opendal here) serves with the earliest arriving response, and properly manages the lifecycle for all ongoing requests.
Notice hedged request should only support idempotent operations, mostly it means read-only operations, write could also be supported by they usually introduce conflict even if idempotent (i.e., database transaction abort).

Proposed solution:
Similar to timeout and retry, hedged request could also be made as a layer and guarded by features; so users are able to decide whether to opt-in based on their requirement.

Additional Context

It's related to another feature request issue I proposed: #7168
I would like to have per-operation configuration, like timeout and hedged request trigger threshold.

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.