Automatically record request/responses for o11y

[coryan]

In the context of invoking the RequestRecorder calls for HTTP requests and responses @westarle notes:

---

Nit: I get a mildly worried about matching the on_x_response to the on_x_requests when they are distributed. I think our tests cover things practicality right now, but I never know what complexity we'll add in the future.

A couple ideas:

• in RequestRecorder have on_http_request return a guard that you must_use to call finish(result)
• Have ClientSnapshot do accounting and put a debug assert in drop

Originally posted by @westarle in #5176 (comment)

---

I think the "One True Answer" is using tower layers... but our ReqwestClient is not structured around them.

A different structural answer is to decorate the future that returns the Result<reqwest::Response, >.

Those are good cleanup opportunities for ReqwestClient.