Discussion: Tracing across threads with non-blocking I/O

[HaloFour]

I'm curious as to how money might support cross-thread tracing in Java when thread pools are not involved? Specifically, I would be interested in correlating traces across one or more HTTP requests performed using an NIO-based HTTP client (e.g. async-http-client) where a callback interface is invoked via a separate worker thread to indicate completion and on which I would want to correlate/continue the trace from the thread that originally initiated the request.

[pauljamescleary]

There is support for a "normal" http client. Have you reviewed that?

What we did there was have a "TraceFriendlyHttpClient" that wrapped all calls to http client and would inject the money header value. I believe we also record the response code which is a common ask.

[HaloFour]

Only to the extent of reading the Wiki about the wrapped implementation of Apache HttpClient. (The links in the TOC of the Wiki are a little messed up, by the way).

What concerns me is the nature of the threading around an NIO HTTP client, since the thread that will initiate the request will be different from the thread(s) servicing the request (writing the headers) and handling the response (reading the status code, response headers, etc.). It's more than likely that the thread initiating the request will be done with its "span" of the trace long before the HTTP request is complete, and likely leaving a period of time where no thread is associated with the trace.

In dealing with this issue with other logging/tracing libraries in the past I've often had to manually handle thread-local or trace context. For example, I would capture and persist MDC to a map associated with the request and then restore on whichever thread happened to handle the response. Using NewRelic's Agent API for tracing I would capture a token to the current transaction in the initiating thread, which would leave that transaction open, and then "link" to that token in the thread that handled the response.