How to log trace id to formatter?

[npuichigo]

For now, OtelInResponseLayer helps to send the trace_id back to user and is helpful for debugging. But how to log that in formatter?

[davidB]

In previous versions, a hack was used to inject trace_id into logs. But sometimes a wrong trace_id was injected, because the trace_id changed after some operations (tracing_opentelemetry generate the opentelemetry span, when the tracing span is closed and the trace_id could change during the life of the span).

During my spare time, I'm looking for a more accurate solution, but it could imply that some log will not have trace_id.

Any contributions, PR,... are welcomes

[davidB]

FYI, in the past, I saw a crate providing a formatter with the trace_id, but I never tried it with the crates of this repository.

I also don't know how it combine with opentelemetry logger, or recent crate like https://crates.io/crates/tracing-logfmt-otel

[npuichigo]

My workaround is including the filled headers in log, maybe it's too verbose, but at least I can track.

let svc = ServiceBuilder::new()
    // log requests and responses
    .layer(
        TraceLayer::new_for_http()
            .make_span_with(DefaultMakeSpan::new().include_headers(true).level(info))
            .on_response(DefaultOnResponse::new().include_headers(true).level(info))
    )
    .service(handler);

[masongup-mdsol]

I was able to write a custom Tracing Subscriber Layer that does this, here's the code cut down to the bare minimum:

impl<S> Layer<S> for CustomJsonLayer
    where S: tracing::Subscriber + for<'lookup> tracing_subscriber::registry::LookupSpan<'lookup>
{
    fn on_event(&self, event: &Event, ctx: Context<S>) {
        let mut output_object = serde_json::Map::<String, Value>::new();
        if let Some(base_span) = ctx.event_span(event) {
            for span in base_span.scope() {
                if span.metadata().target() == "otel::tracing" {
                    if let Some(otel_context) = span.extensions().get::<OtelData>() {
                        if let Some(trace_id) = find_trace_id(&otel_context.parent_cx) {
                            output_object.insert("trace_id".to_string(), Value::String(trace_id.to_string()));
                        }
                    }
                }
            }
        }
        println!("{}", serde_json::to_string_pretty(&Value::Object(output_object)).unwrap())
    }
}

This does the job, but you'd have to write any other output stuff you wanted manually. It's not clear to me that there's any practical way to provide some sort of helper or converter to some other utility to allow any of the standard log formatters to display the OpenTelemetry tracing info though.