How can I track down why certain spans are not shown in Phoenix

[marcjulianschwarz]

Hey everyone, I am using the open telemetry node package with the OTLPTraceExporter to export spans with a BatchSpanProcessor to phoenix and at the same time export to Zipkin with the open telemetry ZipkinExporter. The traces and spans end up in both dashboards but Zipkin shows all spans while phoenix misses some spans.
Traces exported from Zipkin for the same run and imported into phoenix with a TraceDataset shows all spans in phoenix correctly.

Is there some way to check where spans get lost on the way to phoenix?

Here are the spans as seen in Zipkin and Phoenix. As you can see three spans are missing.

[dosubot[bot]]

Hello @marcjulianschwarz! I'm here to help you solve bugs, answer questions, and assist you in becoming a contributor. While you wait for a human maintainer, feel free to share more about your issue, and I'll do my best to assist you!

I found a similar open issue where traces are not registering locally with the new instrumentation technique. Unfortunately, no solution has been found in the comments of that issue ^[1].

To troubleshoot why certain spans are not shown in Phoenix, you might want to:

1. Verify the configuration of the OTLPTraceExporter to ensure that all spans are being sent correctly to Phoenix.
2. Check for any filtering or sampling rules that might be applied in Phoenix, which could be causing some spans to be omitted.
3. Compare the configuration and data between the Zipkin and Phoenix exports to identify any discrepancies.
4. Review the logs for any errors or warnings during the export process that might indicate issues with span transmission.

These steps might help you identify where spans are getting lost on the way to Phoenix.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[mikeldking]

@marcjulianschwarz - is your service distributed? I wonder if some spans are ending up under a different project. In general Phoenix doesn't drop any spans so I wonder if it's just ending up somewhere not intended.

[mikeldking]

@marcjulianschwarz - I can't repro the issue - though if your process exits before the BatchSpanProcessor exports the data, that could cause some to be missing. I created a repro branch here: #7118

[marcjulianschwarz]

@RogerHYang it is mainly happening for more complex, async and longer running examples. It is not consistent for the same set of spans. If I change to SimpleSpanProcessor spans are missing as well. When I switched to BatchSpanProcessor a few days ago it felt like an improvement. Spans that were missing before, appeared again.

This would be an example of a more complex trace:

Also, I noticed that most of the time it is one or more of the "leaf nodes" of traces that were missing (in our case the "semantic search" spans, children of the "relevance" spans).
As you can see from the screenshot, the tool executions are all running asynchronously. I just changed it to run all sequentially and Phoenix received all spans.

Is there any way I can accurately check which spans are actually received at the Phoenix endpoint? It would be great if we could compare the ConsoleSpanExporter messages with the spans received by the endpoint. So far, the ConsoleSpanExporter showed the missing spans.

[marcjulianschwarz]

Now I attached a debugger. First I used the SimpleSpanProcessor and multiple times I got a "socket hang up" error and some TCP read issues. It looks like here the export is failing.

SimpleSpanProcessor

BatchSpanProcessor

Then I also ran the BatchSpanProcessor and again, got a similar error, but only once and not multiple times:

Some more insights

Then I debugged while only exporting to Zipkin and no errors were shown for exporting spans. But debugging while exporting to Phoenix showed the above messages.

[RogerHYang]

Thank you for the additional info. This looks like it could be a low level Python issue. Would you mind giving the gRPC exporter a try as shown below for example?

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";

const provider = new NodeTracerProvider({
  spanProcessors: [
    new SimpleSpanProcessor(
      new OTLPTraceExporter({
        url: "http://localhost:4317",
      }),
    ),
  ],
});

registerInstrumentations({
  instrumentations: [new OpenAIInstrumentation()],
});

provider.register();

[marcjulianschwarz]

@RogerHYang do I have to use the normal hostname for Phoenix? In my case that would be "http://localhost:6006"?

If I send traces to http://localhost:6006 I get following errors:

These are my version numbers:

"@arizeai/openinference-instrumentation-openai": "^0.3.4",
"@opentelemetry/api": "^1.9.0",
"@opentelemetry/exporter-trace-otlp-grpc": "^0.200.0",
"@opentelemetry/exporter-trace-otlp-proto": "^0.200.0",
"@opentelemetry/exporter-zipkin": "^2.0.0",
"@opentelemetry/resources": "^2.0.0",
"@opentelemetry/sdk-trace-node": "^2.0.0",
"@opentelemetry/semantic-conventions": "^1.30.0",

[RogerHYang]

The port for gRPC is http://localhost:4317 instead, without any path

[marcjulianschwarz]

Got it, we weren't exposing the gRPC port, only 6006.

As far as I could test, this seems to be working great! Thanks a lot for helping out with debugging. Are there any shortcomings using gRPC with this exporter? It says that it is still in active development and an experimental package.