Enhance service graph with more information

[chenlujjj]

Is your feature request related to a problem? Please describe

Now the service graph data includes (parent, child, callCount) for each edge. A few APM products provide more information, such as qps, error rate, latency.

Describe the solution you'd like

When generating service graph data, we can query out more data.

Current logsql is:

(NOT parent_span_id:"") AND (kind:~"2|5")
   | fields parent_span_id, resource_attr:service.name
   | rename parent_span_id as span_id, resource_attr:service.name as child
   | join by (span_id) (
       (NOT span_id:"") AND (kind:~"3|4")
       | fields span_id, resource_attr:service.name
       | rename resource_attr:service.name as parent
     ) inner
   | NOT parent:eq_field(child)
   | stats by (parent, child) count() callCount

Suggested logsql:

(NOT parent_span_id:"") AND (kind:~"2|5")
   | fields parent_span_id, resource_attr:service.name, duration, status_code
   | rename parent_span_id as span_id, resource_attr:service.name as child, duration as child_duration, status_code as child_status_code
   | join by (span_id) (
       (NOT span_id:"") AND (kind:~"3|4")
       | fields span_id, resource_attr:service.name, duration, status_code
       | rename resource_attr:service.name as parent, duration as parent_duration, status_code as parent_status_code
     ) inner
   | NOT parent:eq_field(child)
   | stats by (parent, child) count() callCount, sum(parent_duration) as parentDurationSum, sum(child_duration) as clientDurationSum, count() if (child_status_code:=2) childErrorCount, count() if (parent_status_code:=2) parentErrorCount

Moreover, we can include the value of servicegraph.taskLookbehind in each edge to calculate:

• rps = callCount / taskLookbehind
• avg duration = durationSum / taskLookbehind
• error rate = errorCount / taskLookbehind

Describe alternatives you've considered

No response

Additional information

ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/servicegraphconnector/README.md

[xiaoyuan2019]

@chenlujjj Is it possible to consider displaying the number of times each service node calls db.system (mysql/redis) on the node itself? This would be very useful to intuitively see how many times each service calls middleware, allowing for performance optimization.

[chenlujjj]

@xiaoyuan2019 I think we can refer to the virtual node concept in service graph connector:

virtual_node_peer_attributes: the list of attributes, ordered by priority, whose presence in a client span will result in the creation of a virtual server node. An empty list disables virtual node creation.
Default: [peer.service, db.name, db.system]

I'll try to figure out the logsql.

[chenlujjj]

kind:="3" AND `span_attr:db.system`:* 
| rename resource_attr:service.name as parent, span_attr:db.system as child
| stats by (parent, child) count()

[xiaoyuan2019]

@chenlujjj Can the invocation count of middleware be integrated into the service topology? Is there a schedule for this feature?

[jiekun]

Hi friends! Thanks for the great discussion here! Actually, we received a similar request some time ago — the current service graph has very limited information, showing nothing more than A->B.

The reason we didn’t move forward with enhancements back then is this: Jaeger doesn’t seem to support displaying additional details. Even if we made improvements, two issues would remain:

1. Users would have to build a custom UI to present these extra data.
2. These enhancements might conflict with the service graph features in the (yet-to-be-implemented) Grafana Tempo API.

That’s why this enhancement was put on hold. Our original plan is to first implement the Grafana Tempo API (it's under developing right now). During this process, we’ll analyze the differences between the service graph provided by Grafana Tempo and Jaeger, then build a unified implementation that is compatible with both while meeting the requirements for richer data presentation.

But this doesn’t mean the issue and discussion here is not useful. We’ve noticed your suggestions and requests, and we need to ensure that the proposed information has a proper place to be displayed — ideally, it can be used out of the box without requiring any additional user-side development on UI.

[chenlujjj]

the proposed information has a proper place to be displayed — ideally, it can be used out of the box without requiring any additional user-side development on UI.

Hi @jiekun , will VictoriaTraces has built-in UI designed for trace/APM instead of current VictoriaLogs UI and Grafana datasource plugin ?

I did play around with both Jaeger and Tempo Grafana datasource a few months ago, their service graph UI is not that user-friendly compared to some commercial products like Datadog, Signalfx. It would be great if VictoriaTraces can move a step forward.

[jiekun]

We want to have our own UI (but just for browsing traces, in the first version) in Q1. This is what we've discussed internally last year.

---

But we really don't have a BIG plan to cover everything yet. The advancement of the Tempo API was also driven by the high volume of user requests. Originally, our focus was supposed to remain on performance optimization and resource utilization.

For the foreseeable future, after delivering the Tempo API, we will return to performance optimization rather than developing more visualization-related features. From my perspective, users choose VictoriaTraces primarily because they have hit bottlenecks or cost constraints when scaling their existing solutions. This is the key area where VictoriaTraces must maintain a competitive edge. Currently, the team is not satisfied with the resource usage, and we are convinced that there is substantial room for further improvement.

[chenlujjj]

Totally understand and agree that VictoriaTraces should deliver Jaeger and Tempo compatible APIs to attract their users and make the migration effortless.

From my perspective, users choose VictoriaTraces primarily because they have hit bottlenecks or cost constraints when scaling their existing solutions. This is the key area where VictoriaTraces must maintain a competitive edge. Currently, the team is not satisfied with the resource usage, and we are convinced that there is substantial room for further improvement.

I'm curious about the resource usage target of VictoriaTraces and how it compares to Tempo and ClickHouse-storage Jaeger solution. It counts much when people make decisions to use which solution.

[jiekun]

the resource usage target of VictoriaTraces and how it compares to Tempo and ClickHouse-storage Jaeger

It should be in better shape right now than in our initial benchmark. But I'm not going to lie: we haven’t conducted another rigorous and standardized benchmark yet, due to various ongoing optimization efforts.

I'm confident because:

1. In the first benchmark, we only compared ClickHouse against VictoriaTraces, and the resource usage looks closed.
2. We didn't include the data unmarshaling process for Jaeger + ClickHouse in our previous benchmark. Unmarshaling OTLP requests into Go structs is handled by Jaeger, and it is by no means a lightweight move—it consumes a log of resources, which is not counted in the article.

While ClickHouse’s resource usage appears comparable to VictoriaLogs, ClickHouse cannot process OTLP requests directly. This means it does not need to expend CPU resources on decoding Protobuf or JSON data, which made the original comparison an unfair (to VictoriaTraces) benchmark.

We try to avoid making our benchmark results look overly impressive, because VictoriaTraces is a new product. We don’t want to claim something like "This is incredibly efficient, cutting CPU usage by 80%" and then have users encounter crashes in their deployments due to unforeseen bugs.

Delivering something that fails to live up to its claims hurts users far more than just claim it "poor performance".

[jiekun]

See my benchmark with constant 20000 spans/s ingestion rate: https://snapshots.raintank.io/dashboard/snapshot/qST9uFizbVspOYcLYWVvbnVlaxBjzik7

flow:

• vtgen replays traffic captured from OTel demo at 20000 spans/s, and mirrors the request to VictoriaTraces, Tempo, and ClickHouse*.
  • *: ClickHouse cannot accept OTLP requests directly, so an OTel Collector is used (OTLP receiver + batch processor + clickhouse exporter).

You can see the following results:

1. ClickHouse and VictoriaTraces perform close to each other in terms of CPU usage, while ClickHouse consumes more memory.
   • If the OpenTelemetry Collector is also taken into account, the ClickHouse pipeline uses twice the CPU of VictoriaTraces.
2. Tempo does not perform very well.

[chenlujjj]

@jiekun Very impressive benchmark results!

Regarding the ClickHouse part, did you use ClickHouse cloud or deploy a ClickHouse cluster on your own? If it's self-hosted, did you choose ReplicatedMergeTree as table engine?

[jiekun]

It's a self hosted instance, with MergeTree engine, defaulted by the clickhouse exporter: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/clickhouseexporter/README.md