Build a Streaming Aggregation Processor For Service Map

[mahadzaryab1]

Implement a processor in the OTEL pipeline that takes as input a stream of traces and constructs a dependency graph between services.

[mahadzaryab1]

@yurishkuro in this first step, do we just want to compute the dependencies in an apache beam pipeline?

[yurishkuro]

Yes but also implement that as an OTEL Processor. You can probably copy the existing processor from contrib (mentioned in the comments above) as a baseline.

[mahadzaryab1]

@yurishkuro Are you referring the tailsamplingprocessor? https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor

[mahadzaryab1]

@yurishkuro Another question - looks like OTEL has a groupbytraceprocessor. Is there any way we could leverage this or do we need to reuse this logic inside the processor that we write?

[yurishkuro]

No, I was referring to ServiceGraphConnector

groupbytraceprocessor is not needed for memory/badger, but may be needed for distributed storage. It somewhat depends on how Apache Beam implements the streams - if we have to shard spans by trace ID manually then something from OTEL will be needed (it's the same problem as with tail sampling processor, which needs full traces in one collector).