Problems during exporting to OAP endpoint in OTLP gRPC format

[lxx95]

Hi all,

I have a problem during exporter, my Otel collector shows logs like:

2023-04-29T15:10:47.486Z error exporterhelper/queued_retry.go:401 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "Permanent error: rpc error: code = Unimplemented desc = Method not found: opentelemetry.proto.collector.trace.v1.TraceService/Export", "dropped_items": 50}
2023-04-29T15:10:49.788Z error exporterhelper/queued_retry.go:401 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "Permanent error: rpc error: code = Unimplemented desc = Method not found: opentelemetry.proto.collector.logs.v1.LogsService/Export", "dropped_items": 3}

My goal is to configure the metrics/traces/logs exporting (gRPC) from the Otel Demo to Skywalking (that support otlp gRPC).

Here my collector configuration:

apiVersion: v1
kind: ConfigMap
metadata:
name: opentelemetry-demo-otelcol
labels:
app.kubernetes.io/name: otelcol
app.kubernetes.io/instance: opentelemetry-demo
app.kubernetes.io/version: "latest"
data:
relay: |
exporters:
logging: {}
otlp:
endpoint: 'skywalking-oap.skywalking.svc.cluster.local:11800'
tls:
insecure: true
skywalking:
endpoint: 'skywalking-oap.skywalking.svc.cluster.local:11800'
tls:
insecure: true
prometheus:
enable_open_metrics: true
endpoint: 0.0.0.0:9464
resource_to_telemetry_conversion:
enabled: true
extensions:
health_check: {}
memory_ballast:
size_in_percentage: 40
processors:
resource/job-name:
attributes:
- key: job_name
value: otel-demo
action: insert
receivers:
jaeger:
protocols:
grpc:
endpoint: ${MY_POD_IP}:14250
thrift_compact:
endpoint: ${MY_POD_IP}:6831
thrift_http:
endpoint: ${MY_POD_IP}:14268
otlp:
protocols:
grpc:
endpoint: ${MY_POD_IP}:4317
prometheus:
config:
scrape_configs:
- job_name: opentelemetry-collector
scrape_interval: 10s
static_configs:
- targets:
- ${MY_POD_IP}:8888
zipkin:
endpoint: ${MY_POD_IP}:9411
service:
extensions:
- health_check
- memory_ballast
pipelines:
logs:
exporters:
- otlp
- logging
processors:
- resource/job-name
receivers:
- otlp
metrics:
exporters:
- otlp
- logging
processors:
- resource/job-name
receivers:
- otlp
traces:
exporters:
- otlp
- logging
processors:
- resource/job-name
receivers:
- otlp
telemetry:
metrics:
address: ${MY_POD_IP}:8888

All the others components (pods/services/deployments) are the default (otel demo) one.
The Skywalking endpoint is fine, and it retrieves data from the own agent to the same endpoint.

Note: I have also tried with other demos like Online Botique with OTLP the same errors were shown.

Other discussion on this topic was opened (by me) on OTLP exporter Method not found: "opentelemetry.proto.collector.logs.v1.LogsService/Export" and "opentelemetry.proto.collector.trace.v1.TraceService/Export".

Hope that all is clear.

Thank you,
Luigi

[wu-sheng]

AFAIK, we don't support otlp log format for now.
Ref doc https://skywalking.apache.org/docs/main/next/en/setup/backend/log-analyzer/#opentelemetry

You should use the exporter of skywalking

Of course, this log format could be supported by SkyWalking OAP.Just no one added it yet.
If you would like to contribute, welcome

[lxx95]

Hi @wu-sheng, Thank you for the quick response.

It was not clear for me, Basing on this page I didn't found information about what is supported (and what not supported) on OTLP gRPC protocol.

So, this means that also for metrics and traces there is no support?

I see that the skywalking exporter is on beta version for logs and metrics.

Thank you.

[wu-sheng]

Referring to gRPC Skywalking Exporter, can we guarantee also the following flow: OTEL Demo -> gRPC OTLP -> OTEL Collector (receiver: otlp, exporter: skywalking) -> SkywalkingOAP:11800?

This is another receiver for SkyWalking native metric format, such as reporting by SkyWalking's agents. This is NOT for exporting.

---

Meanwhile, exporter docs are what you should take a look. https://skywalking.apache.org/docs/main/v9.4.0/en/setup/backend/exporter/

gRPC channel is for metrics. But notice, there is no OTEL format converter as we are using sub/pub model to do the exporting. So, if you insist it has to be a kind of OTLP metrics, you have to build an extra layer.

[lxx95]

Probably my explanation was not clear:

"This is another receiver for SkyWalking native metric format, such as reporting by SkyWalking's agents. This is NOT for exporting."

• I need: Exporting from OTEL collector to Skywalking OAP with SkyWalking gRPC Exporter

The goal is to use Skywalking like an APM, starting from OTLP (and not metrics/traces/logs from Skywalking agents).
I'm not interested to export data from Skywalking to other tools.

Please see again these two flows:

• OTEL Demo -> gRPC OTLP -> OTEL Collector (receiver: otlp, exporter: otlp) -> SkywalkingOAP:11800
• OTEL Demo -> gRPC OTLP -> OTEL Collector (receiver: otlp, exporter: skywalking) -> SkywalkingOAP:11800

[wu-sheng]

That is what you were referring,

referring to OpenTelemetry metrics receiver (with otlp gRPC protocol):

What exactly is your question? I am confused.

[lxx95]

Here the documentation refers to metric ingestion (to Skywalking) from the OTEL gRPC protocol, because Skywalking provide a OpenTelemetry receiver for metrics. -> please correct this sentence if something is wrong.

With an OTEL Collector we can receive metrics in some formats and export them into another format.
With the following flow we use the OTEL collector just to withdraw metrics and export in the Skywalking OAP component

• OTEL Demo -> gRPC OTLP -> OTEL Collector (receiver: otlp, exporter: otlp) -> SkywalkingOAP:11800

So, because in the OTEL Demo documentation we have these metrics I should expect them in the Skywalking UI.

The other point was related to another OTEL exporter, the SkyWalking gRPC Exporter
In that case:

• Is also valide the point above, can we expect the OTEL Demo metrics? What are the differences if we export with OTLP gRPC protocol or with the native Skywalking format (with the skywalking exporter on OTEL Collector)?

Hope that now is clear.

[wu-sheng]

So, because in the OTEL Demo documentation we have these metrics I should expect them in the Skywalking UI.

That has a precondition, you need to write MAL. Notice this statement in the doc

The rule file should be in YAML format, defined by the scheme described in MAL. Note: receiver-otel only supports the group, defaultMetricLevel, and metricsRules nodes of the scheme due to its push mode.

Then you could set up UI for the new metrics declared in the MAL script.

SkyWalking is not sampling saving metrics, but run a slow query in the query stage. It does most the pre-calculation and persistent the final results per time bucket.

---

The other point was related to another OTEL exporter, the SkyWalking gRPC Exporter

This is an alternative only. SkyWalking as an APM, we don't want to limit users' choices. So, you could use SkyWalking exporter in OTEL collector, or forward OTEL metrics format data directly to OAP. In both ways, you need to write MAL as I mentioned.

[wu-sheng]

@lxx95 I started this #10752. Notice this is for logging only.

[zjj1002]

BUT SkyWalking exporter only supported log and metrics , tracing is most important for most user
Any chance otel tracing would support officially?

beta: metrics, logs

[wu-sheng]

This asking generally makes no sense.
You didn't read previous discussions, and don't analysis why it doesn't, but only say it's important.

If it is, why no one did? It is a totally joke.

[abunet]

HI to everyone. I have very similar problems.
In my opinion, you should add a specific topic in documentation sample on howto configure a collector to receive opentelemetry metrics, logs, and traces and send them to skywalking.

I have an app written in dotnet core.
Metrics, Logs and Traces are generated using opentelemetry dotnet. I don't want to use specific agent libraries like SkyAPM, because I need to keep the app more interchangeable as possible.
My questions are:

1. Can I send my Metrics, Logs and Traces directly to the Sywalking Server via gRPC port 11800?
2. Anyone have a sample of an opentelemetry-collector configuration that can achieve the same objective?

I've tried two other opensource APM: SigNoz and Uptrace. Both of them support directly publishing in OpenTelemetry format.
I'm interested in Apache Skywalking for its architecture (Elastic backend, distributed telemetry with Satellite ...)
Thanks

[abunet]

Sorry but maybe I've not explained enough my intention.

I work for a big company and I need to make a product choice. I've said many time that I like Skywalking. I've spent two days trying to configure the product. I've compared it with other two opensource projects, because it's my job. Evaluate product features based on requirements.
My initial question was technical. Have you any available configuration that allows to receive in skywalking opentelemetry data without a native agent? Just answer: no.

And for trace in otlp, you even set up things through imagination. There is no skywalking trace exporter.

I've found this in documentation
https://skywalking.apache.org/docs/main/v9.1.0/en/setup/backend/zipkin-trace/

The Zipkin receiver makes the OAP server work as an alternative Zipkin server implementation for collecting traces. It supports Zipkin v1/v2 formats through the HTTP service.

so I've configured the oap container with SW_RECEIVER_ZIPKIN environment, and I've seen that skywalking can listen on 9411 port.
so I've configured an exporter in opentelemetry collector to export traces in Zipkin format.

I don't set up things through imagination. I setup things through documentation. Maybe I'm missing something. Or maybe the documentation is not clear.

We don't have any knowledge about those two. And we should not know.

It's only an opinion but you should consider what other APM in opensource landscape are offering.

So at the end of this thread, there's not still a clear answer.

How can I receive metric logs and traces sent from already existing apps that send data with opentelemetry protocol? Anyone have a working example, please?

Thanks

[wu-sheng]

If you want a simple answer, here it is.

1. Trace is only in Zipkin(OTLP traces are converted to Zipkin format) or native. No other. And Zipkin trace would not show on our UI as services on the general dashboard. It is only on the Zipkin widget. (You need to set it up on the UI).
2. Logs could be there, and no analysis by default.
3. Metrics would not show up until you want to write MAL and setup a new dashboard(Edit mode).

[wu-sheng]

For metrics, here is how to setup an example for VM(Prometheus exporter)

• MAL, , https://github.com/apache/skywalking/blob/master/oap-server/server-starter/src/main/resources/otel-rules/vm.yaml
• UI dashboard setup(exported from UI), https://github.com/apache/skywalking/tree/master/oap-server/server-starter/src/main/resources/ui-initialized-templates/os_linux
• OTLP collector, https://github.com/apache/skywalking/blob/master/test/e2e-v2/cases/vm/prometheus-node-exporter/otel-collector-config.yaml, docker compose, https://github.com/apache/skywalking/blob/master/test/e2e-v2/cases/vm/prometheus-node-exporter/docker-compose.yml
• Node exporter(OTLP metrics generator), https://github.com/apache/skywalking/blob/master/test/e2e-v2/cases/vm/prometheus-node-exporter/Dockerfile.nodeExporter

[wu-sheng]

It's only an opinion but you should consider what other APM in opensource landscape are offering.

The frustrating part is, that we are not relying on OTLP ecosystem. We have our own protocol, design, and propagation context, which is the most important part of our ecosystem. We just support OTLP in some cases as add-on(s) for the whole piece of INFRA monitoring.
We are not that close to those two you mentioned.

[wu-sheng]

For Trace of Zipkin and OTLP(converted to Zipkin), majorly, we are providing as Zipkin alternative, https://zipkin.io/pages/extensions_choices.html. We are an alternative Zipkin server, and Zipkin v3 server will be a SkyWalking backend re-distribution(not released yet).

The gap between this and native agent is huge. OLTP trace is basically a mutation/replication of Zipkin format. SkyWalking native agents are working for 100% sampling, and to analysis topology, instance dependency and API dependency.

[abunet]

Thanks for your detailed explanation and your time

I'll try with skywalking native protocol too.

In my microservice environment, there are many services that I can instrument with skywalking agent (I've source code access). Other services are thirdy party projects and they already support OLTP. so the requirement is to integrate all these services in the ecosystem with an APM solution.

Good Luck!

[wu-sheng]

If traces of those services(OTLP) are required, you may be in a tricky status. If only metrics and logs, you should be fine.
But, notice, all OTLP metrics integration would require you to setup MAL and dashboards(native UI or Grafana)

[chenhaipeng]

I have encountered similar problems, opentelemtry or skywalking, or opentelemtry in skywalking; since what I need here is a platform-based APM product that requires tenant isolation, the final solution is to integrate the two standards. How can I For solution, you can refer to similar solutions

trace:
skywalk->skywalkingOAP (transform to otlp format)->kafka ->flink etl->storage

opentelemtry-> collector(otlp format)-> kakfa -> etl->storage(such es)

logs:
skywalking logs/opentelemetry logs/filebeat -> kafka -> flink etl -> storage

metric: The indicator is based on monitoring implemented by skywalking and opentelmtry.

[wu-sheng]

skywalk->skywalkingOAP (transform to otlp format)->kafka ->flink etl->storage

Note, this is not a upstream feature. We never support this.