Skip to content

Commit 6a93872

Browse files
IvanVanIvanVan
authored
OpenTelemetry emitter extension (#12015)
* Add OpenTelemetry emitter extension * Fix build * Fix checkstyle * Add used undeclared dependencies * Ignore unused declared dependencies
1 parent e0c4c56 commit 6a93872

File tree

10 files changed

+948
-1
lines changed

10 files changed

+948
-1
lines changed

‎distribution/pom.xml

+2
Original file line numberDiff line numberDiff line change
@@ -614,6 +614,8 @@
614614
<argument>org.apache.druid.extensions.contrib:gce-extensions</argument>
615615
<argument>-c</argument>
616616
<argument>org.apache.druid.extensions.contrib:aliyun-oss-extensions</argument>
617+
<argument>-c</argument>
618+
<argument>org.apache.druid.extensions.contrib:opentelemetry-emitter</argument>
617619
</arguments>
618620
</configuration>
619621
</execution>

‎extensions-contrib/opentelemetry-emitter/README.md

+166
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
<!--
2+
~ Licensed to the Apache Software Foundation (ASF) under one
3+
~ or more contributor license agreements. See the NOTICE file
4+
~ distributed with this work for additional information
5+
~ regarding copyright ownership. The ASF licenses this file
6+
~ to you under the Apache License, Version 2.0 (the
7+
~ "License"); you may not use this file except in compliance
8+
~ with the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing,
13+
~ software distributed under the License is distributed on an
14+
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
~ KIND, either express or implied. See the License for the
16+
~ specific language governing permissions and limitations
17+
~ under the License.
18+
-->
19+
20+
# OpenTelemetry Emitter
21+
22+
The [OpenTelemetry](https://opentelemetry.io/) emitter generates OpenTelemetry Spans for queries.
23+
24+
## How OpenTelemetry emitter works
25+
26+
The [OpenTelemetry](https://opentelemetry.io/) emitter processes `ServiceMetricEvent` events for the `query/time`
27+
metric. It extracts OpenTelemetry context from
28+
the [query context](https://druid.apache.org/docs/latest/querying/query-context.html). To link druid spans to parent
29+
traces, the query context should contain at least `traceparent` key.
30+
See [context propagation](https://www.w3.org/TR/trace-context/) for more information. If no `traceparent` key is
31+
provided, then spans are created without `parentTraceId` and are not linked to the parent span. In addition, the emitter
32+
also adds other druid context entries to the span attributes.
33+
34+
## Configuration
35+
36+
### Enabling
37+
38+
To enable the OpenTelemetry emitter, add the extension and enable the emitter in `common.runtime.properties`.
39+
40+
Load the plugin:
41+
42+
```
43+
druid.extensions.loadList=[..., "opentelemetry-emitter"]
44+
```
45+
46+
Then there are 2 options:
47+
48+
* You want to use only `opentelemetry-emitter`
49+
50+
```
51+
druid.emitter=opentelemetry
52+
```
53+
54+
* You want to use `opentelemetry-emitter` with other emitters
55+
56+
```
57+
druid.emitter=composing
58+
druid.emitter.composing.emitters=[..., "opentelemetry"]
59+
```
60+
61+
_*More about Druid configuration [here](https://druid.apache.org/docs/latest/configuration/index.html)._
62+
63+
## Testing
64+
65+
### Part 1: Run zipkin and otel-collector
66+
67+
Create `docker-compose.yaml` in your working dir:
68+
69+
```
70+
version: "2"
71+
services:
72+
73+
zipkin-all-in-one:
74+
image: openzipkin/zipkin:latest
75+
ports:
76+
- "9411:9411"
77+
78+
otel-collector:
79+
image: otel/opentelemetry-collector:latest
80+
command: ["--config=otel-local-config.yaml", "${OTELCOL_ARGS}"]
81+
volumes:
82+
- ${PWD}/config.yaml:/otel-local-config.yaml
83+
ports:
84+
- "4317:4317"
85+
```
86+
87+
Create `config.yaml` file with configuration for otel-collector:
88+
89+
```
90+
version: "2"
91+
receivers:
92+
receivers:
93+
otlp:
94+
protocols:
95+
grpc:
96+
97+
exporters:
98+
zipkin:
99+
endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
100+
format: proto
101+
102+
logging:
103+
104+
processors:
105+
batch:
106+
107+
service:
108+
pipelines:
109+
traces:
110+
receivers: [otlp]
111+
processors: [batch]
112+
exporters: [logging, zipkin]
113+
```
114+
115+
*_How to configure otel-collector you can read [here](https://opentelemetry.io/docs/collector/configuration/)._
116+
117+
Run otel-collector and zipkin.
118+
119+
```
120+
docker-compose up
121+
```
122+
123+
### Part 2: Run Druid
124+
125+
Build Druid:
126+
127+
```
128+
mvn clean install -Pdist
129+
tar -C /tmp -xf distribution/target/apache-druid-0.21.0-bin.tar.gz
130+
cd /tmp/apache-druid-0.21.0
131+
```
132+
133+
Edit `conf/druid/single-server/micro-quickstart/_common/common.runtime.properties` to enable the emitter (
134+
see `Configuration` section above).
135+
136+
Start the quickstart with the apppropriate environment variables for opentelemetry autoconfiguration:
137+
138+
```
139+
OTEL_SERVICE_NAME="org.apache.druid" OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" bin/start-micro-quickstart
140+
```
141+
142+
*_More about opentelemetry
143+
autoconfiguration [here](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure)_
144+
145+
Load sample data - [example](https://druid.apache.org/docs/latest/tutorials/index.html#step-4-load-data).
146+
147+
### Part 3: Send queries
148+
149+
Create `query.json`:
150+
151+
```
152+
{
153+
"query":"SELECT COUNT(*) as total FROM wiki WHERE countryName IS NOT NULL",
154+
"context":{
155+
"traceparent":"00-54ef39243e3feb12072e0f8a74c1d55a-ad6d5b581d7c29c1-01"
156+
}
157+
}
158+
```
159+
160+
Send query:
161+
162+
```
163+
curl -XPOST -H'Content-Type: application/json' http://localhost:8888/druid/v2/sql/ -d @query.json
164+
```
165+
166+
Then open `http://localhost:9411/zipkin/` and you can see there your spans.

‎extensions-contrib/opentelemetry-emitter/pom.xml

+217
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Licensed to the Apache Software Foundation (ASF) under one
4+
~ or more contributor license agreements. See the NOTICE file
5+
~ distributed with this work for additional information
6+
~ regarding copyright ownership. The ASF licenses this file
7+
~ to you under the Apache License, Version 2.0 (the
8+
~ "License"); you may not use this file except in compliance
9+
~ with the License. You may obtain a copy of the License at
10+
~
11+
~ http://www.apache.org/licenses/LICENSE-2.0
12+
~
13+
~ Unless required by applicable law or agreed to in writing,
14+
~ software distributed under the License is distributed on an
15+
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
~ KIND, either express or implied. See the License for the
17+
~ specific language governing permissions and limitations
18+
~ under the License.
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0"
21+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
22+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
23+
<parent>
24+
<groupId>org.apache.druid</groupId>
25+
<artifactId>druid</artifactId>
26+
<version>0.23.0-SNAPSHOT</version>
27+
<relativePath>../../pom.xml</relativePath>
28+
</parent>
29+
<modelVersion>4.0.0</modelVersion>
30+
31+
<groupId>org.apache.druid.extensions.contrib</groupId>
32+
<artifactId>opentelemetry-emitter</artifactId>
33+
<name>opentelemetry-emitter</name>
34+
<description>Extension support for emitting OpenTelemetry spans for Druid queries</description>
35+
36+
<properties>
37+
<opentelemetry.version>1.7.0</opentelemetry.version>
38+
<opentelemetry.instrumentation.version>1.7.0-alpha</opentelemetry.instrumentation.version>
39+
<!-- These guava and grpc versions are used only in the opentelemetry-extension.
40+
Look at build section for more details about shading. -->
41+
<shade.guava.version>30.1.1-jre</shade.guava.version>
42+
<shade.grpc.version>1.41.0</shade.grpc.version>
43+
</properties>
44+
<dependencyManagement>
45+
<dependencies>
46+
<dependency>
47+
<groupId>io.opentelemetry</groupId>
48+
<artifactId>opentelemetry-bom</artifactId>
49+
<version>${opentelemetry.version}</version>
50+
<type>pom</type>
51+
<scope>import</scope>
52+
</dependency>
53+
<dependency>
54+
<groupId>io.opentelemetry</groupId>
55+
<artifactId>opentelemetry-bom-alpha</artifactId>
56+
<version>${opentelemetry.version}-alpha</version>
57+
<type>pom</type>
58+
<scope>import</scope>
59+
</dependency>
60+
<dependency>
61+
<groupId>io.opentelemetry.instrumentation</groupId>
62+
<artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
63+
<version>${opentelemetry.instrumentation.version}</version>
64+
<type>pom</type>
65+
<scope>compile</scope>
66+
</dependency>
67+
</dependencies>
68+
</dependencyManagement>
69+
<dependencies>
70+
<dependency>
71+
<groupId>org.apache.druid</groupId>
72+
<artifactId>druid-core</artifactId>
73+
<version>${project.parent.version}</version>
74+
<scope>provided</scope>
75+
</dependency>
76+
<dependency>
77+
<groupId>io.opentelemetry</groupId>
78+
<artifactId>opentelemetry-context</artifactId>
79+
</dependency>
80+
<dependency>
81+
<groupId>io.opentelemetry</groupId>
82+
<artifactId>opentelemetry-api</artifactId>
83+
</dependency>
84+
<dependency>
85+
<groupId>io.opentelemetry</groupId>
86+
<artifactId>opentelemetry-sdk</artifactId>
87+
</dependency>
88+
<dependency>
89+
<groupId>io.opentelemetry</groupId>
90+
<artifactId>opentelemetry-sdk-trace</artifactId>
91+
</dependency>
92+
<dependency>
93+
<groupId>io.opentelemetry</groupId>
94+
<artifactId>opentelemetry-sdk-common</artifactId>
95+
</dependency>
96+
<!-- OpenTelemetry extension bundles the OpenTelemetry auto-instrumentation,
97+
So it could potentially affect performance -->
98+
<dependency>
99+
<groupId>io.opentelemetry</groupId>
100+
<artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
101+
</dependency>
102+
<dependency>
103+
<groupId>io.grpc</groupId>
104+
<artifactId>grpc-netty-shaded</artifactId>
105+
<version>${shade.grpc.version}</version>
106+
</dependency>
107+
<dependency>
108+
<groupId>com.google.code.findbugs</groupId>
109+
<artifactId>jsr305</artifactId>
110+
</dependency>
111+
<dependency>
112+
<groupId>com.google.inject</groupId>
113+
<artifactId>guice</artifactId>
114+
</dependency>
115+
<dependency>
116+
<groupId>com.google.guava</groupId>
117+
<artifactId>guava</artifactId>
118+
<version>${shade.guava.version}</version>
119+
</dependency>
120+
<!-- explicitly include perfmark dependency of grpc we exclude from the shaded jar
121+
Note: we could use promoteTransitiveDependencies=true in the shade plugin, but that promotes all
122+
transitive dependencies as well, which unnecessarily pollutes the final pom -->
123+
<dependency>
124+
<groupId>io.perfmark</groupId>
125+
<artifactId>perfmark-api</artifactId>
126+
<version>0.23.0</version>
127+
<scope>runtime</scope>
128+
</dependency>
129+
<dependency>
130+
<groupId>joda-time</groupId>
131+
<artifactId>joda-time</artifactId>
132+
<scope>provided</scope>
133+
</dependency>
134+
<dependency>
135+
<groupId>com.fasterxml.jackson.core</groupId>
136+
<artifactId>jackson-databind</artifactId>
137+
<scope>provided</scope>
138+
</dependency>
139+
<dependency>
140+
<groupId>com.fasterxml.jackson.core</groupId>
141+
<artifactId>jackson-core</artifactId>
142+
<scope>provided</scope>
143+
</dependency>
144+
<dependency>
145+
<groupId>junit</groupId>
146+
<artifactId>junit</artifactId>
147+
<scope>test</scope>
148+
</dependency>
149+
<dependency>
150+
<groupId>pl.pragmatists</groupId>
151+
<artifactId>JUnitParams</artifactId>
152+
<scope>test</scope>
153+
</dependency>
154+
</dependencies>
155+
156+
<build>
157+
<plugins>
158+
<plugin>
159+
<groupId>org.apache.maven.plugins</groupId>
160+
<artifactId>maven-dependency-plugin</artifactId>
161+
<configuration>
162+
<ignoredUnusedDeclaredDependencies>
163+
<!-- Transitive dependencies from opentelemetry but explicitly added to be shadowed -->
164+
<ignoredUnusedDeclaredDependency>io.grpc:grpc-netty-shaded</ignoredUnusedDeclaredDependency>
165+
<ignoredUnusedDeclaredDependency>com.google.guava:guava</ignoredUnusedDeclaredDependency>
166+
</ignoredUnusedDeclaredDependencies>
167+
</configuration>
168+
</plugin>
169+
<plugin>
170+
<groupId>org.apache.maven.plugins</groupId>
171+
<artifactId>maven-shade-plugin</artifactId>
172+
<executions>
173+
<execution>
174+
<id>opentelemetry-extension</id>
175+
<phase>package</phase>
176+
<goals>
177+
<goal>shade</goal>
178+
</goals>
179+
<configuration>
180+
<transformers>
181+
<!-- grpc stores service providers in META-INF/services/* files,
182+
so we need to relocate the class names of the implementation classes.
183+
More about SPI - https://docs.oracle.com/javase/tutorial/ext/basics/spi.html.
184+
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html. -->
185+
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
186+
</transformers>
187+
<artifactSet>
188+
<includes>
189+
<include>io.opentelemetry</include>
190+
<include>io.grpc</include>
191+
<include>com.google.guava</include>
192+
</includes>
193+
</artifactSet>
194+
<relocations>
195+
<relocation>
196+
<pattern>com.google.common</pattern>
197+
<shadedPattern>org.apache.druid.opentelemetry.shaded.com.google.common</shadedPattern>
198+
</relocation>
199+
<relocation>
200+
<pattern>io.grpc</pattern>
201+
<shadedPattern>org.apache.druid.opentelemetry.shaded.io.grpc</shadedPattern>
202+
<includes>
203+
<include>io.grpc.*</include>
204+
</includes>
205+
</relocation>
206+
<relocation>
207+
<pattern>io.opentelemetry</pattern>
208+
<shadedPattern>org.apache.druid.opentelemetry.shaded.io.opentelemetry</shadedPattern>
209+
</relocation>
210+
</relocations>
211+
</configuration>
212+
</execution>
213+
</executions>
214+
</plugin>
215+
</plugins>
216+
</build>
217+
</project>

0 commit comments

Comments
 (0)