Skip to content

opentelemetry.adoc

Latest commit

 

History

History
408 lines (296 loc) · 15.7 KB

opentelemetry.adoc

File metadata and controls

408 lines (296 loc) · 15.7 KB

Using OpenTelemetry

This guide explains how your Quarkus application can utilize OpenTelemetry (OTel) to provide Observability for interactive web applications.

On these page we show the signal independent features of the extension.

Note

Introduction

OpenTelemetry is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. Crucially, OpenTelemetry is vendor- and tool-agnostic.

Quarkus provides manual and automatic instrumentation for tracing and manual instrumentation capabilities for metrics.

This will allow Quarkus based applications to be observable by tools and services supporting OpenTelemetry.

Note

Automatic metrics instrumentation in Quarkus is done by the Quarkus Micrometer extension.

The quarkus-micrometer-opentelemetry extension enables the use and export of Micrometer metrics via OpenTelemetry.

Quarkus supports the OpenTelemetry Autoconfiguration. The configurations match what you can see at OpenTelemetry SDK Autoconfigure with the quarkus.* prefix.

This guide provides a crosscutting explanation of the OpenTelemetry extension and how to use it. If you need details about any particular signal (tracing or metrics), please refer to the signal specific guide.

With the introduction of OpenTelemetry Metrics, the original, single page guide had to be split according to signal types, as follows:

OpenTelemetry Tracing Guide

The tracing functionality is supported and on by default.

OpenTelemetry Metrics Guide

Enable Metrics

The metrics functionality is tech preview and off by default. You will need to activate it by setting:

quarkus.otel.metrics.enabled=true

At build time on your application.properties file.

OpenTelemetry Logging Guide

Enable Logs

The logging functionality is tech preview and off by default. You will need to activate it by setting:

quarkus.otel.logs.enabled=true

At build time on your application.properties file.

Using the extension

If you already have your Quarkus project, you can add the quarkus-opentelemetry extension to it by running the following command in your project base directory:

This will add the following to your build file:

pom.xml
<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-opentelemetry</artifactId>
</dependency>
build.gradle
implementation("io.quarkus:quarkus-opentelemetry")

Create the configuration

Disable all or parts of the OpenTelemetry extension

Once you add the dependency, the extension will generate tracing data by default. To enable metrics or disable the OpenTelemetry extension globally or partially these are the properties to use (they are extracted from the config reference below):

Affected Signal Property name Default value Description

All

quarkus.otel.enabled

true

If false, disable the OpenTelemetry usage at build time.

All

quarkus.otel.sdk.disabled

false

Comes from the OpenTelemetry autoconfiguration. If true, will disable the OpenTelemetry SDK usage at runtime.

All output

quarkus.otel.exporter.otlp.enabled

true

Deprecated for removal.

If false will disable the default OTLP exporter at build time.

Traces

quarkus.otel.traces.enabled

true

If false, disable the OpenTelemetry tracing usage at build time.

Traces output

quarkus.otel.traces.exporter

cdi

List of exporters to be used for tracing, separated by commas. Has one of the values from ExporterType: otlp, cdi, none. This is a build time property and setting it to none will disable tracing data output.

Metrics

quarkus.otel.metrics.enabled

false

Metrics are disabled by default at build time because they are tech preview.

Metrics output

quarkus.otel.metrics.exporter

cdi

List of exporters to be used for metrics, separated by commas. Has one of the values from ExporterType: otlp, cdi, none. This is a build time property and setting it to none will disable metrics data output.

Logs

quarkus.otel.logs.enabled

false

Logs are disabled by default at build time because they are tech preview.

Logs output

quarkus.otel.logs.exporter

cdi

List of exporters to be used for logs, separated by commas. Has one of the values from ExporterType: otlp, cdi, none. This is a build time property and setting it to none will disable logs data output.

Logs output

quarkus.otel.logs.handler.enabled

true

If false, disable the OpenTelemetry logs handler at runtime. This removes the bridge between the Quarkus logging system (JBoss LogManager) and OpenTelemetry logs.

If you need to enable or disable the exporter at runtime, you can use the sampler because it has the ability to filter out all the spans if needed.

Particular instrumentation components can be disabled in tracing, like ignore client requests but keep server ones. For more details, please check the OpenTelemetry Tracing Guide.

Resource

A resource is a representation of the entity that is producing telemetry, it adds attributes to the exported trace or metric to characterize who is producing the telemetry. Quarkus follows the resources auto-configuration specified by the Java OpenTelemetry SDK.

Default

The following attributes are added by default to resources.

Attribute name Content example Origin

service.name

"opentelemetry-quickstart"

Value comes from the artifactId, from the quarkus.application.name property or from quarkus.otel.resource.attributes=service.name=cart property.

host.name

"myHost"

Resolved at startup

service.version

"1.0-SNAPSHOT"

Resolved at build time from the artifact version

telemetry.sdk.language

"java"

Static value

telemetry.sdk.name

"opentelemetry"

Resolved at build time

telemetry.sdk.version

"1.32.0"

Resolved at build time

webengine.name

"Quarkus"

Static value

webengine.version

"999-SNAPSHOT"

Quarkus version resolved at build time

Using configuration

You can add additional attributes by setting the quarkus.otel.resource.attributes config property that is described in the OpenTelemetry Configuration Reference. Since this property can be overridden at runtime, the OpenTelemetry extension will pick up its value following the order of precedence that is described in the Quarkus Configuration Reference.

quarkus.otel.resource.attributes=deployment.environment=dev,service.name=cart,service.namespace=shopping

This will add the attributes for deployment.environment, service.name and service.namespace to the resource and be included in traces and metrics.

Using CDI beans

If by any means you need to use a custom resource or one that is provided by one of the OpenTelemetry SDK Extensions you can create multiple resource producers. The OpenTelemetry extension will detect the Resource CDI beans and will merge them when configuring the OTel SDK.

@ApplicationScoped
public class CustomConfiguration {

    @Produces
    @ApplicationScoped
    public Resource osResource() {
        return OsResource.get();
    }

    @Produces
    @ApplicationScoped
    public Resource ecsResource() {
        return EcsResource.get();
    }
}

Semantic conventions

OpenTelemetry provides a set of semantic conventions to standardize the data collected by the instrumentation.

When creating manual instrumentation, while naming metrics or attributes you should follow those conventions and not create new names to represent existing conventions. This will make data correlation easier to perform across services.

Exporters

The Default

The Quarkus OpenTelemetry extension uses its own signal exporters built on top of Vert.x for optimal performance and maintainability. All Quarkus built in exporters use the OTLP protocol through a couple of data senders, using grpc (the default) and http/protobuf.

The active exporter is automatically wired by CDI, that’s why the quarkus.otel.traces.exporter, quarkus.otel.metrics.exporter and quarkus.otel.logs.exporter properties default value is cdi. This is not because of the protocol being used in the data transfer but because of how the exporters are wired.

CDI (Context Dependency Injection) will manage the exporters to use, according to the selected protocol or when applications implement their own CDI exporter, like in tests.

The quarkus.otel.exporter.otlp.protocol property instructs Quarkus to switch the senders and defaults to grpc but http/protobuf can also be used.

Note
 If you change the protocol, you also need to change the port in the endpoint. The default port for grpc is 4317 and for http/protobuf is 4318.

Using CDI to produce a test exporter

Leaving the default as CDI is particularly useful for tests. In the following example a Span exporter class is wired with CDI and then the telemetry can be used in test code.

Creating a custom SpanExporter bean:

    @ApplicationScoped
    static class InMemorySpanExporterProducer {
        @Produces
        @Singleton
        InMemorySpanExporter inMemorySpanExporter() {
            return InMemorySpanExporter.create();
        }
    }

Where InMemorySpanExporter is a class from the OpenTelemetry test utilities dependency:

pom.xml
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk-testing</artifactId>
        <scope>test</scope>
    </dependency>
build.gradle
implementation("io.opentelemetry:opentelemetry-sdk-testing")

The bean of that class can be injected to access the telemetry data. This is an example to obtain the spans:

    @Inject
    InMemorySpanExporter inMemorySpanExporter;

    //...

    List<SpanData> finishedSpanItems = inMemorySpanExporter.getFinishedSpanItems();

If this is used in an integration test, you should access the class from inside the running process and not from the test class. A viable option could be to expose that data through a REST endpoint method:

    @GET
    @Path("/export")
    public List<SpanData> exportTraces() {
        return inMemorySpanExporter.getFinishedSpanItems()
                .stream()
                .filter(sd -> !sd.getName().contains("export")) (1)
                .collect(Collectors.toList());
    }

  1. This excludes calls to the export endpoint itself.

For more details please take a look at the ExporterResource in the Quarkus integration tests.

The OpenTelemetry OTLP exporter

This is currently not supported in Quarkus. Configuration example for traces: quarkus.otel.tracing.exporter=otlp.

However, it’s also not needed because Quarkus own default exporters will send data using the OTLP protocol.

On Quarkiverse

Additional exporters will be available in the Quarkiverse quarkus-opentelemetry-exporter project.

Currently, are available the following exporters (may be outdated) for:

  • Legacy Jaeger

  • Microsoft Azure

  • Google Cloud

Logging exporter (for debugging)

You can output all metrics to the console, for debugging/development purposes.

Important
 Don’t use this in production.

You will need to add the following dependency to your project:

pom.xml
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-logging</artifactId>
</dependency>
build.gradle
implementation("io.opentelemetry:opentelemetry-exporter-logging")

Then, setting the exporter to logging in the application.properties file:

quarkus.otel.metrics.exporter=logging (1)
quarkus.otel.metric.export.interval=10000ms (2)
quarkus.otel.traces.exporter=logging (3)

  1. Set the metrics exporter to logging. Normally you don’t need to set this. The default is cdi.

  2. Set the interval to export the metrics. The default is 1m, which is too long for debugging.

  3. Set the traces exporter to logging. Normally you don’t need to set this. The default is cdi.

Visualizing the data

This provides a Quarkus Dev service using an "all-in-one" Grafana OTel LGTM.

Grafana is used to visualize data, Loki to store logs, Tempo to store traces and Prometheus to store metrics. Also provides and OTel collector to receive the data.

This provides an easy way to visualize all OpenTelemetry data generated by the application.

You can also use the logging exporter to output all traces and metrics to the console.

OpenTelemetry Configuration Reference

Quarkus supports the OpenTelemetry Autoconfiguration for Traces. The configurations match what you can see at OpenTelemetry SDK Autoconfigure adding the usual quarkus.* prefix.

Quarkus OpenTelemetry configuration properties now have the quarkus.otel.* prefix.