Installation

To use this extension, add following dependency to your mule application project.

    <dependency>
      <groupId>com.avioconsulting.mule</groupId>
      <artifactId>mule-opentelemetry-module</artifactId>
      <version>${mule-opentelemetry-module-version}</version> //(1)
      <classifier>mule-plugin</classifier>
    </dependency>

The latest version of the module as published on Maven Central.

Warning

Dependency group id for 1.x is com.avioconsulting, while for 2.x is com.avioconsulting.mule.

Note	This module supports OpenTelemetry Traces. Logs are not supported, yet.

Auto Configuration

Extension uses OpenTelemetry SDK v{opentelemetry-version} in its autoconfigured mode. In this mode, SDK will configure itself based on the environment variables. Supported environment variable details can be seen on open-telemetry/opentelemetry-java.

Extension Configuration

Extension allows to configure some resource and exporter attributes at individual application level. This configuration is minimal required to successfully send traces to OpenTelemetry collector.

Following example shows an OpenTelemetry Config with OTLP Exporter configured -

<opentelemetry:config name="OpenTelemetry_Config" doc:name="OpenTelemetry Config" doc:id="91477cb5-36f7-48ad-90b7-c339af87b408" serviceName="api-app-1">
    <opentelemetry:exporter >
        <opentelemetry:otlp-exporter collectorEndpoint="http://localhost:55681/v1" protocol="HTTP_PROTOBUF" requestCompression="GZIP">
            <opentelemetry:headers >
                <opentelemetry:header key="testHeader" value="testHeaderValue" />
            </opentelemetry:headers>
            <opentelemetry:config-properties >
                <opentelemetry:config-property key="otel.logs.exporter" value="otlp" />
            </opentelemetry:config-properties>
        </opentelemetry:otlp-exporter>
    </opentelemetry:exporter>
    <opentelemetry:resource-attributes >
        <opentelemetry:attribute key="mule.env" value="Dev" />
    </opentelemetry:resource-attributes>
</opentelemetry:config>

When additional properties from SDK Auto-configuration are needed for exporter, config-properties can be used to add those. Environment and System properties will still override those entries.

Resource Attributes

Common Resource Attributes

Extension uses the OpenTelemetry SDK’s Resource Provider SPI to gather data of common resources such as Host, OS, Processes of the Mule runtime Server and host. SDK supported all common resources providers are configured by this module.

Example Set of tags captured with common resource providers

  {
    "host.arch": "x86_64",
    "host.name": "ac0098.local",
    "os.description": "Mac OS X 10.16",
    "os.type": "darwin",
    "process.command_line": "/Applications/AnypointStudio.app/Contents/Eclipse/plugins/org.mule.tooling.jdk.v8.macosx.x86_64_1.1.1/Contents/Home/jre:bin:java -Dmule.home=/Applications/AnypointStudio.app/Contents/....d=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang",
    "process.executable.path": "/Applications/AnypointStudio.app/Contents/Eclipse/plugins/org.mule.tooling.jdk.v8.macosx.x86_64_1.1.1/Contents/Home/jre:bin:java",
    "process.pid": "9778",
    "process.runtime.description": "AdoptOpenJDK OpenJDK 64-Bit Server VM 25.282-b08",
    "process.runtime.name": "OpenJDK Runtime Environment",
    "process.runtime.version": "1.8.0_282-b08"
  }

See Disabling Common Resource Providers if any resource provider must be disabled.

Mule Resource Attributes

Extension is aware of CloudHub Reserved Properties and automatically adds some of that data into trace data.

Example Set of tags captured with Mule Resource Provider

{
    "mule.app.awsRegion": "us-west-2",
    "mule.app.domain": "mule-opentelemetry-app",
    "mule.app.fullDomain": "mule-opentelemetry-app.us-w2.cloudhub.io",
    "mule.csOrganization.id": "f2ea2cb4-c600-gh87-gg78-e952ff5591ee",
    "mule.organization.id": "f2ea2cb4-c600-gh87-gg78-e952ff5591ee",
    "mule.env": "Dev",
    "mule.environment.id": "c06ef9b7-19c0-ss78-kk44-598058b20aad",
    "mule.environment.type": "sandbox",
    "mule.home": "/opt/mule/mule-4.4.0",
    "mule.worker.id": "0"
}

Note	CloudHub does not define any property for organization id. `mule.organization.id` refers to value of a system property `csOrganization.id` defined by CloudHub.

Mule Application Host Resource Attributes

This extension (since 2.6.0) applies a custom resource attribute strategy for host and container related attributes. The values differ based on whether the applications are deployed to CloudHub V1 or V2 or other.

Following table summarizes the attributes value strategy -

Table 1. Mule Application Host and Container Resource Attributes

Deployment Target	Attribute Name	Value Strategy	Description	Example
CloudHub v1	host.name	Anypoint Environment Id	Uses environment id in which application is deployed	522689ad-f2c6-481b-9165-6af716cf0312
CloudHub v1	host.ip	Public IP of the worker	Uses CloudHub worker’s public IP Address	30.40.50.60
CloudHub v1	container.id	Application domain + worker number	Uses the applications CloudHub domain name suffixed with the worker number	org-employee-xapi-dev-0, org-employee-xapi-dev-1
CloudHub v1	container.name	Application domain + worker number	Uses the applications CloudHub domain name suffixed with the worker number	org-employee-xapi-dev-0, org-employee-xapi-dev-1
CloudHub v2	host.name	RTF Node Name	Being a containerized environment, underlying RTF’s Node name is used	ip-10-20-30-40-us-west-2-compute
CloudHub v2	host.ip	Not available	Not available	Not available
CloudHub v2	container.id	Actual container id of the application	Being a containerized environment, container id of the application is used	84e29fd130cbf32e5cb86ca75cbc6c847307ae02fa49c4b230d82490cd298602
CloudHub v2	container.name	Application replica name	Uses the individual replica name	org-employee-xapi-dev-9c5df98ff-x5xww, org-employee-xapi-dev-9c5df98ff-pdccr

Note	To get telemetry data of an individual workers/replica, use `service.name` and `container.name` attributes to filter data.

Cloud Resource Attributes

The extension captures (since 2.6.0) following cloud attributes for CloudHub environment-

Table 2. Mule CloudHub Resource Attributes

Deployment Target	Attribute Name	Value Strategy	Description	Example
Any	cloud.provider	mulesoft	Static value	`mulesoft`
CloudHub v1	cloud.region	Application AWS Region	AWS region as available from CloudHub reserved properties	us-west-2
CloudHub v1	cloud.platform	mulesoft_cloudhub_v1	Static value	`mulesoft_cloudhub_v1`
CloudHub v2	cloud.platform	mulesoft_cloudhub_v2	Static value	`mulesoft_cloudhub_v2`
CloudHub v1/v2	cloud.account.id	Anypoint Organization Id	Uses organization id in which application is deployed	`81517987-fd9f-4035-b579-c2caa87b36aa`

Exporters

Extension supports following exporter configurations -

OTLP Exporter
Logging Exporter
Generic Exporter

Note	Configured exporter is used for all supported signals.

OTLP Exporter

Extension contains all dependencies needed to send supported signals to an OpenTelemetry Collector endpoint.

Note

When configuring the OTLP Exporter with HTTP Protobuf protocol, OpenTelemetry collector endpoint must be set to the base endpoint of OTEL collector. The module will build signal-specific endpoints such as {collectorEndpoint}/traces based on OpenTelemetry specification guidelines. For example, if opentelemetry collector is listening on localhost:4317, then set collectorEndpoint=http://localhost:4317/v1 and NOT collectorEndpoint=http://localhost:4317/v1/traces.

Note

When configuring the OTLP Exporter with GRPC protocol, OpenTelemetry collector endpoint must be set to the base endpoint of OTEL collector. For example, if opentelemetry collector is listening on localhost:4317, then set collectorEndpoint=http://localhost:4317 and NOT collectorEndpoint=http://localhost:4317/v1.

Configuring TLS for Exporter Endpoint

If OTLP collector endpoint is using HTTPS, this exporter may require additional configuration when non-public CA certified certificates are used.

Following attributes can be used to configure certificates on OTLP Exporter -

Server (Receiver Endpoint) Certificates:

endpointCertPath - The path to the file containing trusted certificates to use when verifying an OTLP trace, metric, or log server’s TLS credentials. The file should contain one or more (Server, CA chain etc.) X.509 certificates in PEM format. By default, the host platform’s trusted root certificates are used. For example, if file is included in src/main/resources/certs/server-certs.pem, set this attribute to certs/server-certs.pem.

Client (Mule app) Certificates:

clientCertPath - The path to the file containing trusted certificates to use when verifying an OTLP trace, metric, or log client’s TLS credentials. The file should contain one or more X.509 certificates in PEM format. By default, no chain file is used.
clientCertKeyPath - The path to the file containing private client key to use when verifying an OTLP trace, metric, or log client’s TLS credentials. The file should contain one private key PKCS8 PEM format. By default, no client key is used.

Troubleshooting

OTLP Exporter failed to export spans. Server responded with HTTP status code 404.

OTLP Exporter is being used but no traces are seen in APM. The logs show below (or similar) error message -

Failed to export spans. Server responded with HTTP status code 404. Error message: Unable to parse response body, HTTP status message: Not Found

This is most likely caused due to incorrect value set for collectorEndpoint. See OTLP Exporter configuration reference for how to configure collector endpoint correctly.

OTLP Exporter with GRPC Protocol failed to export spans with "OTLP endpoint must not have a path: /v1"

This is most likely caused due to incorrect value set for collectorEndpoint. See OTLP Exporter configuration reference for how to configure GRPC collector endpoint correctly.

APM Collector does not support OTEL standard endpoint format

A few APMs may not have the OTEL standard endpoint format of {collectorEndpoint}/{signal}. In that case, the default config property of collectorEndpoint may not work to auto-build the single endpoints.

In such cases, opentelemetry-config-properties can be used to define trace endpoint with otel.exporter.otlp.{signal}.endpoint property where signal can be traces or metrics.

Example OTEL exporter with Traces endpoint

<opentelemetry:otlp-exporter collectorEndpoint="${otel.collectorEndpoint}">
    <opentelemetry:config-properties >
        <opentelemetry:config-property key="otel.exporter.otlp.traces.endpoint" value="${my-custom-otel-trace-endpoint-url}" />
    </opentelemetry:config-properties>
</opentelemetry:otlp-exporter>

Logging Exporter

When troubleshooting generated signal data, sending it to logs may be useful. Extension supports a simple logging exporter that can send signal data to application’s log file.

Logging Exporter Configuration

<opentelemetry:config name="OpenTelemetry_Logging" doc:name="OpenTelemetry Config" serviceName="app1" >
    <opentelemetry:exporter >
        <opentelemetry:exporter>
            <opentelemetry:logging-exporter />
        </opentelemetry:exporter>
    </opentelemetry:exporter>
</opentelemetry:config>

Example span entry from log file -

Trace Log

[INFO ] [2022-10-13 15:54:37,141] [[MuleRuntime].uber.08: [orders-exp-api].submit-order-flow.CPU_INTENSIVE @1be1852e] [event: dd4e8f20-4b30-11ed-87e6-c889f3a9023b] [io.opentelemetry.exporter.logging.LoggingSpanExporter]: '/api/*' : 0cda0930cbf01126b91402861dbffc74 38d96ac87afdbbe1 SERVER [tracer: mule-opentelemetry-module:1.1.0] AttributesMap{data={http.status_code=201, http.route=/api/*, http.user_agent=PostmanRuntime/7.29.2, mule.app.flow.source.configRef=HTTP_Listener_config, http.scheme=http, http.method=POST, mule.app.flow.name=order-exp-main, http.flavor=1.1, mule.serverId=abcd..orders-exp-api, http.target=/api/orders, mule.correlationId=dd4e8f20-4b30-11ed-87e6-c889f3a9023b, mule.app.flow.source.namespace=http, http.host=localhost:8081, mule.app.flow.source.name=listener}, capacity=128, totalAddedValues=14}

Generic Exporter

This generic exporter allows to configure any other signal exporters supported by sdk-extensions/autoconfigure#exporters.

Following example shows possible configuration for sending traces to Zipkin.

Warning

If the generic exporter is used to configure signal specific exporter, then it must be configured appropriately for all supported signals.

Generic Exporter Configuration

<opentelemetry:config name="OpenTelemetry_Generic" doc:name="OpenTelemetry Config" serviceName="app1" >
    <opentelemetry:exporter >
        <opentelemetry:generic-exporter >
            <opentelemetry:config-properties >
                <opentelemetry:config-property key="otel.traces.exporter" value="zipkin" />
                <opentelemetry:config-property key="otel.exporter.zipkin.endpoint" value="http://localhost:9411/api/v2/spans" />
            </opentelemetry:config-properties>
        </opentelemetry:generic-exporter>
    </opentelemetry:exporter>
</opentelemetry:config>

The required Zipkin exporter dependencies must be configured as an Additional Plugin Dependencies for Mule Maven Plugin.

<plugin>
    <groupId>org.mule.tools.maven</groupId>
    <artifactId>mule-maven-plugin</artifactId>
    <version>${mule.maven.plugin.version}</version>
    <extensions>true</extensions>
    <configuration>
        <additionalPluginDependencies>
            <plugin>
                <groupId>com.avioconsulting.mule</groupId>
                <artifactId>mule-opentelemetry-module</artifactId>
                <additionalDependencies>
                <!--
                    Module uses OpenTelemetry SDK v{opentelemetry-version}.
                     Any opentelemetry dependencies used here must be at-least v{opentelemetry-version}
                     or a compatible one.
                -->
                    <dependency>
                        <groupId>io.opentelemetry</groupId>
                        <artifactId>opentelemetry-exporter-zipkin</artifactId>
                        <version>{opentelemetry-version}</version>
                    </dependency>
                </additionalDependencies>
            </plugin>
        </additionalPluginDependencies>
    </configuration>
</plugin>

Tracing Configuration

Span Processors

For non-logging exporters, Tracing SDK uses Batch Span Processor. Global Configuration allows to customize Batch span processor settings -

OpenTelemetry config with Batch span processor default values

<opentelemetry:config name="OpenTelemetry_Config"
    serviceName="otel-comparison-test"
    maxQueueSize="2048"
    maxBatchExportSize="512"
    batchExportDelayInterval="5000"
    exportTimeout="30000">
.... other config ....
</opentelemetry:config>

Span Sampler

By default, every span is recorded. In a high transaction environment, this can become noisy or needing high storage requirements for backend APM. In such cases, it is possible to reduce the span recorded and sent to the APM. This can help reduce the network traffic as well as data sent to the backend. Although, it comes at a cost of not collecting all traces, which maybe acceptable in certain use cases.

The Sampler configures weather spans will be recorded when they are started by the module. Unrecorded spans are skipped from exporting to backend APM.

Module config doesn’t have any elements to set the sampler configuration, but it can be applied using OpenTelemetry’s system properties.

System property	Environment variable	Description
otel.traces.sampler	OTEL_TRACES_SAMPLER	The sampler to use for tracing. Defaults to `parentbased_always_on`
otel.traces.sampler.arg	OTEL_TRACES_SAMPLER_ARG	An argument to the configured tracer if supported, for example a ratio.

See Sampler documentation for more details.

For example, to set the TraceId Ratio based sampler, you can add following two properties -

otel.traces.sampler=parentbased_traceidratio
otel.traces.sampler.arg=0.001   // (1)

Sets the trace id ratio to 1 in 1000. Resulting configuration will record 1 in 1000 traces.

Trace Spans

By default, this module will create trace spans for following mule components -

Flows
HTTP Listener and Request
Database Connector
Anypoint MQ Connector

More verbose span generation can be configured. See setting Trace Levels below.

Trace Levels

Module can create spans for every mule processors by setting spanAllProcessors = "true". This can be overridden by setting a system property mule.otel.span.processors.enable to true|false.

When the span generation for all processors is enabled, opentelemetry:ignore-mule-components allows to set a list of processors to exclude from span generation.

OpenTelemetry Config with trace level configuration

<opentelemetry:config name="OpenTelemetry_Generic" doc:name="OpenTelemetry Config" serviceName="app1"  spanAllProcessors="true">
    <opentelemetry:exporter >
        <opentelemetry:generic-exporter >
            <opentelemetry:config-properties >
                <opentelemetry:config-property key="otel.traces.exporter" value="zipkin" />
                <opentelemetry:config-property key="otel.exporter.zipkin.endpoint" value="http://localhost:9411/api/v2/spans" />
            </opentelemetry:config-properties>
        </opentelemetry:generic-exporter>
    </opentelemetry:exporter>
    <opentelemetry:ignore-mule-components >
        <opentelemetry:mule-component namespace="MULE" name="LOGGER" />
        <opentelemetry:mule-component namespace="os" name="*" />
    </opentelemetry:ignore-mule-components>
</opentelemetry:config>

To disable span generation for all processors in a specific namespace, set the name attribute to * -

<opentelemetry:mule-component namespace="os" name="*" />

Instead of enabling span generation for all processors, if some additional processors need to be included in span generation, then opentelemetry:span-additional-mule-components can be used to include those in generic span generation process.

OpenTelemetry Config additional mule components to generate spans for

<opentelemetry:config name="OpenTelemetry_Generic" doc:name="OpenTelemetry Config" serviceName="app1" spanAllProcessors="false"> // (1)
    <opentelemetry:exporter >
        <opentelemetry:generic-exporter >
            <opentelemetry:config-properties >
                <opentelemetry:config-property key="otel.traces.exporter" value="zipkin" />
                <opentelemetry:config-property key="otel.exporter.zipkin.endpoint" value="http://localhost:9411/api/v2/spans" />
            </opentelemetry:config-properties>
        </opentelemetry:generic-exporter>
    </opentelemetry:exporter>
    <opentelemetry:span-additional-mule-components >        // (2)
        <opentelemetry:mule-component namespace="custom" name="operation" />
        <opentelemetry:mule-component namespace="mule" name="remove-variable" />
    </opentelemetry:span-additional-mule-components>
</opentelemetry:config>

Span All processors is set false to reduce span generation for common components
Add opentelemetry:span-additional-mule-components to generate spans for a sample custom:operation and mule:remove-variable processors

Add Custom Transaction Tags

In addition to all the trace attributes captured by the module, it is possible to add custom tags to the current trace using an operation opentelemetry:add-transaction-tags.

Warning

All custom tag keys are transformed to custom.{keyName}. This also prevents accidentally overriding other standard keys-value pairs in trace tags. Depending on the APM (elastic, etc.) you use, they may be displayed differently. For example, elastic will display them as label.custom_{keyName}.

These could be any business data that you may want to capture as a part of your telemetry data. For example, an order number for an order processing transaction.

Adding custom tag from variable

    <opentelemetry:add-transaction-tags doc:name="Add Custom Tags"
                config-ref="OpenTelemetry_Config">
        <opentelemetry:tags >
            <opentelemetry:tag key="orderNumber" value="#[vars.orderNumber]"/>
        </opentelemetry:tags>
    </opentelemetry:add-transaction-tags>

You can also use dataweave to set the tags.

Adding custom tags as DataWeave map

    <opentelemetry:add-transaction-tags doc:name="Add Custom Tags"
                config-ref="OpenTelemetry_Config"
                tags="#[output java --- {orderNumber: payload.orderNumber}]" />

Global Config Span tags

Some APMs may require additional tags on spans for the correct display of traces. For example, Splunk APM can use peer.service on http request spans when creating inferred services in service maps.

The module may not be capturing those tags out of the box but there is a way to add additional tags to the spans of components that use global configuration elements. Some examples would be http:listener using http:listener-config, db:insert using db:config.

For such cases, the module recognizes system properties defined with property names following the pattern {global_config_element_name}.otel.{tag_name} and adds {tag_name}:{property_value} as tags to spans generated for all components using {global_config_element_name} named global element.

Caution

Any tags set using this system properties, will override module generated value for same tags.

Few things to consider when using Global Tags:

Global Config spans are supported via System Properties. Setting global-property in mule configuration does not go in System Properties and hence won’t work. Check System Properties for On-Prem or Runtime Manager to set attribute values.
This feature can ONLY add/modify Span attributes, and not the name, kind or other non-attribute data on the span. Most of the APMs may use Span name in Trace UI, and it is NOT possible to change how this module uses OpenTelemetry guidelines to compute span names.

In case of Splunk, peer.service attribute should have the name of the remote http system being invoked. Consider following mule requester example -

    <!-- Global HTTP Request Configuration element -->
	<http:request-config name="Remote_Request_configuration" doc:name="HTTP Request configuration"> // (1)
		<http:request-connection host="${http.host}" port="${http.port}" />
	</http:request-config>

    <!-- Flow including http:request that references above global config -->
	<flow name="mule-opentelemetry-app-requester-remote" >
		<http:listener doc:name="Listener" config-ref="HTTP_Listener_config" path="/test-remote-request"/>
		<http:request method="GET" doc:name="Request" config-ref="Remote_Request_configuration" path="/test/remote/target"/> // (2)
		<logger level="INFO" doc:name="Logger"/>
	</flow>

To add a tag peer.service=my_remote_api to http:request 's span, you can set following system property on mule runtime -

Remote_Request_configuration.otel.peer.service=my_remote_api

Context Propagation

This module supports context propagation in W3C Trace Context and W3C Baggage Context formats.

Context Extraction

Extension supports extracting Open Telemetry Trace context extraction for certain source components. For these components if the Context information is received in appropriate place, the module will establish the parent-child relation for the traces.

Source Components supporting context extraction:

HTTP Listener: Context information, if exists, is extracted from request headers
Anypoint MQ Subscription: Context information, if exists, is extracted from Anypoint MQ Message properties

For any other source components, the module looks for the Context information in following locations of the event and in the order mentioned below -

attributes.headers
- Example components are Kafka, AMQP, WebSockets, etc.
attributes.properties
- For example, Anypoint MQ Module.
attributes.properties.userProperties
- Example components are JMS, IBM MQ etc.
payload.message.messageAttributes
- For example, Amazon SQS Module

Note	OpenTelemetry Trace Context is extracted/injected using configured Propagators. The entries in the context may vary depending on the propagators used and validations it applies. All examples here are with W3C Trace Context.

Context Injection

To help with the context propagation, module allows to inject context into flow variables. This context includes following trace attributes -

TRACE_TRANSACTION_ID - An internal transaction id within Mule Context
traceId - Trace id of the current request
traceIdLongLowPart - Long value of the Trace Id Low part
spanId - Span Id for the component used for creating context
OpenTelemetry Trace attributes such as traceparent, tracestate

Context can be injected in two ways, as described below.

Auto Injection to Flow Variables

Extension uses a processor interceptor. OpenTelemetry’s tracing context will be automatically added to a flow variable before the first processor is invoked. It is always injected under a key OTEL_TRACE_CONTEXT.

Before the first processor is invoked - this context relates to the flow span instead of any specific processor
Before certain outbound specific processors are invoked - this context is specific to the span of the processor being intercepted

Table 3. Supported Properties to control interceptor behavior

System Property	Environment Variable	Description	Default Value
mule.otel.interceptor.processor.enable	MULE_OTEL_INTERCEPTOR_PROCESSOR_ENABLE	Enable or Disable the interceptor feature. Disabling the interceptor processing can result in incorrect context propagation.	`false`
mule.otel.interceptor.first.processor.only	MULE_OTEL_INTERCEPTOR_FIRST_PROCESSOR_ONLY	When interception is enabled, intercept only the first processor in each flow	`false`

First Processor Interceptor

When the flow execution starts, OpenTelemetry context is injected into flow variables before the first processor of the flow is invoked.

Warning

This context relates to the main flow span and if used for propagation to external services then span rendering may not look accurate.See processor interceptor below.

Note	OTEL_TRACE_CONTEXT.spanId will be of the flow container span.

Following examples show a W3C Trace Context extracted from incoming http request and injected into flow variables:

Figure 1. Context Injection - First Processor

Another variation when tracestate is received with traceparent

Figure 2. Context Injection - First Processor (variation)

Processor Interceptor

For the context propagation accuracy, certain processors are intercepted to inject current span’s context into the flow variable.

Note	OTEL_TRACE_CONTEXT.spanId will be of the span of the intercepted component.

The connector is pre-configured to intercept following connector operations -

Table 4. List of intercepted operations

../../main/resources/com/avioconsulting/mule/opentelemetry/internal/interceptor/intercept-components.txt

As a result of this, for example, when http:request makes an outbound request and context is injected, http:request processor’s span is propagated as a prent span.

Figure 3. Span View in Elastic APM

If the intercepted processors needs fine-tuning such as including or excluding certain processors then it can be done in the Trace Level global configuration.

Warning

Excluding the intercepted components may affect the context propagation.

Figure 4. Trace Level - Processor Interception Configuration

Manual Injection

If needed, <opentelemetry:get-current-trace-context /> operation can be used to manually inject trace context into flow.

Note	`target` must be used to set operation output to a flow variable.

Note	OTEL_TRACE_CONTEXT.spanId will be of the flow container span.

Warning

Similar to First Processor Interceptor, this context relates to the main flow span and if used for propagation to external services then span rendering may not look accurate.

<opentelemetry:get-current-trace-context doc:name="Get Current Trace Context" config-ref="OpenTelemetry_Config" target="traceContext"/>

HTTP Request Context Injection

The Module does NOT support automatic context propagation. In order to propagate the trace header information to other web applications, the Mule HTTP Requester Operation OR Configuration must be configured to propagate following http headers -

Table 5. HTTP Headers for Trace Context

Key	Value
`traceparent`	`#[vars.OTEL_TRACE_CONTEXT.traceparent default '' as String]`
`tracestate`	`#[vars.OTEL_TRACE_CONTEXT.tracestate default '' as String]`

Using Operation vs. Global Configuration:

HTTP Outbound headers can be added at HTTP Request Operation level OR as a default headers in HTTP Request Global Configuration.

HTTP Global configuration may be easier from code modifications perspective i.e. just at one global level. That comes at the cost of performance because using expressions in configurations makes it a dynamic configuration.

On the other hand, using headers section in HTTP Request operation may need modification of every http:request operation, but then it keeps the global configuration as a static configuration for runtime to reuse the same instance.

Warning

Using Static configurations is a preferred approach from performance perspective. Dynamic is only recommended for lower request volume uses cases. For more information on performance impact, see performance report section.

Propagation with Request Operation: To propagate context using request operation, add/append request headers section of every http:request operation to include following entries -

Example GET Request with OTEL Trace context headers

<http:request method="GET" doc:name="Request" doc:id="f187cd33-70ce-4e09-96e0-70115c3ac727" config-ref="HTTP_Request_configuration" path="/api/path">
    <http:headers ><![CDATA[#[output application/java
---
{
"traceparent": vars.OTEL_TRACE_CONTEXT.traceparent as String,
"tracestate": vars.OTEL_TRACE_CONTEXT.tracestate default '' as String
}]]]>
    </http:headers>
</http:request>

Propagation with Requester Global Configuration:

In order to propagate the trace header information to other web applications, the Mule HTTP Requester Configuration must have default headers configured in the following way:

Mule configuration xml for setting default headers in the HTTP Requester Configuration

<http:request-config name="HTTP_Request_configuration"
    doc:name="HTTP Request configuration">
	<http:request-connection host="some-api.us-e1.cloudhub.io" />
	<http:default-headers >
		<http:default-header key='traceparent' value="#[vars.OTEL_TRACE_CONTEXT.traceparent default '' as String]" />
		<http:default-header key='tracestate' value="#[vars.OTEL_TRACE_CONTEXT.tracestate default '' as String]" />
	</http:default-headers>
</http:request-config>

Anypoint MQ Context Injection

When using Anypoint MQ, the publish operation can add vars.OTEL_TRACE_CONTEXT to user properties. If this module is being used by the Anypoint MQ Subscription application, the Context will be extracted from the user properties and linked to the parent incoming trace.

    <anypoint-mq:publish doc:name="Publish" doc:id="8e707130-9ead-4dac-a31e-f7bcb5ce7740" config-ref="Anypoint_MQ_Config" destination="otel-test-queue-1">
        <anypoint-mq:properties ><![CDATA[#[vars.OTEL_TRACE_CONTEXT]]]></anypoint-mq:properties>
    </anypoint-mq:publish>

Context Injection to other modules

The module supports extracting the Context from specific locations. Check Context Extraction section for list of supported locations.

Following example shows use of JMS User properties for sending the context information when publishing a message to the queue -

    <jms:publish doc:name="Publish" doc:id="ff1a9b50-b399-41b7-a43e-0b3b7403c93c" config-ref="JMS_Config" destination="mule.test">
        <jms:message >
            <jms:body ><![CDATA[#[output application/json
---
{
    id: uuid(),
    name: "Random"
}]]]></jms:body>
            <jms:properties ><![CDATA[#[output application/java
---
{
    traceparent: vars.OTEL_TRACE_CONTEXT.traceparent as String default '',
    tracestate: vars.OTEL_TRACE_CONTEXT.tracestate as String default '',
}]]]></jms:properties>
        </jms:message>
    </jms:publish>

Note	If the consumer listener Mule App is also instrumented using this module, the context will be extracted appropriately to create a distributed trace.

Turn Off Tracing

Once you have configured the module in your application, there may be a need to remove or temporarily turn it off.

To permanently remove the tracing -

Remove the module dependency from pom.xml
Remove the global configuration element and xml declaration references
Remove any changes made to other Connector configurations for context propagation.

To temporarily disable the tracing without any code changes -

Set turnOffTracing="true" on global config. You may use a property placeholder for the value.
Alternately, you can set the mule.otel.tracing.disabled system property to true.
To re-enable the tracing, just reset the property to false (default value).

Logs Correlation

When APM backends are used to capture Logs as well as Traces, they might support log and trace correlation. Depending on the APM, some specific attributes maybe needed in the log records to correlate them to the corresponding traces.

Most commonly used attributes include -

Trace Id
- Loggers can access the current trace id with vars.OTEL_TRACE_CONTEXT.traceId
- Some APM backends (eg. DataDog) may require the Long trace Id instead of the 32-hex-character trace Id value. In that case, vars.OTEL_TRACE_CONTEXT.traceIdLongLowPart (since v1.6.0) can be used.
Span Id
- Introduced with v1.5.0, Loggers can access the flow container span id with vars.OTEL_TRACE_CONTEXT.spanId
- Some APM backends (eg. DataDog) may require the Long span Id instead of the 16-hex-character span Id value. In that case, vars.OTEL_TRACE_CONTEXT.spanIdLong (since v1.6.0) can be used.
Service name
- Usually a static value, name of the application which can be injected through application properties such as ${domain}
Deployment Environment
- Usually a static value, name of the environment which can be injected through application properties such as ${mule.env}

Using Mule Tracing Module

To add trace ID and span ID, you may add a mule tracing module to target the Mule application.

    <dependency>
        <groupId>org.mule.modules</groupId>
        <artifactId>mule-tracing-module</artifactId>
        <version>1.0.0</version>
        <classifier>mule-plugin</classifier>
    </dependency>

In the first APIKit flow, add tracing module operations -

    <tracing:set-logging-variable doc:name="Set Trace Id"
        variableName="#['trace_id']"
        value="#[vars.OTEL_TRACE_CONTEXT.traceId]"/>
    <tracing:set-logging-variable doc:name="Set Span Id"
        variableName="#['span_id']"
        value="#[vars.OTEL_TRACE_CONTEXT.spanId default '']" />

Use Log4J JSON Layout

When logging with Log4J, JSONLayout can be used to structure the log records with additional attributes.

Example JSONLayout writing to CONSOLE

    <!-- Console JSON layout that can be used to see full JSON format that will be sent for log aggregation in a real environment -->
    <console name = "CONSOLE_JSON" target = "SYSTEM_OUT">
        <JSONLayout includeTimeMillis="true" compact="false" eventEol="true" objectMessageAsJsonObject="true" properties="true" stacktraceAsString="true">
            <KeyValuePair key="trace_id" value="${ctx:trace_id}"/>
            <KeyValuePair key="span_id" value="${ctx:span_id}"/>
            <KeyValuePair key="service.name" value="$${sys:domain}"/>
            <KeyValuePair key="deployment.environment" value="$${env:mule.env}"/>
            <KeyValuePair key="timestamp" value="$${date:yyyy-MM-dd'T'HH:mm:ss.SSSZZZZ}" />
        </JSONLayout>
    </console>

Generated Documentation

Connector documentation
Java API Documentation

Limitations

Automatic header/attribute injections for outbound requests is not supported
When using in on-premise mode, all applications deployed to the same runtime will share the same instance of OpenTelemetry configuration. It is unpredictable that which application’s configuration wins. Ideally, the configuration should be same across the applications.
When using mule domain projects for global configurations, the generated spans do not include any global configuration or connection tags.

Files

module-config.adoc

Latest commit

History

module-config.adoc

File metadata and controls

Installation

Auto Configuration

Extension Configuration

Resource Attributes

Common Resource Attributes

Mule Resource Attributes

Mule Application Host Resource Attributes

Cloud Resource Attributes

Exporters

OTLP Exporter

Configuring TLS for Exporter Endpoint

Troubleshooting

OTLP Exporter failed to export spans. Server responded with HTTP status code 404.

OTLP Exporter with GRPC Protocol failed to export spans with "OTLP endpoint must not have a path: /v1"

APM Collector does not support OTEL standard endpoint format

Logging Exporter

Generic Exporter

Tracing Configuration

Span Processors

Span Sampler

Trace Spans

Trace Levels

Add Custom Transaction Tags

Global Config Span tags

Context Propagation

Context Extraction

Context Injection

Auto Injection to Flow Variables

First Processor Interceptor

Processor Interceptor

Manual Injection

HTTP Request Context Injection

Anypoint MQ Context Injection

Context Injection to other modules

Turn Off Tracing

Logs Correlation

Using Mule Tracing Module

Use Log4J JSON Layout

Generated Documentation

Limitations