v2.5.6 - Experimental Akka Cluster Metrics

Akka Cluster Metrics

This release comes with experimental support for a bunch of Akka Cluster-related metrics! We can break down the metrics into per-state member counts that help you get a general view of the cluster status, and per-member status counts, which let you see exactly which nodes are on each state.

Per-state Member Counts

These metrics track how many cluster members are on each of the possible states in the cluster. The metric names for these states are:

• akka.cluster.members.joining.count
• akka.cluster.members.weakly-up.count
• akka.cluster.members.up.count
• akka.cluster.members.leaving.count
• akka.cluster.members.exiting.count
• akka.cluster.members.down.count
• akka.cluster.members.removed.count
• akka.cluster.members.total.count

All members of the cluster expose these metrics with the number of members they see in each state. Since these metrics are exposed as gauges, you could always make a chart of the max(akka.cluster.members.[state].count) to see how many members are on a given state. For example, for members in the Up state you can see how a cluster starts growing in members:

Per-member Status Counts

The metrics we mentioned above will help you know, for example, that there are 3 members in the Joining state, but what if you want to know whichare those members? That's what the per-member metrics are for. These metrics have a member tag that identifies the member being measured, and the value of the metric tells you the state it is currently in. Here is a breakdown of the possible values:

akka.cluster.member.status

• Joining = 1
• WeaklyUp = 2
• Up = 3
• Leaving = 4
• Exiting = 5
• Down = 6
• Removed = 7

akka.cluster.member.reachability

• Reachable = 0
• Unreachable = 1

The same as with the status metrics, you can plot the max(akka.cluster.members.status) and see what's the latest on each member of the cluster! In the example below, all members are on state 3 (Up):

Enabling Cluster Metrics

This feature isn't fully tested yet so we are not enabling it by default. To enable Akka Cluster Metrics, add this setting to your application.conf file:

kamon.instrumentation.akka.cluster.track-cluster-metrics=yes

New Features

• akka: Publish Akka Cluster metrics for members' status and reachability. Contributed by @ivantopo via #1189

v2.5.5 - ClassLoading Improvements for Play Framework

Improvements

• play: We are now using the Play Environment's class loader to create all Kamon-related objects. This ensures that custom classes (like samples or SlowStatementProcessors) can be loaded from the classpath when running on development mode. Contributed by @ivantopo via #1185

v2.5.4 - Maintenance Update

Fixes

• bundle: The new kamon-aws-sdk module had a Compile dependency on the Dynalite test container that was brining a bunch of extra classes into the Bundle and messing with user-provided dependencies 🤦. Reported by @fggarcia on #1171 and fixed by @ivantopo via #1178
• aws-sdk: Duplicate execution attribute definitions when using more than one AWS SDK Client. Reported by @hhalex on #1173 and fixed by @ivantopo on #1175

v2.5.3 - AWS SDK for Java & Basic Alpakka Kafka

🔥 🚨 🔥 Skip this release! This release has a dependency packaging issue described in #1171. Please jump straight to the latest and greatest Kamon Telemetry version!

AWS SDK for Java Support

This release ships automatic instrumentation for the AWS SDK for Java, both 1.x and 2.x! The support is pretty basic: just tracing interactions with the SDK client libraries. Of course, the automatic metrics collected from spans will be there to help, but that's it.

The immediate next steps for the AWS SDK instrumentation are to ensure context propagation through SQS (same as we do for Kafka) and start extracting more request-specific information to enrich those spans. Feel free to stop by our Discord or open an issue to tell us what you are missing!

New Features

• aws-sdk: We are shipping a new kamon-aws-sdk instrumentation module that comes with basic tracing capabilities for the AWS Java SDK. Contributed by @ivantopo via #1167
• alpakka-kafka: We are shipping a new kamon-alpakka-kafka instrumentation module that includes automatic context propagation for ProducerMessage instances. Contributed by @ivantopo via #1168

v2.5.2 - System Metrics on Scala 3 & Network Metrics Fixes

Improvements:

• system-metrics": This module is now built and published for Scala 3 as well. Contributed by @ornicar via #1155
• otel: The OpenTelemetry reporter now supports the OTEL_RESOURCE_ATTRIBUTES environment variable. Contributed by @hughsimpson via #1157
• otel: The OpenTelemetry exporter dependencies are now bumped to 1.13.0. Contributed by @hughsimpson via #1158

Fixes:

• system-metrics: No network metrics were reported by the System Metrics module when running inside docker containers. Fixed by @jypma via #1165

v2.5.1 - OpenTelemetry over HTTP & Datadog Propagation

New Features

• core: This release ships with a new datadog Span propagation scheme that uses the x-datadog-trace-id, x-datadog-parent-id, and x-datadog-sampling-priority headers for all tracing information, exactly as Datadog does it. Contributed by @seglo and @leonwlaw via #1145
• core: It is now possible to filter the metrics sent to a metrics reporter via configuration. You can use kamon.modules.reporter-name.metric-filters to configure an includes/excludes filter with the metrics you want to be exported. Contributed by @ivantopo via #1148
• otel: The OpenTelemetry traces reporter now supports http/protobuf protocol. The entire OTel reporter was rewritten on top of the official OTel exporter and should make it really easy to get into exporting metrics soon 🎉. Contributed by @hughsimpson via #1102

Fixes

• twitter-future: Fixed context propagation issues that were haunting us for ages in the Twitter Future. Many thanks to @dispalt for providing reproducibles for the issue and helping debug this issue. Fixed by @ivantopo via #1147
• akka-http: We were unable to trace HTTP/1 requests on Akka HTTP servers that had HTTP/2 enabled, now we can. Many thanks to @jtjeferreira for writing a failing test for the issue. Fixed by @ivantopo via #1094

v2.5.0 - Akka gRPC Server Support & Improvements

Akka gRPC Server Support

Starting on this release, Kamon Telemetry ships with support for tracing server-side Akka gRPC requests implemented with Akka HTTP, Play Framework, and Lagom. This instrumentation is the first step towards full Akka gRPC support and we will soon add:

• gRPC client tracing
• Context propagation across gRPC channels

Improvements

• core: Add the error.type tag to traces, getting us closer to follow the OpenTelemetry semantic conventions. Contributed by @hughsimpson via #1112
• kafka: Prevent the Kafka producer instrumentation from starting traces when there is no current trace on the application. There is a new kamon.instrumentation.kafka.client.tracing.start-trace-on-producer setting to control this behavior. Contributed by @ivantopo via #1125
• akka-grpc: New instrumentation for server-side Akka gRPC. Contributed by @ivantopo via #1119
• akka: #1093 New instrumentation for Akka's Scheduler scheduler.scheduleOnce function, which should bring proper context propagation for pattern.retry and pattern.after as well. Contributed by @ivantopo via #1135
• executors: #641 Avoid reflective access on the executors instrumentation. Contributed by @ivantopo via #1136

Fixes

• influxdb: #1129, #1131 There two bugs regarding encoding and configuration settings on the InfluxDB Authorization Token support introduced on Kamon Telemetry 2.4.8, those are gone now. Fixed by @duese via #1130
• redis: #1082 The Jedis instrumentation wasn't measuring the entire roundtrip time to Redis, only the time it took to send out the commands. Fixed by @ivantopo via #1117
• redis: #1072 Ensure the Akka instrumentation takes the Rediscala actors' exclusions into account. Reported by @jtjeferreira and fixed by @ivantopo via #1122
• akka:" #1116 Stop generating spans for Akka Streams-related actors. Fixed by @ivantopo via #1133
• akka: #1132 Fix module registration warnings for Akka Dispatchers. Fixed by @ivantopo via #1138

Breaking Changes 🚨

• We removed the Play gRPC-specific instrumentation that was included in kamon-play to the new kamon-akka-grpc hooks at a lower level that will work with both, coming with two important changes:
  • If you are not using the Kamon Bundle but instead adding dependencies one by one, you should add the new kamon-akka-grpc dependency for to your classpath
  • There is no custom operation name generator support in kamon-play anymore. All gRPC operations are now instrumented with the tag names defined by the OpenTelemetry semantic conventions and there shouldn't be any need to deviate from that with a custom generator
• The operation tag in Play gRPC requests will no longer have a / prefix, meaning that operations that previously were reported as /ReplyService/SayHello will now show up as ReplyService/SayHello. You might need to update your dashboards and alert conditions to account for that
• Kamon can no longer instrument Executors created by these helper functions from java.util.concurrent.Executors:
  • newSingleThreadExecutor
  • newSingleThreadScheduledExecutor
  • unconfigurableExecutorService
  • unconfigurableScheduledExecutorService

v2.4.8 - Token Authentication for InfluxDB

Starting on this release you can use API Token authentication when sending data to InfluxDB using this setting:

kamon.influxdb.authentication {
  token = "your-glorious-token-here"
}

Improvements

• influxdb. Introduced a new authentication.token configuration setting for cases where the InfluxDB reporter needs to authenticate with an API token. Contributed by @ivantopo via #1110.

v2.4.7 - Controlling the db.statement tag in JDBC calls

You can now use the kamon.instrumentation.jdbc.add-db-statement-as-span-tag setting to control the db.statement tag in JDBC calls. This is especially important when your JDBC calls might contain sensitive information that you don't want to share with third parties, like your tracing vendor. The possible setting values are:

• always: (default) will always add the db.statement tag to spans
• prepared: will only add the the db.statement tag when the traced execution was made with a PreparedStatement, which should ensure no placeholder values are leaked to the tracing backend
• never: completely disables adding the db.statement tag to spans

Improvements

• jdbc: control what kind of JDBC statements will be added to the db.statement tag in JDBC spans. Contributed by @ivantopo via #1103

v2.4.6 - Namespaced Environment Tags

Namespaced Environment Tags

Now you can add namespaced (i.e. tags with dots in their name) environment tags to the kamon.environment.tags settings in your configuration file.

kamon.environment.tags {
  container.id = "4444444444"
  k8s {
    namespace.name = "production"
    cluster {
      name = "strong-cluster"
    }
  } 
}

For example, the settings above in your application.conf file would add the container.id, k8s.namespace.name, and k8s.cluster.name tags to the environment. Most reporters will automatically add all of these tags to metrics and spans before they get sent out.

Improvements

• core: It is now possible to have namespaced tags inside the kamon.environment.tags configuration. Contributed by @ivantopo via #1101.