|
Note
|
This repository contains the guide documentation source. To view the guide in published form, view it on the Open Liberty website. |
Learn how to add manual instrumentation to collect custom spans in traces and custom metrics from microservices by using MicroProfile Telemetry and the Grafana stack.
Automatic instrumentation in MicroProfile Telemetry makes it easy to capture traces, metrics, and logs without changing your code. However, it might not capture all the details that you need. Often, you need to monitor specific operations, track custom events, or measure performance indicators unique to your application.
Manual instrumentation lets you add custom spans and metrics directly to your code. These custom signals are collected alongside the automatically captured data, allowing you to view and analyze everything together in the same observability backend.
In this guide, you’ll learn how to add manual instrumentation to your microservices by creating custom spans in traces and custom metrics to extend the default telemetry. You’ll use the Grafana Docker OpenTelemetry LGTM image, a preconfigured OpenTelemetry observability backend based on the Grafana stack.
This guide builds on the Enabling observability in microservices with traces, metrics, and logs using OpenTelemetry and Grafana guide. If you are not familiar with enabling automatic telemetry collection, it will be helpful to read that guide before you proceed.
The diagram shows multiple services, but for simplicity, this guide configures only the system and inventory services to demonstrate observability in a distributed environment.
Before you begin, ensure that Docker is installed and running on your system. For installation instructions, see the official Docker documentation.
Start a container from the grafana/otel-lgtm Docker image by running the following command:
docker run -d --name otel-lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm
You can monitor the container startup by viewing its logs:
docker logs otel-lgtm
It may take a minute for the container to start. After you see the following message, your observability stack is ready:
The OpenTelemetry collector and the Grafana LGTM stack are up and running.
When the container is running, you can access the Grafana dashboard at the http://localhost:3000 URL.
The finish directory in the root of this guide contains the finished application. Give it a try before you proceed.
To try out the application, go to the finish directory and run the following Maven goal to build the system service and deploy it to Open Liberty:
mvnw.cmd -pl system liberty:run./mvnw -pl system liberty:run./mvnw -pl system liberty:runNext, open another command-line session in the finish directory and run the following command to start the inventory service:
mvnw.cmd -pl inventory liberty:run./mvnw -pl inventory liberty:run./mvnw -pl inventory liberty:runAfter you see the following message in both command-line sessions, both of your services are ready:
The defaultServer server is ready to run a smarter planet.
When both services are running, navigate your browser to the http://localhost:9081/inventory/systems/localhost URL.
When you visit this endpoint, it sends an HTTP GET request to the inventory service. The inventory service then makes two outbound GET requests to the system service. One request goes to the /system/properties endpoint, and the other goes to the /health endpoint.
In addition, the inventory service automatically pings the system service every 30 seconds to refresh the health status of all systems in the inventory.
To view the traces, go to the http://localhost:3000 URL and follow these steps:
-
Open the Explore view from the left menu.
-
Select Tempo as the data source.
-
Set Query type to
Search. -
Click Run query at the upper-right corner to list recent traces.
-
Click a trace ID with the
GET /inventory/systems/{hostname}name that you generated. You see the following result:
The trace contains seven spans, five from the
inventoryservice and two from thesystemservice. Under Service & Operation, you see the spans in this trace. You can inspect each span by clicking it to reveal more detailed information, such as the times that a request was received and a response was sent. -
Expand the Node graph to see the relationship of spans between the
inventoryandsystemmicroservices.
To view the metrics in Grafana, go to Drilldown → Metrics from the left menu. In the Filters section, set service_name = inventory to view metrics for the inventory service only.
After you’re finished reviewing the application, stop the Open Liberty instances by pressing CTRL+C in the command-line sessions where you ran the system and inventory services. Alternatively, you can run the following goals from the finish directory in another command-line session:
mvnw.cmd -pl system liberty:stop
mvnw.cmd -pl inventory liberty:stop./mvnw -pl system liberty:stop
./mvnw -pl inventory liberty:stop./mvnw -pl system liberty:stop
./mvnw -pl inventory liberty:stopMicroProfile Telemetry automatically instruments Jakarta RESTful Web Services and MicroProfile REST clients. To trace extra operations, such as internal logic or calls to external systems like a database, you can add manual instrumentation to the source code.
Navigate to the start directory to begin.
system/server.xml
link:finish/system/src/main/liberty/config/server.xml[role=include]inventory/server.xml
link:finish/inventory/src/main/liberty/config/server.xml[role=include]system/bootstrap.properties
link:finish/system/src/main/liberty/config/bootstrap.properties[role=include]inventory/bootstrap.properties
link:finish/inventory/src/main/liberty/config/bootstrap.properties[role=include]The mpTelemetry feature, which enables MicroProfile Telemetry support in Open Liberty, is already enabled for both the system and inventory services. The OpenTelemetry SDK is also enabled by setting the otel.sdk.disabled property to false for both the system and inventory services, allowing telemetry data to be collected from each.
Start the services to begin collecting telemetry data.
When you run Open Liberty in dev mode, dev mode listens for file changes and automatically recompiles and deploys your updates whenever you save a new change. Run the following command to start the system service in dev mode:
mvnw.cmd -pl system liberty:dev./mvnw -pl system liberty:dev./mvnw -pl system liberty:devOpen another command-line session and run the following command to start the inventory service in dev mode:
mvnw.cmd -pl inventory liberty:dev./mvnw -pl inventory liberty:dev./mvnw -pl inventory liberty:devAfter you see the following message, your Liberty instance is ready in dev mode:
************************************************************** * Liberty is running in dev mode.
Dev mode holds your command-line session to listen for file changes. Open another command-line session to continue, or open the project in your editor.
Now, you can trace your Jakarta CDI beans by annotating a method with the @WithSpan annotation.
Replace theInventoryManagerclass.inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java
InventoryManager.java
link:staging/inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java[role=include]The list(), add(), and update() methods are annotated with the @WithSpan annotation. The list() method uses its method name as the default span name, while the add() and update() methods define custom span names, Inventory Manager Add and Inventory Manager Update, respectively. Each time one of these methods is called, the instrumentation automatically creates a new span.
To enrich spans with more context, you can annotate method parameters with the @SpanAttribute annotation. In this example, the host parameter in the add() and update() methods is labeled as hostname in the span.
You can now view the traces that are generated by the @WithSpan annotation.
Open the http://localhost:9081/inventory/systems URL in your browser to trigger a trace, then open the Grafana dashboard at the http://localhost:3000 URL. In the Explore view, select Tempo as the data source, set Query type to Search, and click Run query. Select a trace ID with the GET /inventory/systems name that you generated. You see the result as:
Verify that there are two spans from the inventory service. Expand each span to view its details. You see the InventoryManager.list span that is created by the @WithSpan annotation.
To check out the information generated by the @SpanAttribute annotation, visit the http://localhost:9081/inventory/systems/localhost URL, then rerun your Grafana query. Select a trace ID with the GET /inventory/systems/{hostname} name that you generated. You see the following result:
Verify that there are four spans from the inventory service and two spans from the system service. Expand the Inventory Manager Add span and its Span Attributes field. You can see the hostname attribute with the localhost value that is created by the @SpanAttribute annotation.
If you visit the http://localhost:9081/inventory/systems/localhost URL again, the trace shows the Inventory Manager Update span instead.
The MicroProfile Telemetry specification makes the underlying OpenTelemetry Tracer instance available. You can create spans manually by injecting the OpenTelemetry Tracer into your application.
Replace theInventoryResourceclass:inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java
InventoryResource.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[role=include]The OpenTelemetry Tracer bean is injected into the InventoryResource class to manually instrument your code for trace collection. Before the InventoryManager calls the system service to check its health status, it creates and starts a span that is named GettingProperties.
Each span must be ended by calling end(). To ensure that the span always completes, the call is placed inside a finally block.
Calling the makeCurrent() method sets the span as the current span. This method returns a Scope that must be closed to restore the previous context. A try-with-resources block is used so the scope closes automatically.
Calling the addEvent() method adds events to the span. In this example, the code adds one event when the properties are successfully received, and another when the request fails.
For more information about OpenTelemetry distributed tracing, see the OpenTelemetry distributed tracing API documentation.
To see these spans and events, visit the http://localhost:9081/inventory/systems/localhost URL and open the Grafana dashboard at the http://localhost:3000 URL. In the Explore view, select the Tempo data source and Run query. Select a trace ID with the GET /inventory/systems/{hostname} name that you generated. You see the GettingProperties span:
Expand the GettingProperties span and check the Events section to see the Received properties event:
To test a failure case, go to the http://localhost:9081/inventory/systems/unknown URL and run the same trace query in Grafana. Select a trace ID with the GET /inventory/systems/{hostname} name that you generated. The trace contains two spans from the inventory service, including the GettingProperties span. Expand this span and check the Events section to see the Cannot get properties event:
In addition to the runtime metrics automatically collected with MicroProfile Telemetry, you can define custom metrics in your application by using the OpenTelemetry API.
You can manually instrument your application to define custom metrics.
Replace the HealthCheckScheduler class.
inventory/src/main/java/io/openliberty/guides/inventory/HealthCheckScheduler.java
HealthCheckScheduler.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/HealthCheckScheduler.java[role=include]The OpenTelemetry Meter bean is injected into the HealthCheckScheduler class to define and collect application-specific metrics. This interface provides access to the OpenTelemetry metrics API, including instruments such as counters, histograms, and gauges to monitor application behavior and performance.
The Meter.counterBuilder() method defines a counter instrument named inventory.health.check.scheduler.runs. This metric records the total number of health check scheduler executions over time.
The Meter.histogramBuilder() method defines a histogram instrument named inventory.health.check.scheduler.duration. This metric measures the time taken for each scheduled health check that is run. The start time is captured when the performHealthChecks() method begins, and the duration is calculated and recorded when the run completes.
When naming metrics, OpenTelemetry’s general naming guidelines are followed to support compatibility across observability tools.
Replace the InventoryManager class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java
InventoryManager.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java[role=include]The OpenTelemetry Meter bean is injected and used to define a gauge instrument by using the Meter.gaugeBuilder() method. The gauge is named inventory.size and reports the current number of systems in the inventory. The buildWithCallback() method updates the gauge dynamically so that it always reflects the latest inventory size.
To learn more about how to define and use metrics in your application, refer to the Meter operations for a full list of available metrics, or the OpenTelemetry metrics API reference.
To view the custom metrics, first trigger some activity in the application by visiting the following URLs in your browser:
-
Visit the http://localhost:9081/inventory/systems/localhost URL to add your local system properties to the inventory.
-
Visit the http://localhost:9081/inventory/systems/127.0.0.1 URL to add the system properties for host
127.0.0.1to the inventory. -
Visit the http://localhost:9080/system/unhealthy URL to set both
localhostand host127.0.0.1health status toDOWN.
Next, open the Grafana dashboard at the http://localhost:3000 URL. From the left menu, navigate to Drilldown → Metrics, then and enter inventory in the Quick search metrics field to find metrics for the inventory service. In the upper-right corner, click Set auto refresh interval and choose 10s to see live updates.
OpenTelemetry metrics are exported to the collector every 60 seconds by default, so it may take up to a minute for new metrics to appear. If you don’t see them right away, wait for the next update cycle.
You see a result similar to this:
Note the difference between the metric name that you define and the name that is displayed in Grafana. For details on how OpenTelemetry metrics are transformed when exported to Prometheus, see the OTLP Metric points to Prometheus section of the OpenTelemetry specification.
Because the health check scheduler runs every 30 seconds, you can wait a few minutes and refresh the dashboard to watch the metrics change in real time.
Manually verify the telemetry signals by inspecting them in the Grafana dashboard. You can also run the included tests to check the basic functionality of the services. If any of the tests fail, you might have introduced a bug into the code.
Because you started Open Liberty in dev mode, you can run the tests for the system and inventory services by pressing the enter/return key from the command-line sessions where you started the services.
When you are done checking out the services, exit dev mode by pressing CTRL+C in the shell sessions where you ran the system and inventory services.
Finally, run the following command to stop the container that you started from the grafana/otel-lgtm image in the Additional prerequisites section.
docker stop otel-lgtm
You just used MicroProfile Telemetry in Open Liberty to go beyond automatic instrumentation by adding custom traces and metrics to your microservices.
Try out one of the related MicroProfile guides. These guides demonstrate more technologies that you can learn to expand on what you built in this guide.