feat: integrate OTEL/Jaeger #3815

hthieu1110 · 2025-02-24T14:25:15Z

Implementation for #2434

We can filter by BlockHeight

We can trace the calls

Gno2D2 · 2025-02-24T14:26:16Z

🛠 PR Checks Summary

All Automated Checks passed. ✅

Manual Checks (for Reviewers):

IGNORE the bot requirements for this PR (force green CI check)

✅ Automated Checks (for Contributors):

🟢 Maintainers must be able to edit this pull request (more info)
🟢 Pending initial approval by a review team member, or review from tech-staff

☑️ Contributor Actions:

Fix any issues flagged by automated checks.
Follow the Contributor Checklist to ensure your PR is ready for review.
- Add new tests, or document why they are unnecessary.
- Provide clear examples/screenshots, if necessary.
- Update documentation, if required.
- Ensure no breaking changes, or include BREAKING CHANGE notes.
- Link related issues/PRs, where applicable.

☑️ Reviewer Actions:

Complete manual checks for the PR, including the guidelines and additional checks if applicable.

📚 Resources:

Debug

Automated Checks

Maintainers must be able to edit this pull request (more info)

If

🟢 Condition met
└── 🟢 And
    ├── 🟢 The base branch matches this pattern: ^master$
    └── 🟢 The pull request was created from a fork (head branch repo: hthieu1110/gno)

Then

🟢 Requirement satisfied
└── 🟢 Maintainer can modify this pull request

Pending initial approval by a review team member, or review from tech-staff

If

🟢 Condition met
└── 🟢 And
    ├── 🟢 The base branch matches this pattern: ^master$
    └── 🟢 Not (🔴 Pull request author is a member of the team: tech-staff)

Then

🟢 Requirement satisfied
└── 🟢 If
    ├── 🟢 Condition
    │   └── 🟢 Or
    │       ├── 🔴 At least 1 user(s) of the organization reviewed the pull request (with state "APPROVED")
    │       ├── 🟢 At least 1 user(s) of the team tech-staff reviewed pull request
    │       └── 🔴 This pull request is a draft
    └── 🟢 Then
        └── 🟢 And
            ├── 🟢 Not (🔴 This label is applied to pull request: review/triage-pending)
            └── 🟢 At least 1 user(s) of the team tech-staff reviewed pull request

Manual Checks

**IGNORE** the bot requirements for this PR (force green CI check)
If
🟢 Condition met
└── 🟢 On every pull request
Can be checked by

Any user with comment edit permission

…-jaeger

codecov · 2025-02-25T06:26:27Z

Codecov Report

Attention: Patch coverage is 45.04505% with 122 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
tm2/pkg/telemetry/traces/traces.go	0.00%	34 Missing ⚠️
tm2/pkg/telemetry/metrics/metrics.go	0.00%	21 Missing ⚠️
tm2/pkg/bft/consensus/state.go	81.73%	18 Missing and 1 partial ⚠️
tm2/pkg/telemetry/init.go	13.63%	19 Missing ⚠️
gno.land/cmd/gnoland/start.go	10.00%	14 Missing and 4 partials ⚠️
tm2/pkg/telemetry/config/global.go	25.00%	6 Missing ⚠️
tm2/pkg/telemetry/tracer.go	50.00%	4 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

…u1110/gno into feat/integrate-otel-jaeger

…-jaeger

hthieu1110 · 2025-04-22T02:13:07Z

hi @gfanton, I've updated the code with your suggested changes, could you take a look again pls :)
I've checked, normally all the code related to addSpan are already after the lock acquisition.

thehowl · 2025-05-15T08:14:57Z

Can you address the failing CI?

…-jaeger

…u1110/gno into feat/integrate-otel-jaeger

hthieu1110 · 2025-05-26T10:20:07Z

hiii @gfanton , sorry for my late reply. Seems that some tests are broken on master, I'm waiting a little bit for that to be fixed first.

zivkovicmilos

Maybe the consensus module is not the best place, for now, to showcase tracing.

What do you think about hooking up the keeper query handlers, or RPC handlers?

zivkovicmilos · 2025-05-26T13:32:52Z

tm2/pkg/telemetry/config/config.go

 	MeterName         string `json:"meter_name" toml:"meter_name"`
 	ServiceName       string `json:"service_name" toml:"service_name" comment:"in Prometheus this is transformed into the label 'exported_job'"`
 	ServiceInstanceID string `json:"service_instance_id" toml:"service_instance_id" comment:"the ID helps to distinguish instances of the same service that exist at the same time (e.g. instances of a horizontally scaled service), in Prometheus this is transformed into the label 'exported_instance"`
 	ExporterEndpoint  string `json:"exporter_endpoint" toml:"exporter_endpoint" comment:"the endpoint to export metrics to, like a local OpenTelemetry collector"`
+
+	TracesEnabled             bool `json:"traces_enabled" toml:"traces_enabled"`
+	GracefulShutdownTelemetry bool `json:"graceful_shutdown_telemetry" toml:"graceful_shutdown_telemetry"`


Why would we ever not want to gracefully shut down telemetry?

Originally there was no Graceful shutdown, so I added this for case where we need that. I think that we will need GracefulShutdown when we run it in production and does not need it when running locally or test. Wdyt ?

zivkovicmilos · 2025-05-26T13:39:25Z

tm2/pkg/telemetry/tracer.go

+	"go.opentelemetry.io/otel/trace/noop"
+)
+
+type TracerFactory func() trace.Tracer


You can drop this indirection

zivkovicmilos · 2025-05-26T13:40:52Z

tm2/pkg/telemetry/init.go

-	// Check if the metrics are enabled at all
-	if !c.MetricsEnabled {
-		return nil
+func Init(c config.Config, logger *slog.Logger) (*sdkTrace.TracerProvider, *sdkMetric.MeterProvider, error) {


Passing a logger just to log that something happened in the func is overkill

This function should also be split up into 2 different ones, one for the tracing, and one for the metrics -- because in the current state you still require the caller to check whether something is nil

zivkovicmilos · 2025-05-26T13:42:27Z

tm2/pkg/telemetry/traces/traces.go

+			return nil, fmt.Errorf("unable to create http traces exporter, %w", err)
+		}
+	default:
+		exp, err = otlptracegrpc.New(


We have a convention after talking to Jae, and that's that grpc fallbacks like this should be dropped entirely -- only support http / https

zivkovicmilos · 2025-05-26T13:43:01Z

tm2/pkg/telemetry/traces/traces.go

+		return nil, fmt.Errorf("error parsing tracer exporter endpoint: %s, %w", cfg.ExporterEndpoint, err)
+	}
+
+	// Use oltp metric exporter with http/https or grpc


zivkovicmilos · 2025-05-26T13:51:16Z