rpc: add OpenTelemetry tracing for JSON-RPC calls by jrhea · Pull Request #33452 · ethereum/go-ethereum

jrhea · 2025-12-18T20:50:32Z

Summary

This PR adds tracing inside the RPC handler to help attribute runtime costs within handler.handleCall(). In particular, it allows us to distinguish time spent decoding arguments, invoking methods via reflection, actually executing the method and constructing/encoding JSON responses.

Concretely, the tracing breaks down execution along the following path:

handleCall()
|- parsePositionalArguments  // argument decode
|- runMethod
|  |- call                   // reflection + method invocation
|  |  |- engineAPI.method
|  |- msg.response           // response construction / JSON encode

Tracing is disabled by default by relying on the global OpenTelemetry tracer provider, which is a noop unless configured elsewhere. Server.SetTracerProvider is supported for tests, but we might want to ditch this and just rely on helpers accessing the global provider.

Follow-ups

Testing

If you want to get a feel for how this will work end to end, then checkout this PR on my geth fork: jrhea#1

jrhea · 2025-12-19T23:00:48Z

@fjl I am getting this error when the lint step is run:

https://github.com/ethereum/go-ethereum/actions/runs/20384597266/job/58582793251?pr=33452

looks like the otel dependencies I added to geth changed some transitive dependencies and now they don't match what is in cmd/keeper.

I can get it to pass by running go mod tidy in cmd/keeper and checking in cmd/keeper/go.mod and cmd/keeper/go.sum, but I just wanted to verify with you that this is normal and expected.

…ding overhead

- ensure runMethod() doesn't record spans if parentSpan isn't recording - ensure unsubscribe() isn't recorded - add tests to verify that subscribe/unsubscribe don't record

jrhea · 2026-01-03T00:14:14Z

Here are some screenshots of what it looks like in SigNoz:

…nsa function to end the span and set its status

jrhea · 2026-01-05T20:16:19Z

rpc/handler.go

+// startSpan starts a tracing span for an RPC call and returns a function to
+// end the span. The function will record errors and set span status based on
+// the error value.
+func (h *handler) startSpan(ctx context.Context, msg *jsonrpcMessage, spanName string) (context.Context, func(*error)) {


I added an almost identical method to the engine api in #33521. if I add a generic way to pass in attributes, then this could be packaged in a helper library and we could potentially avoid adding otel imports everywhere. This could also make it easier to swap implementations later.

If this is desirable, then I just need to know where I should put it so it is conveniently accessible from all other packages.

I think making it generic would be a great improvement. I think I would just create a new package for it. /otel or internal/otel or internal/tracing

I think making it generic would be a great improvement. I think I would just create a new package for it. /otel or internal/otel or internal/tracing

This is helpful, thanks @MariusVanDerWijden. On naming, calling this internal/otel ties it to a specific implementation. We may want to avoid that in case the underlying tracing backend changes in the future (e.g. similar to the OpenCensus -> OpenTelemetry migration).

internal/tracing seems like a good choice, but i was concerned that it would be confusing with other types of tracing that already exists in the codebase. I was talking to @lightclient about this and he had a similar concern. What do you think about internal/telemetry? I might start with that and see if anyone objects.

@MariusVanDerWijden I pushed up the generic change we were discussing. Lmk, if you have any feedback. 🙏

rpc/server.go

rpc/handler.go

MariusVanDerWijden · 2026-01-07T11:10:51Z

I like the new package!

lightclient

This is starting to come together nicely. There are more standardized conventions than I realized for JSON-RPC servers. Since we aren't exporting OpenTelemetry to the RPC users yet, it isn't quite as important, but I think we can follow the conventions fairly easily here.

internal/telemetry/telemetry.go

rpc/handler.go

jrhea · 2026-01-07T18:24:30Z

@lightclient here’s my take on the convention we should follow:

SpanKind

Two kinds are relevant for geth here: SERVER and INTERNAL.
Spans created at the JSON-RPC boundary are SpanKind=SERVER.
Spans created within implementation code are SpanKind=INTERNAL.

Attributes

For SpanKind=SERVER, follow the JSON-RPC semconv that you linked to:
- rpc.system="jsonrpc"
- rpc.service="engine"
- rpc.method="newPayloadV4"
- rpc.request_id="1"
For SpanKind=INTERNAL, attributes are domain-specific and up to the implementor (e.g. tx count, block hash, block number for engine payload processing).

Span name

For SpanKind=SERVER, follow the recommended $package.$service/$method pattern that you linked to. For JSON-RPC we can treat rpc.system as the package component. Example for engine_newPayloadV4: jsonrpc.engine/newPayloadV4
For SpanKind=INTERNAL, we should be flexible, but here are some rules of thumb:
- If a span covers the whole method, name it the same as the package.method (e.g. catalyst.newPayloadV4)
- For smaller spans around blocks of logic, name them descriptively (implementation-defined), and we can bikeshed as needed in review.

…RNAL spans

internal/telemetry/telemetry.go

rpc/server.go

internal/telemetry/telemetry.go

lightclient

Looks good to me. Would like to get a quick check from @fjl to make sure this is along the lines of what he was thinking, but seems ready to merge.

jrhea mentioned this pull request Dec 18, 2025

eth/catalyst: add initial OpenTelemetry tracing for Engine API newPayload #33441

Closed

jrhea force-pushed the rpc-otel-tracing branch 2 times, most recently from 7c245f1 to c35e582 Compare December 19, 2025 22:51

jrhea mentioned this pull request Dec 23, 2025

cmd/geth, internal/telemetry: wire OpenTelemetry tracing via CLI flags #33484

Merged

This comment was marked as off-topic.

Sign in to view

jrhea force-pushed the rpc-otel-tracing branch 2 times, most recently from 04615b8 to fa8860d Compare December 29, 2025 20:59

jrhea marked this pull request as ready for review January 2, 2026 13:18

jrhea requested a review from fjl as a code owner January 2, 2026 13:18

jrhea added 6 commits January 2, 2026 07:57

rpc: add generic OpenTelemetry tracing for JSON-RPC calls

1eb79f0

cmd/keeper: revert accidental go.mod/go.sum changes

7b7ab1d

cleanup comments

8254381

rpc: use global OpenTelemetry tracer provider by default.

479a2f8

address review feedback

d4ba577

add fast path to startSpan().

ecf30d9

jrhea force-pushed the rpc-otel-tracing branch from fa8860d to ecf30d9 Compare January 2, 2026 13:58

jrhea added 2 commits January 2, 2026 09:51

add spans for parsing args and calling method

6229224

add an additional span inside runMethod to capture JSON response enco…

5961c20

…ding overhead

jrhea changed the title ~~rpc: add generic OpenTelemetry tracing for JSON-RPC calls~~ rpc: add OpenTelemetry tracing for JSON-RPC calls Jan 2, 2026

jrhea mentioned this pull request Jan 2, 2026

Test otel jrhea/go-ethereum#1

Closed

- ensure span.End() is called on error

afb4f91

- ensure runMethod() doesn't record spans if parentSpan isn't recording - ensure unsubscribe() isn't recorded - add tests to verify that subscribe/unsubscribe don't record

jrhea mentioned this pull request Jan 3, 2026

eth/catalyst: add initial OpenTelemetry tracing for newPayload #33521

Merged

jrhea added 4 commits January 4, 2026 17:36

consolidate span creation and cleanup into startSpan, which now retur…

4cd89d4

…nsa function to end the span and set its status

removed isBatch bc isn't useful for initial usecase

2ae8b7d

rename done to spanEnd

70eda41

fix issue where span status wasn't properly passed to parents

c50d3e6

jrhea commented Jan 5, 2026

View reviewed changes

rpc/server.go Outdated Show resolved Hide resolved

lightclient reviewed Jan 5, 2026

View reviewed changes

rpc/handler.go Outdated Show resolved Hide resolved

rpc/handler.go Outdated Show resolved Hide resolved

jrhea added 2 commits January 6, 2026 12:08

remove code that propagates errors to parent spans. improve comments

57a536e

moved otel logic to internal/telemetry and simplified logic

60df872

jrhea force-pushed the rpc-otel-tracing branch 2 times, most recently from 0cce02e to c0235ed Compare January 6, 2026 23:32

add additional comments and license header

d2372c0

jrhea force-pushed the rpc-otel-tracing branch from c0235ed to d2372c0 Compare January 6, 2026 23:43

lightclient reviewed Jan 7, 2026

View reviewed changes

internal/telemetry/telemetry.go Outdated Show resolved Hide resolved

internal/telemetry/telemetry.go Outdated Show resolved Hide resolved

rpc/handler.go Outdated Show resolved Hide resolved

rpc/handler.go Outdated Show resolved Hide resolved

jrhea force-pushed the rpc-otel-tracing branch from 6ecfbe1 to 1b49f73 Compare January 7, 2026 21:59

reworked the telemetry package into helpers for SERVER spans and INTE…

bde2766

…RNAL spans

jrhea force-pushed the rpc-otel-tracing branch from 1b49f73 to bde2766 Compare January 7, 2026 22:05

lightclient reviewed Jan 7, 2026

View reviewed changes

internal/telemetry/telemetry.go Outdated Show resolved Hide resolved

rpc/server.go Outdated Show resolved Hide resolved

lightclient reviewed Jan 7, 2026

View reviewed changes

internal/telemetry/telemetry.go Outdated Show resolved Hide resolved

jrhea and others added 5 commits January 7, 2026 18:33

implement review feedback from lightclient

f6289f1

removed fast path in startSpan

0897f08

internal/telemetry: some styling changes

6e6bd76

internal/telemetry: use correct span kind for server

a91d7ad

StartServerSpan had the wrong spankind

cd9665f

jrhea force-pushed the rpc-otel-tracing branch from 2cf2efc to cd9665f Compare January 8, 2026 16:37

jrhea added 2 commits January 8, 2026 11:13

added back rpc.batch attribute + test

85d0220

update comments in tests

f90668f

lightclient approved these changes Jan 8, 2026

View reviewed changes

lightclient merged commit a9acb3f into ethereum:master Jan 14, 2026
7 of 8 checks passed

lightclient added this to the 1.17.0 milestone Jan 14, 2026

Conversation

jrhea commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Follow-ups

Testing

Uh oh!

jrhea commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as off-topic.

Uh oh!

jrhea commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jrhea Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrhea Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

MariusVanDerWijden Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrhea Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrhea Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MariusVanDerWijden commented Jan 7, 2026

Uh oh!

lightclient left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jrhea commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightclient left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jrhea commented Dec 18, 2025 •

edited

Loading

jrhea commented Dec 19, 2025 •

edited

Loading

jrhea commented Jan 3, 2026 •

edited

Loading

jrhea Jan 5, 2026 •

edited

Loading

MariusVanDerWijden Jan 6, 2026 •

edited

Loading

jrhea Jan 6, 2026 •

edited

Loading

jrhea commented Jan 7, 2026 •

edited

Loading