Observability: Server-side instrumentation

[chargome]

Hey there!

I'd like to open the discussion on instrumenting the React Router server runtime for emitting telemetry signals.

Current Problem

At Sentry, we've been running into difficulties instrumenting React Router Framework on the server-side since the switch from Remix. The problem boils down to this:

• With react-router v7 the server-runtime package has been moved to the core react-router package which is where our instrumentation needs to take place now.
• We cannot patch the createRequestHandler function anymore, details on this can be found in this issue.

Background on ESM instrumentation

Further, since React Router is emitting ESM (like many other frameworks), we and potentially other OpenTelementry SDKs rely on Node's register hooks for instrumenting server code. Unfortunately this comes with the downside that some deployment providers do not expose these options (e.g. Vercel or Netlify). In case anyone is interested this is well documented in this observability guide.

Two Potential Solutions for a more stable way forward

React Router OTEL support

• The framework could expose an instrumentation entrypoint for the server. This could likely be a standardized instrumentation.ts file which is guaranteed to run before any server code. An example for this approach would be Next.js.
• Building on top of this, the framework could natively support emitting OTEL spans from within the framework itself for crucial observability signals e.g. within createRequestHandler. This way 3rd parties could simply pick up these spans without having to patch any internals.

Diagnostics Channel

Leveraging the native Node diagnostics channel functionality, React Router could emit signals via this channel for requests, server-loaders, server-actions, etc. – which could then be picked up by third party observability providers. This feature is also runtime agnostic (e.g. Node, Bun, Deno, Cloudflare). We could even build and maintain an official OpenTelemetry instrumentation around this approach.

A good example for this approach would be fastify.

Impact

The React Router community would benefit from having more structured guidance on observability, as current resources on best practices in this area are still quite limited.

I would appreciate any thoughts on this topic and would also offer help contributing this feature to the framework once we identify a clear way forward!

[brophdawg11]

Have you seen the Events proposal? That is along the lines of what I've been envisioning to give devs client-side hooks to know when certain things are happening inside the router. This feels like the server-side equivalent of that?

I think ideally we'd end up with an API that is consistent on the server and client and agnostic of how it's gets consumed. So if we had a generic set of events that fired on both the server and the client, they could be transformed into OTEL on the server (via a userland/community adapter) and window.performance.mark/measure calls on the client for dev tools profiling.

Do you think a generic events approach could meet the needs here?

[chargome]

This looks pretty much like what we're looking for yes! Depending on the implementation details this would be great for observability providers like us if we can consume these events on the server.

Any rough estimates when you plan on starting to work on this? Would you be open for contributions on this topic?

[brophdawg11]

Yeah I'm interested in getting something like this added and so consider this a 👍 from me in our new feature process. I probably won't have time to work on this in the very near future so I'd love community help.

We need one more SC member to express interest before moving this to Stage 1 where we'll accept a PR but feel free to start a investigating in a fork and share back here until we get to an official PR. You should be able to test out your changes locally in playground/framework-express.

I'd love to start with something very simple that maybe just tracks the start/end of a request (on the server) and a navigation (on the client) through a generic JS event setup (not tied to OTEL or node diagnostics etc.).

// client
createBrowserRouter(routes, {
  onEvent(event) {
    console.log(event)
  },
});

// server
createRequestHandler({
  build,
  onEvent(event) {
    console.log(event)
  }  
});

Once we proof that out and align on an API and ensure it can be adapted for OREL and performance.mark/measure we can start expanding it deeper to track loaders, actions, redirects, errors, etc.

Another thing I was wondering is whether RR should be in charge of the event/trace ID generation process - that feels like something maybe better left to app code so it can use whatever the app uses elsewhere and/or prefix ids with a user id or something. Maybe we could expand the API to something like:

createBrowserRouter(routes, {
  events: {
    getId(request) {
      // app code to generate an id
    },
    onEvent(event) {
      // event.id would be populated using the above
      console.log(event)
    }
  },
});

[brookslybrand]

Seems reasonable to me! I also wouldn't be implementing, but I'd love to see us move forward with a stage 1 issue and open up to community help 👍

[brophdawg11]

cc: @rossipedia who I know has done a bunch with RR+OTEL so probably has some thoughts on this API too

[rossipedia]

👍 to onEvent from me. Although I'm wondering if it might be more idiomatic to pass in an EventTarget:

const events = new EventTarget();
events.addEventListener('navigate', (e) => {
  // ...
});

events.addEventListener('error', (e) => {
  // ...
});

// etc

createRequestHandler(routes, { events });

which would be consistent with relying on Standard Primitives.

In the browser it might make even more sense to make the router itself an EventTarget:

const router = createBrowserRouter(routes);

router.addEventListener(...);

<RouterProvider router={router} />

In framework mode, you could expose an events: EventEmitter prop to <HydratedRouter /> so you could set that up in your entry.client.tsx (that's where we're currently setting up client-side error monitoring, so IMO it makes sense to expose telemetry hooks from there).

Another thing I was wondering is whether RR should be in charge of the event/trace ID generation process - that feels like something maybe better left to app code so it can use whatever the app uses elsewhere and/or prefix ids with a user id or something.

100%. In the OTEL world at least, that's a concern of the Context Propagation utilities. When you enter into a new context, you either inherit the current trace ID or you get assigned a new one. Generally you already have one by the time the React Router handler gets invoked.

[brophdawg11]

yeah I'm on board with standard primitives. It would be nice if they all felt the same so maybe events: EventTarget can be used in all locations instead of switching between EventTarget and EventEmitter?

// data - client
createBrowserRouter(routes, { events })

// data - server
createStaticHandler(routes, { events })

// framework - server
createRequestHandler(routes, { events });

// framework - client
<HydratedRouter events={events} />

[rossipedia]

I believe that should be a viable approach. I think all of the major runtimes have built-in support for Event / EventTarget 🤔

[rossipedia]

I've started working on implementing a solution to this using the pattern outlined here, initial results look promising (Event even comes with high-ish resolution timestamps 💪).

Screen.Recording.2025-08-17.at.7.09.50.PM.mov

[rossipedia]

Oof. Yeah, forgot about that particular bit 🤔 and the browser doesn't even have proper context tracking facilities, so an isomorphic solution seems like it might be off the table.

The trick is designing this so that it can support OTEL, but not require it. the (data, next) => { ... } "middleware" style approach might be what we land on, but that makes implementation a lot less straightforward in the react-router codebase, as we'd have to wrap a lot of router.ts internals in next() functions that get handed off to the event dispatch.

[brophdawg11]

I'm not super familiar with OTEL so forgive any naive questions here, but who creates this "context" and what's the lifetime? Is it per request/navigation and then has child spans such as "1 action ran and 3 loaders ran"? Could we latch onto RR's existing request/navigation-scoped context somehow? It should be trivial to expose that to the event handlers

const otelContext = unstable_createContext<ReturnType<typeof createOtelContext>>();
const events = new EventTarget();

events.addEventListener('navigate', (e, reactRouterContext) => {
  // Create an OTEL context and store it in the RR context for this navigation
  reactRouterContext.set(otelContext, createOtelContext());
});

events.addEventListener('loader:start', (e, reactRouterContext) => {
  // Grab the OTEL context for this navigation
  let otelContext = reactRouterContext.get(otelContext);
});

Also, what is next doing here? I was assuming events were sort of fire and forget?

[rossipedia]

OTEL context is basically just an arbitrary storage area that OTEL uses for keeping track of execution spans. However, it's tightly managed by the OTEL implementation itself, and you can't really manage it by pushing/popping contexts off the stack manually.

It's implementation is much closer to AsyncLocalStorage.run() where you have to provide a context value and a callback, and that context value lives for the life of the callback and any async operations initiated by that callback (OTEL's default context manager in Node actually uses AsyncLocalStorage under the hood).

So in order to support OTEL interop, there'd have to be a kind of middleware-style API where the event handler was responsible for calling next(). An off the cuff handler could look something like this:

import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("my-app", "v1.0");
const events = new EventTarget();

events.addListener("navigate.start", async (e) => {
  const { next, data } = e.detail; // assuming RR dispatches `CustomEvent` derived event types
  tracer.startActiveSpan("router.navigate.start", async (span) => {
    try {
      span.setAttribute(
        "http.route",
        convertPathnameToRoutePattern(data.location.pathname),
      );
      // ... additional attributes...

      // Continue on
      await next();
    } catch (e) {
      if (e instanceof Error) {
        span.recordException(e);
      }
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: e?.message || String(e),
      });
    } finally {
      span.end();
    }
  });
});

However, the implementation there is tricky, because it forces providers of events to call next() or else continuation never occurs. Ideally you'd want a pattern where if the event handler never calls next() then RR would do it itself, kinda like how unstable_middleware's next() function is handled. However, in order to detect that, you'd have to ensure that next() is destructured/accessed off the event at the scope of the handler, because often if you want to instrument it, you need to embed that next() call in a nested function scope and you wouldn't be able to tell if it was called during the sync execution of the event handler.

Something like:

let nextAccessed = false;
const detail = {
  data: { location },
  get next() {
    nextAccessed = true;
    return continueFn;
  }
};

events.dispatchEvent(
  new RouterCustomEvent(
    'navigation.start', 
    { detail },
  ),
);

needless to say this starts looking super messy.

Whiiiiiiich is kinda why I'm leaning towards handling events separately from handling the trace touch points.

Events should be moments. ie: "This thing happened". They have definite time stamps, but no duration. Traces (in OTEL parlance), are hierarchical timespans of increasing granularity (like flame graphs). Trying to shoehorn one (traces) into the other (events) feels a bit like... well...

[brophdawg11]

I see - thanks for the explanation.

let nextAccessed = false;

You should have written middleware lol

I'm leaning towards handling events separately from handling the trace touch points.

Yeah I think this would make for a much cleaner implementation - do you have an idea how this might look? Any way we could we aggregate events into traces at the end of a request/navigation/fetch? If so, could potentially include all prior events for a given request/navigation/fetch in the final end event in something like e.details.events and OTEL could be fired then. Based on my quick reading through this seems like it might not be possible because it seems like the spans capture their timestamp info in real time?

[rossipedia]

Based on my quick reading through this seems like it might not be possible because it seems like the spans capture their timestamp info in real time?

Yeah, that's been my experience with OTEL so far. Technically you can create "independent" spans (docs), although TBH I haven't played around with enough yet to know if it's a viable approach. But I'm thinking probably not.

The primary challenge there is that because those spans don't become the "active" span, then you wouldn't get automatic nesting (ie: you'd lose the parent/child relationship), and none of the React Router spans would show as containing any of the child spans that user-land code might create, like in a loader or action or a SSR component render.

RR would also have to manage span context propagation, but that might be a solved problem as you mentioned since RR has a context propagation primitive already.

So for example if we did something like:

// Let's assume that server code (ie: express or whatever) has its own "request" span,
// and that the following events fire at the start and end of the staticHandler's `query`
// method (ie: wrapping route loaders)

events.addEventListener('staticHandler.query.start', ({ data }) => {
  // This span is a child of the "request" span
  const querySpan = tracer.startSpan("staticHandler.query");
  // Let's assume there's a RouterContextProvider attached
  // and TraceProvider is basically a Map<key, Span>
  data.context.get(TraceProvider).set("staticHandler.query", querySpan);
});

events.addEventListener('staticHandler.query.complete', ({ data }) => {
  // Mark the span as complete
  data.context.get(TraceProvider).get("staticHandler.query").end();
});

// Then somewhere in some loader you have some utility function or API client that creates its own spans:
return tracer.startActiveSpan('api.client.fetch', (span) => {
  try {
    // ... 
  } finally {
    span.end();
  }
});

The problem here is that your trace would look something like this:

request                    |----------------------------------- |
  staticHandler.query        |---------------------|
  api.client.fetch                        |----|
  render                                               |------|

Whis isn't quite right, since it indicates that staticHandler.query and api.client.fetch are happening in parallel, rather than staticHandler.query containing api.client.fetch.

Thinking about it some more, I think really there's no way around providing some kind of onInstrument or similar handler that's basically:

createStaticHandler(routes, {
  onInstrument(identifier, data, next) {
    // instrument next() however suits your needs
  }
});

The caveat here is that it would need to be documented that next() must be called, so it'd probably need some big warning labels in the docs.

With that approach, an OTEL trace implementation could look something like this at its simplest:

createStaticHandler(routes, {
  onInstrument(identifier, data, next) {
    try {
      return await tracer.startActiveSpan(`react-router.${identifier}`, (span) => {
        span.addAttributes(data);
        return await next();
      });
    } catch (e) {
      if (e instanceof Error) {
        span.recordException(e);
      }
      span.setStatus(SpanStatus.ERROR, e.message || String(e));
      throw e;
    } finally {
      span.end();
    }
  }
});

React Router would own the definitions for identifier and data, but instead of dictating how those things are used, it's up to the developer to consume them and turn them into usable observability artifacts.

Note

This is just for the tracing aspect. I still think that the events / EventTarget solution would complement this approach.
events would be: "This thing happened", onInstrument (or whatever it ends up being called) would be: "This logical unit of execution is about to happen"

You'd have to ensure that nothing that happens in onInstrument would be able to alter the behavior of RR itself though, just observe. The data param would be purely informational.

Would love to know what you think.

[brophdawg11]

Related: blog post on SvelteKit's OTEL implementation: https://svelte.dev/blog/sveltekit-integrated-observability

[brophdawg11]

I spent a chunk of time yesterday and today digging into this and familiarizing myself with the Node OTEL flows and APIs.

tl;dr; Check out https://github.com/brophdawg11/rr-otel-playground

1. OTEL via Events

The first thing I did was stick a few simple events in the underlying RR static handler to see if it would be possible to listen to those events and reconstruct nested OTEL spans (see discussion around this in #13749 (comment)). This approach seemed promising and we could create spans from a navigate:start event (source):

  events.addEventListener("navigate:start", (e) => {
    let evt = e as CustomEvent;
    console.log("✨ navigate:start event", evt.detail.request.url);

    let tracer = otel.trace.getTracer("react-router");
    let rrContext = evt.detail.context as RouterContextProvider;
    let navigateSpan = tracer.startSpan("navigate");
    let navigateContext = otel.trace.setSpan(otel.context.active(), navigateSpan);
    rrContext.set(otelContext, {
      tracer: tracer,
      navigateContext,
      navigateSpan,
    });
  });

But the downside here is that this breaks automatic tracing of startActiveSpan so you need to set that up properly in your loader so that it works correctly in nested function invocations. This can be done automatically via server build instrumentation though which eliminates the need for any special code in user loaders/actions.

2. Middleware to the Rescue?

The main issue with the "OTEL via Events" idea is that events are point in time and don't provide any way to get automatic nested context tracking via startActiveSpan. It seems possible to reconstruct that via events and some instrumentation but it still felt a bit hacky. Then I realized that we already have a way to wrap contextual logic from the top of the tree in middleware. If we add a root middleware that calls startActiveSpan around next, then all downstream middlewares, loaders, and actions will execute inside of that new span context and startActiveSpan should "just work" inside of them.

// app/root.tsx
export const tracer = otel.trace.getTracer("react-router");

const otelMiddleware: MiddlewareFunction<Response> = async (_, next) => {
  return tracer.startActiveSpan("request", async (span) => {
    try {
      return await next();
    } finally {
      span.end();
    }
  });
};

export const middleware = [otelMiddleware, /* .. other middleware ... */;

Then, anything downstream can stick in a tracer.startActiveSpan call and it will show up nested under the proper spot. With server build instrumentation, we can automatically add this middleware to the root loader, and also automatically add spans for all middlewares, loaders, actions, and event the entry.server default export. This could be wrapped up in react-router-otel package (or potentially added to core but I think I would prefer a community solution here since we're not OTEL experts).

This method feels better than events since it can be done in production today without any changes to React Router.

Here's an example trace of a route from the playground in https://github.com/brophdawg11/rr-otel-playground:

[chargome]

Thanks for looking into this! Will take a look this week, but staying away from building the instrumentation around events/diagc sounds promising 👍

[brophdawg11]

👍 If we go this route, then we'll also want some form of API to instrument the build when using @react-router/serve if we don't want this to be limited to folks using their own server. We'll also need a way to do something similar on the client - can probably instrument the router but would need to expose a way to do that from entry.client (either a prop to HydratedRouter, or a utility that operates on the internal/private router)

[rossipedia]

FWIW I'm putting together a vite plugin that will do some AST wrangling on the generated React Router server bundle to add OTEL spans in the various server bits (loaders / server entry / root middleware). The approach can probably be adapted to other non-OTEL instrumentation packages, but as @brophdawg11 mentioned, this can all be done today with zero updates to react-router core.

[chargome]

@rossipedia we talked about this yesterday internally – might be worth looking into https://github.com/apm-js-collab/orchestrion-js for doing exactly that

[rossipedia]

:o TIL about TracingChannel. That looks super compelling

[brophdawg11]

I know this proposal is about server side instrumentation, but since we're considering doing this all via wrapping the server build, I wanted to also put together an example of the same idea client side: https://stackblitz.com/edit/github-9tmz6ueq?file=src%2Fmain.tsx. If you record in the Performance tab you'll see the performnance measurements:

Supporting this in Framework mode would likely require new entry.client or HydratedRouter APIs.

[brophdawg11]

ok folks, after a good bit of experimentation and internal discussions, we've landed on an instrumentations API we're pretty happy with and ready to open it up for some alpha testing! This should make it fairly easy to layer on any type of instrumentation, be it console logging, OTEL, window.performance.measure, etc.

We'd love to get any early feedback from folks who want to kick the tires a bit!

• Draft PR: Add unstable_instrumentations API #14412
• Decision doc: https://github.com/remix-run/react-router/blob/brophdawg11/instrument/decisions/0015-observability.md
• Alpha release: 0.0.0-experimental-e0b6ce9
• Demo repo: https://github.com/brophdawg11/rr-otel-playground

[brophdawg11]

Check out the unstable_pattern argument passed alongside the request - that's the uninterpolated path like you're doing in the first example above. We don't have great docs on it yet since it's all still unstable - I think it's only exposed via some of the examples currently: https://reactrouter.com/how-to/instrumentation#opentelemetry-integration

[brettburley]

Ah yeah that's exactly what I was looking for! I do see it in the client-side hooks for the route-level instrumentation, but not with the router operations (e.g. router.instrument { navigate() }). I imagine it's harder to get that information there since that gets invoked before the route has been discovered.

[brettburley]

Trying out returning unstable_pattern and params in brettburley@9c97d94 so that they're usable in the top-level navigate instrumentation. These are useful in tandem with something like datadog RUM:

import { datadogRum } from "@datadog/browser-rum";

const datadogInstrumentation: unstable_ClientInstrumentation = {
  router(instrumentableRouter) {
    instrumentableRouter.instrument({
      navigate: async (callHandler, info) => {
        // start page view to get full timing information
        datadogRum.startView({ name: String(info.to) });

        const result = await callHandler();

        if (result.status === "success") {
          datadogRum.setViewName(result.unstable_pattern);
        }
      },
    });
  },
};

[brophdawg11]

since that gets invoked before the route has been discovered

That and before any matching has been done. The server-side request instrumentation and the client-side navigate instrumentation are intended to capture everything include route matching so by definition they can't provide a pattern for the route you are navigating to.

In your linked branch, you're using router.state.matches which represents the route being navigated away from so that's not accurate with the request/params that represent the destination.

If you want to track a full navigation duration excluding matching so that unstable_pattern is available, a root middleware is the best bet.

[rossipedia]

Update: We discussed this topic again in yesterday's steering committee call and we're going to explore returning relevant information in a .meta property on the result returned from the instrumented handlers, with one of those pieces of information being the unstable_pattern that was navigated to 🥳

[brophdawg11]

Starting a comment here to discuss the issue raised in #14832 regarding whether we can also expose throw data() calls as errors. cc @froston

[froston]

I’m not advocating changing loader/action error patterns, just making the instrumentation API more informative. Would it be possible for the callback to also return an http status (when known)?

[brophdawg11]

@froston What level of granularity do you need to properly report the trace - do you just need to know that it threw? Or the status code? Or the actual thing that was thrown? Maybe we could just expose the status code from any return/throw data() in a loader/action?

[froston]

@brophdawg11
Exposing the status code would already cover most needs for me. 🙂
Being able to distinguish between 4xx and 5xx is the main thing I’m after.

[rossipedia]

IMO it would be nice to have HTTP status codes in the handler instrumentation, and possibly an error (frozen) representing the thing that was thrown.

It's been rubbing me the wrong way that we have all these instrumentation hooks (which are awesome), but I still need a root middleware in order to grab the overall http status code 😅