Expose the `asyncLocalStorage` for context propagation in tracing channels

[logaretm]

We've been testing using tracing channels for some new instrumentations we are working on at Sentry, at first I thought the context propagation wasn't working and tried some of the stuff discussed in #1263 (comment)

However it does work with bindStore bound to the OTEL local storage and a transform function that returns the OTEL context.

// Get OTel's AsyncLocalStorage for bindStore
const otelStorage = (context as any)._getContextManager()._asyncLocalStorage;

if (otelStorage) {
  channel.start.bindStore(otelStorage, (data) => {
    const span = Sentry.startSpanManual(
      {
        // attrs...
      },
      (span) => span
    );

    data.span = span;

    // Return the context to store in AsyncLocalStorage
    return trace.setSpan(context.active(), span);
  });
}

channel.subscribe({
  // do stuff and handle span end/error
});

The problematic part here is using private APIs to grab the AsyncLocalStorage

const otelStorage = (context as any)._getContextManager()._asyncLocalStorage;

We think it would be great to offer a public API that let's us grab it for use-cases like that, I'm proposing either a:

• A getter

import { context, trace } from '@opentelemetry/api';

// or `__dangerouslyXXX` if we are worried about abuse
const otelStorage = context.getAsyncLocalStorage();

• Or a function that preps a tracing channel for context propagation:

function tracingChannelWithOtelContext(
  channel: TracingChannel,
  transform: (data: any) => Span
) {
  // Get OTel's AsyncLocalStorage for bindStore
  const otelStorage = 'Get the storage internally somehow...';

  channel.start.bindStore(otelStorage, (data) => {
    const span = transform(data);

    return trace.setSpan(context.active(), span);
  });
  return channel;
}

const channel = tracingChannelWithOtelContext(
  tracingChannel('unjs.unstorage'),
  (data) => {
    const span = Sentry.startSpanManual(
      {
        // attrs...
      },
      (span) => span
    );

    return span;
  }
);

I'm not aware of the implications of exposing this, so I think the public function approach is a little better since it keeps the private internals private, and allows SDKs to use tracing channels. At the same time I think exposing it is helpful for other usecases around AsyncLocalStorage.

[logaretm]

We think it might be beneficial to do both suggestions actually, having access to the internal AsyncLocalStorage is beneficial in other cases like in getsentry/sentry-javascript#18040.

Also I put together a package as an example of what a tracing channel with an OTEL context propagation would look like, this can be added to the OTEL API since it can be used in SDK/Provider implementations and is not required on 3rd-party side.

[pichlermarc]

Hi @logaretm 🙂

This looks interesting, I think understand the underlying issue this is trying to solve (looks like it's somewhat similar to what context.bind() solves for event handlers/functions), but I feel like I have some gaps in understanding how this would be used with OTel (also: since my background is mostly metrics, I'd love for other @open-telemetry/javascript-maintainers to chime in here).

Would you mind creating an example how all involved actors may use this end-to-end?

I suppose the idea of how this would be used is:

• author of the instrumented library uses plain tracing channels from node:diagnostics_channel, emits events for their operations
• author of an OTel instrumentation library calls the code proposed here that ensures AsyncLocalStorage is bound to the channel, creates a span, consumes these later on
• any other third-party authors also listen to node:diagnostics_channel to do whatever they need to do

What I'm wondering is: Would it be possible to pack the channel.bind() operation into some form of context.bind() so that we don't have to expose the AsyncLocalStorage to end users at all? My concerns are mostly abuse + not all ContextManagers are AsyncLocalStorageContextManagers. Adding non-specified APIs is difficult. Maybe we could do something like:

context.bind(deferredContext, channel) // deferredContext is fictional right now, we may discard it in the AsyncLocalStorageContextManager and just bind the storage directly.

The proposed channel creation function could then live in @opentelemetry/instrumentation as an instrumentation utility.

[logaretm]

Hey @pichlermarc, thanks for replying. you have the correct idea about the flow of this.

Libraries

Here is a PR as an example of a library that wants to be observable-friendly. Think of db0 as a library we would like to instrument, but with tracing channels rather than monkey patching.

It boils down to the library calling channel.tracePromise or traceSync:

import { tracingChannel } from "node:diagnostics_channel";

const channel = tracingChannel('some:op')

// Later on
channel.tracePromise(() => storage.hasItem(key, opts), {
  // context data
});

As you can see the library doesn't need to cater for OTEL or any other vendor, they just need to emit connected events via the diagnostic channels. More specifically "tracing channels".

SDKs/Providers

Any Instrumentation or really anyone interested in those diagnostic events can listen and do whatever is needed.

But from an OTEL Instrumentation library, it needs to bind the start channel to the OTEL context so it can be propagated through the async execution.

I would imagine an API that "preps" a tracing channel for context propagation can be useful here, like the one you suggested. For example this example library I wrote does that under the hood by accepting an OTEL span and it goes from there.

I don't have a PR for usage yet, as we are still pending how to grab this async local storage instance. But in my POC, the following is working:

// Binds otel ALS under the hood if a span is returned in the callback
import { tracingChannel } from 'otel-tracing-channel';

const queryChannel = tracingChannel<Db0TraceContext>(`db0.query`, (data) => {
  const span = Sentry.startSpanManual(
    {
      name: `db0 ${data.query}`,
      attributes: {
        'db.query.text': data.query,
        [SEMANTIC_ATTRIBUTE_SENTRY_OP]: 'db.query',
      },
    },
    (span) => span
  );

  Sentry.addBreadcrumb({
    category: 'query',
    message: data.query,
    data: {
      'db.query.text': data.query,
    },
  });

  return span;
});

From there, the instrumentation library just need to listen for the channel events (start, asyncStart, end, asyncEnd, error) and do what is needed.

// Subscribe to events (span already created in bindStore)
queryChannel.subscribe({
  asyncEnd: (data) => {
      data.span?.setStatus({ code: SPAN_STATUS_OK });
      data.span?.end();
  },
  error: (data) => {
    Sentry.captureException(data.error);
    data.span?.setStatus({ code: SPAN_STATUS_ERROR });
    data.span?.end();
  },
});

In my example we rely on a span being created first in the start transform call.

I think a simple wrapper around tracingChannel like this works well, but really anything that handles returning the context in the start channel would work.

[logaretm]

Hey! I built a complete runnable demo showing exactly this use case with Tracing Channels + OTEL:

https://github.com/logaretm/roll-dice-tracing-channels-demo

It demonstrates:

• Library (dice-roll) emitting tracing channel events (zero OTEL deps)
• The onChannelStartSpan() helper for context propagation (imaginary future API that would be provided by OTEL)
• Custom OTEL instrumentation for dice-roll extending InstrumentationBase
• A user app with their own listeners (3rd party) for logging the dice rolls using tracing channels as well.

The only blocker is accessing _asyncLocalStorage from the context manager. I will try to think of proposals for such an API and if it can work in other ways.

[pichlermarc]

thanks @logaretm - this is a perfect showcase of how things could work.
I'll look into how we can make that fit over the next week and will summarize my findings and feedback here on this issue.