@opentelemetry/instrumentation-aws-sdk doesn't work with NextJS

[alexandr2110pro]

What version of OpenTelemetry are you using?

    "@opentelemetry/api": "^1.9.0",
    "@opentelemetry/api-logs": "^0.57.2",
    "@opentelemetry/exporter-metrics-otlp-proto": "^0.57.2",
    "@opentelemetry/exporter-trace-otlp-proto": "^0.57.2",
    "@opentelemetry/instrumentation": "^0.57.2",
    "@opentelemetry/instrumentation-aws-sdk": "^0.49.1",
    "@opentelemetry/instrumentation-http": "^0.57.2",
    "@opentelemetry/propagator-aws-xray": "^1.26.2",
    "@opentelemetry/resources": "^1.30.1",
    "@opentelemetry/sdk-logs": "^0.57.2",
    "@opentelemetry/sdk-metrics": "^1.30.1",
    "@opentelemetry/sdk-node": "^0.57.2",
    "@opentelemetry/sdk-trace-node": "^1.30.1",
    "@opentelemetry/semantic-conventions": "^1.30.0",

What version of Node are you using?

23

What did you do?

instrumentation.node.ts

import { DiagConsoleLogger, DiagLogLevel, diag } from '@opentelemetry/api';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { AwsInstrumentation } from '@opentelemetry/instrumentation-aws-sdk';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { Resource } from '@opentelemetry/resources';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { NodeSDK } from '@opentelemetry/sdk-node';
import {
  AlwaysOnSampler,
  BatchSpanProcessor,
  ParentBasedSampler,
} from '@opentelemetry/sdk-trace-node';
import { FetchInstrumentation } from '@vercel/otel';

import { NextJSSampler } from '@com/shared-otel';

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ERROR);

const sdk = new NodeSDK({
  resource: new Resource({}),
  instrumentations: [
    new FetchInstrumentation(),
    new HttpInstrumentation(),
    new AwsInstrumentation({
      suppressInternalInstrumentation: true,
    }),
  ],
  spanProcessors: [
    new BatchSpanProcessor(new OTLPTraceExporter(), {
      exportTimeoutMillis: 15000,
    }),
  ],
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
  }),
  sampler: new ParentBasedSampler({
    root: new NextJSSampler({
      base: new AlwaysOnSampler(),
      // base: new TraceIdRatioBasedSampler(0.1),
    }),
  }),
});

sdk.start();

instrumentation.ts

export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    await import('./instrumentation.node');
  }
}

Just in case, NextJSSampler is

import type { Attributes } from '@opentelemetry/api';
import {
  AlwaysOnSampler,
  type Sampler,
  SamplingDecision,
  type SamplingResult,
} from '@opentelemetry/sdk-trace-node';
import {
  ATTR_HTTP_ROUTE,
  SEMATTRS_HTTP_TARGET,
} from '@opentelemetry/semantic-conventions';

export class NextJSSampler implements Sampler {
  static readonly DEFAULT_IGNORE_PATTERNS = [
    /^\/_next\//,
    /^\/images\//,
    // /\?_rsc=/,
    /^\/health/,
  ];

  private readonly baseSampler: Sampler;
  private readonly ignorePatterns: RegExp[];

  constructor(props: { base?: Sampler; ignorePatterns?: RegExp[] } = {}) {
    this.baseSampler = props.base ?? new AlwaysOnSampler();

    this.ignorePatterns =
      props.ignorePatterns || NextJSSampler.DEFAULT_IGNORE_PATTERNS;
  }

  shouldSample(...args: Parameters<Sampler['shouldSample']>): SamplingResult {
    const attributes: Attributes = args[4];

    // TODO: keep an eye on https://github.com/vercel/otel/issues/143#issue-2874289175
    const route =
      // this is not populated at the moment of execution of the sampler
      this.getNextRoute(attributes) ??
      // this is not populated at the moment of execution of the sampler
      this.getHttpRoute(attributes) ??
      // this is populated
      this.getHttpTarget(attributes);

    return this.isIgnoredRoute(route)
      ? { decision: SamplingDecision.NOT_RECORD }
      : this.baseSampler.shouldSample(...args);
  }

  private isIgnoredRoute(route?: string): boolean {
    if (!route) return false;
    return this.ignorePatterns.some(re => re.test(route));
  }

  private getNextRoute(attributes: Attributes): string | undefined {
    const value = attributes['next.route'];
    if (typeof value === 'string') return value;
    return undefined;
  }

  private getHttpRoute(attributes: Attributes): string | undefined {
    const value = attributes[ATTR_HTTP_ROUTE];
    if (typeof value === 'string') return value;
    return undefined;
  }

  private getHttpTarget(attributes: Attributes): string | undefined {
    // TODO: replace SEMATTRS_HTTP_TARGET when better alternative is available
    //       https://github.com/vercel/otel/issues/143#issuecomment-2678223912
    const value = attributes[SEMATTRS_HTTP_TARGET];
    if (typeof value === 'string') return value;
    return undefined;
  }

  toString(): string {
    return 'Next JS Sampler - Exclude Health Route';
  }
}

What did you expect to see?

SNS, SQS, and DynamoDB calls via AWS SDK v3 are traced alongside everything else.

What did you see instead?

SNS, SQS, and DynamoDB calls via AWS SDK v3 are not traced.
Http requests are traced, manual trace spans are recorded as expected, NexJS internally emitted spans are recorded as expected.

Additional context

I am on NextJS v15.1.7.
Tried both server actions and api route handlers for executing AWS related business logic on the backend - same result.

Tried with @vercel/otel - passing instrumentations array to registerOTel fn. Same result.

[alexandr2110pro]

I will create a similar issue at NextJS repo.

[pichlermarc]

cc @trivikr @jj22ee @blumamir (component owners)

[trentm]

Is bundling involved? If so, I'd expect undici and http instrumentation to work (instr of node core modules), but other instrumentations to not work.

[alexandr2110pro]

@trentm

Yes it is. NextJS, in general, requires a bundling.

We are using the @nx/next plugin to build it, and then we deploy it to ECS on Fargate.

Nx/next does this https://github.com/nrwl/nx/blob/master/packages/next/src/executors/build/build.impl.ts under the hood,
which runs next build with some tweaks, as you can see.

Our config for it is:

    "build": {
      "executor": "@nx/next:build",
      "inputs": [
        "default",
        "^production"
      ],
      "outputs": [
        "{options.outputPath}/.next",
        "{options.outputPath}/.next/!(cache)"
      ],
      "defaultConfiguration": "production",
      "configurations": {
        "development": {
          "outputPath": "."
        },
        "production": {
          "outputPath": "{workspaceRoot}/dist/com/sa/website",
          "generateLockfile": true
        }
      }
    }

and the tsconfig is

{
  "extends": "../../../tsconfig.base.json",
  "compilerOptions": {
    "jsx": "preserve",
    "strict": true,
    "noEmit": true,
    "emitDeclarationOnly": false,
    "esModuleInterop": true,
    "module": "esnext",
    "moduleResolution": "bundler",
    "resolveJsonModule": true,
    "isolatedModules": true,
    "lib": [
      "dom",
      "dom.iterable",
      "esnext"
    ],
    "allowJs": true,
    "allowSyntheticDefaultImports": true,
    "forceConsistentCasingInFileNames": true,
    "incremental": true,
    "plugins": [
      {
        "name": "next"
      }
    ]
  },
  "include": [
    "**/*.js",
    "**/*.jsx",
    "**/*.ts",
    "**/*.tsx",
    "../../../com/sa/website/.next/types/**/*.ts",
    "../../../dist/com/sa/website/.next/types/**/*.ts",
    ".next/types/**/*.ts",
    "next-env.d.ts",
    "../../../dist/com/web/app/.next/types/**/*.ts"
  ],
  "exclude": [
    "node_modules",
    "jest.config.ts",
    "**/*.spec.ts",
    "**/*.test.ts"
  ]
}

---

When you say it won't work, can you elaborate on why (what the root cause is specifically) and what can be done to fix it? I could try to contribute.

[alexandr2110pro]

Is bundling involved? If so, I'd expect undici and http instrumentation to work (instr of node core modules), but other instrumentations to not work.

Btw, are there many js apps that aren't bundled?
NestJS, Lambda, Express - all usually go through bundling process.
Most of projects are written in TS, then transpiled and bundled to minimize the size of the code and perform optimisations.

How can aws onstrumentation in otel js contrib not support bundling?

Let's fix it!

[pichlermarc]

@alexandr2110pro - most instrumentations work by hooking require or by modifying what's provided by import - since bundlers remove these statements, there's no place to hook and modify the code. We would have to inject code during bundling via some plugin to ensure that the library is patched.

People have started work on supporting this in the past, but we have not gotten around to doing this yet. The instrumentation base package is up for a major overhaul soon and we'll consider it in the new design. This is not a single-line fix, this will take a lot of effort and we need to get it right from the start - adoption of such features happens quite rapidly.

(Side note: some instrumentations work as the authors of the instrumented library provide diagnostics channel messages that we can use to generate telemetry, this does not require patching of the library and it's usually the cleaner approach)

Btw, are there many js apps that aren't bundled?

It's fairly common, but bundled apps are also common.

[alexandr2110pro]

@pichlermarc, thanks for the clarification.

Just FYI, I've tried to use serverExternalPackages, which supposedly should exclude packages from bundling.

I am not sure if they support wildcards, so I listed everything from my package.json related to otel and aws, just in case:

  serverExternalPackages: [
    '@opentelemetry/api',
    '@opentelemetry/api-logs',
    '@opentelemetry/core',
    '@opentelemetry/exporter-metrics-otlp-proto',
    '@opentelemetry/exporter-trace-otlp-proto',
    '@opentelemetry/id-generator-aws-xray',
    '@opentelemetry/instrumentation',
    '@opentelemetry/instrumentation-aws-sdk',
    '@opentelemetry/instrumentation-http',
    '@opentelemetry/propagator-aws-xray',
    '@opentelemetry/resource-detector-aws',
    '@opentelemetry/resources',
    '@opentelemetry/sdk-logs',
    '@opentelemetry/sdk-metrics',
    '@opentelemetry/sdk-node',
    '@opentelemetry/sdk-trace-node',
    '@opentelemetry/semantic-conventions',

    '@vercel/otel',

    '@aws-sdk/client-dynamodb',
    '@aws-sdk/client-sqs',
    '@aws-sdk/lib-dynamodb',
  ],

After that, I noticed that HTTP instrumentation started working (it shows span for calling aws apis), but not AWS.

---

I knew that instrumentations have to do with import/require patching (very briefly).
But we are bundling our lambda fns too (with esbuild, setting aws and otel packages as excluded from the bundling, too).
And with lambdas, aws instrumentation seems to work as expected with both ADOT and standard instrumentation lambda layers.

This is what puzzled me the most.

[alexandr2110pro]

@pichlermarc Another thing to consider - @aws/aws-distro-opentelemetry-node-autoinstrumentation/register does the thing. However I am not sure how to customize its behaviour since it doesn't export anything useful.

Here is the code:

// instrumentation.ts
export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    await import('./instrumentation.adot');
  }
}

and

// instrumentation.adot.ts

import '@aws/aws-distro-opentelemetry-node-autoinstrumentation/register';
// this package exports only one fn - `setAwsDefaultEnvironmentVariables`
// which doesn't help with customizing

With this, I see the expected instrumentation of AWS SDK.

But now I can't apply my custom sampler to ignore healthcheck-related traces:

import '@aws/aws-distro-opentelemetry-node-autoinstrumentation/register';

// No idea where to register this sampler
const sampler =  new ParentBasedSampler({
    root: new NextJSSampler({
      base: new AlwaysOnSampler(),
      // base: new TraceIdRatioBasedSampler(0.1),
    }),
});

Maybe somebody can look at @aws/aws-distro-opentelemetry-node-autoinstrumentation/register and see what exactly makes it work even in bundled apps?

Then, we can add a note to the README.MD to use it if you want to auto-instrument a bundled app with instrumentation that relies on require/import patching.

[alexandr2110pro]

I guess I will go on with my initial setup (but with the serverExternalPackages config option), embracing that I won't get AWS SDK auto-instrumented and instrument it manually. 🤷‍♂
For the time being...

Too bad. 😞

[dyladan]

I don't think this is a bug but more of a feature request that "instrumentations should work in non-require contexts like bundlers". This is a tricky issue that would likely require some changes in the underlying instrumentation package. Individual instrumentations may be able to work around this but a general solution is not trivial.

Related issue asking for a bundler plugin which autoinstruments at bundle time: open-telemetry/opentelemetry-js#4174

I'll create an issue in the main repo for non-require instrumentation support and link it back here.