Description
What version of OpenTelemetry are you using?
Latest version (instrumentation-aws-sdk: v0.49.1)
What version of Node are you using?
v20.10.0
What did you do?
Use AWS SDK v3 Client to call an AWS service (like API GW), while doing either:
-
Set Env Var
OTEL_NODE_DISABLED_INSTRUMENTATIONS="http" -
Set
suppressInternalInstrumentation: truefor aws-sdk instrumentation ininstrumentationConfigs
const instrumentationConfigs: InstrumentationConfigMap = {
'@opentelemetry/instrumentation-aws-sdk': {
suppressInternalInstrumentation: true,
}
};
const instrumentations: Instrumentation[] = getNodeAutoInstrumentations(instrumentationConfigs);
What did you expect to see?
AWS SDK Instrumentation should be able to propagate X-Ray Context, via propagator-aws-xray, when http instrumentation is disabled or suppressed. Use cases include using AWS SDK to call S3 buckets, API GW, Lambda, etc..
What did you see instead?
AWS SDK Instrumentation does not propagate X-Ray Context when making calls to S3, API GW, Lambda, etc..
Additional context
See Specification here that clarifies that AWS SDK instrumentations should use X-Ray propagator specifically.
Here is how this is currently done in the AWS SDK Instrumentations for OTel Java and OTel Python:
Today, the AWS SDK Instrumentation for JS relies on the HTTP Instrumentation to perform the context propagation via request headers, this means that the Parent Span ID that is propagated belongs to the HTTP Span. So the aim for this issue is to allow context propagation in the absence of HTTP Instrumentation, where the Parent Span ID will belong to the AWS SDK Span. However, the fix should still ensure that the Parent Span ID belongs to the HTTP Span if the HTTP Instrumentation is present.
X-Ray SDK performs context propagation for AWS SDK V3 via adding Middleware. I also had this discussion with @trivikr before, one suggestion was also to use middleware.
One idea is to add middleware to the AWS SDK Instrumentation over here that will inject X-Ray Context like:
this.middlewareStack?.add(
(next: any, context: any) => async (middlewareArgs: any) => {
awsXrayPropagator.inject(otelContext.active(), middlewareArgs.request.headers, defaultTextMapSetter);
const result = await next(middlewareArgs);
return result;
},
{
step: 'build',
name: '_otelInjectXrayContextMiddleware',
override: true,
}
);
From my own testing, if HTTP Instrumentation is enabled, it will override the XRay Context injected by the middleware. So I can help add a fix for this.