[BUG]: Botocore distributed tracing causes invalid AWS SigV4 signatures for S3 requests

[andrey-berenda]

Tracer Version(s)

4.12.1, installed through Datadog Single Step Instrumentation

Python Version(s)

Python 3.14.6

Pip Version(s)

Not applicable. The application environment is managed with uv, and ddtrace is injected by Datadog Single Step Instrumentation under /opt/datadog/apm/library/python/.

Bug Report

After pods started receiving ddtrace 4.12.1 through Single Step Instrumentation, boto3 S3 requests made from an instrumented FastAPI request began failing AWS SigV4 authentication.

Observed failures:

• HeadObject: HTTP 403 Forbidden
• CopyObject: SignatureDoesNotMatch with The request signature we calculated does not match the signature you provided. Check your key and signing method.

The application uses normal boto3 APIs with a standard SigV4 S3 client. AWS authentication is provided by EKS Pod Identity. The IAM role, bucket access, region, and object existence were verified.

The same HeadObject request succeeds from a fresh Python process in the same Kubernetes pod, using the same Pod Identity, bucket, key, region, boto3 version, and botocore client configuration.

The problem was reproduced across multiple Kubernetes environments and newly created pods. Disabling the botocore integration resolves it. More specifically, setting DD_BOTOCORE_DISTRIBUTED_TRACING=false resolves the failure while allowing botocore instrumentation to remain enabled.

This appears related to the ddtrace 4.12.0 change that injects W3C/Datadog propagation headers into signed HTTP requests for all AWS services. The 4.12.0 release notes mention a fix for strict-signature endpoints, but S3 HeadObject and CopyObject still fail in this configuration on 4.12.1.

No account IDs, bucket names, object keys, credentials, session tokens, or customer data are included in this report.

Reproduction Code

import boto3
from botocore.config import Config
from ddtrace import tracer

client = boto3.client(
    "s3",
    config=Config(
        signature_version="s3v4",
        s3={
            "addressing_style": "virtual",
            "us_east_1_regional_endpoint": "regional",
        },
    ),
)

with tracer.trace("s3-reproducer"):
    client.head_object(
        Bucket="example-private-bucket",
        Key="example/object.pdf",
    )

The production application triggers the call inside an inbound distributed FastAPI trace. With the default botocore distributed-tracing setting, S3 rejects the request. With the following setting, it succeeds:

DD_BOTOCORE_DISTRIBUTED_TRACING=false

The application flow also uses:

client.copy_object(
    CopySource={
        "Bucket": "example-private-source-bucket",
        "Key": "example/object.pdf",
    },
    Bucket="example-private-destination-bucket",
    Key="example/copied-object.pdf",
    MetadataDirective="COPY",
)

For CopyObject, S3 returns the more explicit SignatureDoesNotMatch response.

Error Logs

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

ClientError: An error occurred (SignatureDoesNotMatch) when calling the CopyObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

The application stack includes:

ddtrace/contrib/internal/botocore/patch.py
botocore/client.py

Libraries in Use

ddtrace==4.12.1
boto3==1.43.2
botocore==1.43.2
Python==3.14.6
FastAPI
Starlette

Datadog library injection path indicates Linux manylinux2014 aarch64.

Operating System

Linux, manylinux2014, aarch64, running in Kubernetes on AWS EKS with EKS Pod Identity.

[andrey-berenda]

Additional verification and correction to the original report:

We tested the same read-only S3 HeadObject call from a fresh Python process inside the affected application pod, using the same existing object and an active Datadog span.

Results with ddtrace 4.12.1:

• DD_TRACE_BOTOCORE_ENABLED=true

  • botocore.client._datadog_patch == True
  • BaseClient._make_api_call was wrapped by Datadog
  • HeadObject returned HTTP 200

• DD_TRACE_BOTOCORE_ENABLED=false

  • botocore.client._datadog_patch was not set
  • botocore was not wrapped by Datadog
  • HeadObject returned HTTP 200

Therefore, the isolated script does not reproduce the failure merely by enabling the botocore integration. The environment variable behaves as expected and disabling it removes the botocore instrumentation.

The failure is reproducible through the real application HTTP endpoint while botocore tracing is enabled, and stops occurring when the deployed application is restarted with DD_TRACE_BOTOCORE_ENABLED=false. This suggests that additional request context is required to trigger it, most likely the active inbound web/distributed-trace context or another difference in the long-running instrumented application process.

Also, DD_BOTOCORE_DISTRIBUTED_TRACING=false has not been demonstrated to be a workaround. The confirmed workaround in our environment is disabling the complete botocore integration with DD_TRACE_BOTOCORE_ENABLED=false.

We are continuing to narrow the reproducer to include the application request context.

[andrey-berenda]

We now have a stable, read-only reproducer and have isolated the interaction to an active Sentry span plus Datadog botocore instrumentation.

Environment:

• Python 3.14
• Linux aarch64
• ddtrace==4.12.1 (Datadog admission-controller library injection)
• sentry-sdk==2.66.0
• boto3/botocore client using AWS SigV4

Minimal reproducer:

import boto3
import sentry_sdk
from botocore.config import Config
from fastapi import FastAPI
from fastapi.testclient import TestClient
from sentry_sdk.integrations.fastapi import FastApiIntegration

sentry_sdk.init(
    dsn=None,
    traces_sample_rate=1.0,
    integrations=[FastApiIntegration()],
)

s3 = boto3.client(
    "s3",
    config=Config(
        signature_version="s3v4",
        s3={
            "addressing_style": "virtual",
            "us_east_1_regional_endpoint": "regional",
        },
    ),
)

app = FastAPI()

@app.post("/reproduce")
def reproduce():
    with sentry_sdk.start_span(op="dependency.resolve", name="FileStorage"):
        response = s3.head_object(
            Bucket="YOUR_BUCKET",
            Key="YOUR_EXISTING_KEY",
        )
    return {"status": response["ResponseMetadata"]["HTTPStatusCode"]}

response = TestClient(app).post("/reproduce")
print(response.status_code, response.text)

Run with Datadog auto-instrumentation enabled (in our environment this is admission-controller injection; with a normal installation the equivalent is DD_TRACE_BOTOCORE_ENABLED=true ddtrace-run python reproducer.py).

Observed result on the first call:

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Control matrix, using the same existing S3 object and pod identity:

Datadog botocore integration	Nested sentry_sdk.start_span	Result
enabled	present	403 Forbidden
enabled	absent	200 OK
disabled	present	200 OK

The confirmed workaround is DD_TRACE_BOTOCORE_ENABLED=false. The earlier hypothesis involving only DD_BOTOCORE_DISTRIBUTED_TRACING should be disregarded. We also reproduced the same failure through the real long-running FastAPI endpoint.

[dgolombek]

This bit us too, the DD + Sentry combo has to be pretty common, and the impact is pretty bad, please fix -- or at least add a warning, I didn't see anything in the 4.12.X release notes about this.

[dgolombek]

Some details on what we saw. Same exception, but on SQS rather than S3:

botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling
the SendMessage operation: The request signature we calculated does not match the
signature you provided.

Versions: ddtrace==4.12.0, sentry-sdk==2.66.1, Python 3.13, Linux x86_64 on ECS Fargate. Onset correlated exactly with the deploy that took us from 4.11.1 to 4.12.0 — no other change. Holding below 4.12 resolved it.

Mechanism: two writers produce a duplicate baggage header

The SignatureDoesNotMatch body echoes back the canonical string AWS computed. In ours, baggage appears in SignedHeaders, and its value contains two concatenated Sentry contexts from two different Sentry orgs — ours, and one belonging to an upstream third party whose webhook initiated the request:

baggage:sentry-trace_id=<A>,sentry-environment=production,sentry-release=<our-release>,
sentry-public_key=<ours>,sentry-org_id=<our-org>,sentry-environment=production,
sentry-release=<their-agent-version>,sentry-public_key=<theirs>,sentry-trace_id=<B>,
sentry-org_id=<their-org>,sentry-transaction=POST%20https%3A%2F%2F<redacted>

That is one header value formed by HTTP folding of two separate baggage headers. The two writers:

1. ddtrace 4.12's _inject_trace_headers_handler, registered on botocore's before-sign, writes baggage into the AWSRequest. Because this runs before signing, baggage lands in the SigV4 canonical request and is declared in SignedHeaders.
2. sentry-sdk's stdlib integration then calls putheader for each of iter_trace_propagation_headers() at the httplib layer — after signing. putheader appends rather than replaces, so this emits a second baggage header that ddtrace's _http_propagation_suppressed seam does not and cannot suppress, since sentry-sdk knows nothing about it.

AWS folds the two into one comma-joined value, which no longer matches what was signed.

Relevant sentry-sdk detail: that injection is gated on should_propagate_trace(client, real_url), which resolves trace_propagation_targets. That option defaults to match-all, so sentry-sdk decorates every outbound request, including signed AWS API calls. Setting trace_propagation_targets to first-party hosts stops sentry-sdk touching AWS calls and is the application-side fix.

Why ours only affected a subset of requests

_BaggageHeader._inject early-returns when the span context carries no baggage items:

baggage_items = span_context._baggage.items()
if not baggage_items:
    return

So ddtrace only writes the pre-sign baggage header when something populated baggage — for us, an inbound baggage header on the incoming webhook. Requests without one got only sentry-sdk's unsigned baggage, which AWS ignores because it isn't in SignedHeaders. That is why this was invisible for the nine minutes between our deploy and the first webhook carrying inbound baggage, and why the blast radius was confined to two inbound routes rather than the whole service.

This is the one place our variant may differ from the reproducer above, which does not obviously involve an inbound baggage header — so the trigger there may be a different populator of baggage, or a different duplicated header entirely. Does the S3 failure also show a duplicated baggage in the echoed canonical string? If it does, the two are the same bug. If it does not, there may be a second path into the same class of failure.