Datadog trace sampling doesn't appear to be working

[davidegreenwald]

Describe the bug

We use the Hive fork of the Apollo Router and have configured OTEL, following the docs:

telemetry:
  instrumentation:
    spans:
      mode: spec_compliant
  exporters:
    metrics:
      common:
        resource:
          service.name: router-public
      prometheus:
        enabled: true
    tracing:
      common:
        service_name: router-public
        preview_datadog_agent_sampling: true
        sampler: 0.02
      otlp:
        enabled: true
        endpoint: "${env.DD_AGENT_HOST}:4317"

Setting sampler from 0.1 to 0.02 today had no impact on our ingestion data:

In Datadog APM, it says that 100% of traces from the router are being ingested.

Expected behavior

Setting preview_datadog_agent_sample: true should send all spans to the agent but only forward the sampler percentage to Datadog. Changing sampler should reduce our Datadog usage and costs.

Additional context

We run dozens of microservices of different types and the router and subgraph services occupy most of our Datadog usage and cost. It is a high priority for our organization that sampling is functional and straightforward. This has been a recurring issue for the last several years.

Any support here is extremely welcome.