[Spike] Investigate feasibility of only adding distributed tracing headers when a trace is marked as sampled #2742
Description
In some very high throughput scenarios it can be prohibitively expensive to always add distributed tracing headers. For example, adding headers to kafka messages in very high throughput systems can impact performance/cost due to the size of the headers.
The Java agent supports two different forms of DT headers, our legacy newrelic header and W3C Trace Context headers. The legacy newrelic header is especially large and almost never needs to be used unless distributed traces involve very old APM agents that pre-date W3C support. In most cases, it is recommend to disable the legacy headers as follows:
distributed_tracing:
exclude_newrelic_header: true
In cases where the W3C headers are still problematic, we should investigate the feasibility of only adding DT headers when a trace is marked as sampled=true. Such functionality should be behind a config toggle, that can flip between the existing behavior of always adding DT headers or only adding them when a trace is marked as sampled. Ideally this should be a general solution that applies regardless of which instrumentation modules might actually be triggering the headers to be added.
A new config might look something like the following where always and when_sampled are options:
distributed_tracing:
add_headers: always/when_sampled