Null-spans, or disabled spans? Null tracer?

[kristjanvalur]

In our stack, I have been using this project for a particular system. Some colleagues have been adding tracing to Rust projects. We have various components here and there.

One feature they talk about is to check if an incoming request has a trace, and only in this case propagate the tracing through the system.

I don't see how this is possible transparently with dd-trace-cpp. The library relies heavily on auto allocated Span objects and it would be cumbersome to conditionally create and propagate spans.

Wouldn't it be cleaner to be able to set a flag on the captured or root span which then propagates to child spans, declaring the whole trace as disabled, and making trace injection a no-op?

Similarly, a report_traces member might control trace reporting on a trace-by-trace basis.

This would also tie in with a NullTracer a tracer you could create if you don't want any tracing to happen, or if tracer configuration somehow failed. Such a Tracer would create normal spans, but could optionally set the default to not inject any traces. ( I have mentioned this elsewhere, such a Null tracer would allow application code to be unchanged even if the library failed to configure correctly, or if it was decided to override all environment variables and not do any tracing at all)

In short, two flags:

class Span {
   ...
   void  set_report_trace(bool);  // call this to set whether this trace should be reported or not.
   void set_inject_trace(bool);  // call this to set whether this trace should be injected into outgoing dicts or not
};

This would affect the entire trace, would probably be booleans on the trace.
A Null tracer would have both false, although it might want to create trace_ids and inject them for outgoing traces
A tracer might also have a flag as to whether to extract would be a no-op or not.

Having such extra flags might allow us to:

• customize trace handling for incoming traces, disabling trace reporting and trace propagation if there are no incoming traces.
• globally disable trace extraction from the Tracer
• customize the behaviour reporting of the library in null mode, e.g. does it create new trace ids and propagate them, or not?

It is possible to implement the above features by wrapping the library, and I have done so for our implementation where custom classes contain optional std::unique_ptr members to a dd::Span, but having it supported by the core library might be useful for other integrators.

Any thoughts?

[dmehala]

Hi @kristjanvalur

Thanks for taking the time to describe the issue you're encountering.

Currently, the library doesn't have built-in support for no-op mode, and I understand how this syntactic sugar could be useful in your case. It's a reasonable request, and I'll prioritize exploring enhancements to improve the usability of the library early in January 2025.

In the meantime, you may need to implement a custom solution. One approach is to create a Span interface with two implementations:

• NoopSpan: A no-op implementation that does nothing (useful when no span is extracted from an incoming request).
• DatadogSpan: A wrapper around datadog::tracing::Span that forwards method calls to the actual span object.

Then depending of extract_span instanciate either a NoopSpan or DatadogSpan.

To illustrate:

namespace mainframe {
   class ISpan {
      public:
         virtual std::unique_ptr<ISpan> create_child(const SpanConfig& config) const = 0;
         ...
   };

   class NoopSpan : ISpan {
   public:
      std::unique_ptr<ISpan> create_child(const SpanConfig&) const { return std::make_unique<NoopSpan>(); }
   };

   class DatadogSpan : ISpan {
      datadog::tracing::Span span_;
      
      public:
         std::unique_ptr<ISpan> create_child(const SpanConfig& config) const override {
            return std::make_unique<DatadogSpan>(span_.create_child(config));
        }
   };

   std::unique_ptr<ISpan> handle_request(Request& req) {
      ...
      auto maybe_span = tracer.extract_span(req);
      if (!maybe_span) {
         return std::make_unique<NoopSpan>();
      }

      return std::make_unique<DatadogSpan>(*maybe_span);
   }
}

The handle_request function demonstrates how you could use these objects conditionally.

Although this solution involves writing some boilerplate code, it provides flexibility and allows you to decouple your application logic from the specific behaviour if tracing context exists or not.

I know it's cumbersome, you probably don't want to write it, unfortunately at the moment, I can't offer an out of the box solution to handle this scenario directly.

Let me know if you need further clarification.

[kristjanvalur]

Thanks for your reply.
In fact, I have already written a bunch of boilerplate to work around the auto semantics of the dd::Span object. Since I'm already doing that, I can in fact add the functionality that I describe. However, my musings here are intended to make it simpler for future integrators to enjoy more flexibility.
Also, the documentation and examples do not show much in the way of async and optional handling.

I am working on an integration for UnrealEngine. Mostly this is a set of C++ modules and interfaces, but also some Blueprint classes. There is a thread local span stack and provision for storing spans in lambdas and other callbacks. Also a bunch of niceties such as unicode transforms. Once this is ready I'll open up the repo and it could be used as an example reference.

In our game, the server performs a lot of http requests, and gprc requests and we need more visibility, hence the integration. We are also looking at integrating this with Unreal's own RPC mechanism, but it remains to be seen how widely we can deploy this. Needless to say, we don't want to send traces from a client, (end user game), even if the user accidentally sets a DD_AGENT_HOST variable in his environment. But we may want to inject trace ids into outgoing requests to be able to group them on the server, even if the traces from the client are not sent themselves.

All work in progress. Cheers!

[KenFigueiredo]

Hey @dmehala - any updates on if this is still to be dropped this month or do you have a working branch that's available for testing this feature?

Working through the same issue right now in trying to setup a library to abstract away this feature and running into some issues with proper scope cleanup when following your example.

Thanks!

[kristjanvalur]

I have a file with a complete implementation of these features, allow me to upload it here.
We wrap both the Tracer and Span using simple wrapper classes employing std::optional.

// OptTrace.h
// Provide a wrapper around the datadog::tracing::Tracer and datadog::tracing::Span
// classes, to allow for the case when the Tracer is not initialized, or when we
// want to keep it disabled.  This allows us to use the same code paths for both.

#pragma once

#include <optional>
#include "datadog/span.h"
#include "datadog/tracer.h"

namespace DDTrace {

class OptSpan {
    using Span = datadog::tracing::Span;
    using SpanConfig = datadog::tracing::SpanConfig;
    using TraceID = datadog::tracing::TraceID;
    template <typename Value>
    using Optional = datadog::tracing::Optional<Value>;
    using TimePoint = datadog::tracing::TimePoint;
    using StringView = datadog::tracing::StringView;
    using DictReader = datadog::tracing::DictReader;
    using DictWriter = datadog::tracing::DictWriter;
    using InjectionOptions = datadog::tracing::InjectionOptions;
    template <typename Value>
    using Expected = datadog::tracing::Expected<Value>;
    using TraceSegment = datadog::tracing::TraceSegment;
 public:
    OptSpan() = default;
    OptSpan(const OptSpan&) = delete;
    OptSpan(OptSpan&& span) = default;
    OptSpan(Span&& span) : span_(std::move(span)) {}
    // child constructor, copies flags from parent
    OptSpan(const OptSpan &parent, Span&& span) :
        span_(std::move(span)),
        no_inject(parent.no_inject)
    {}

    OptSpan& operator=(const OptSpan&) = delete;
    OptSpan& operator=(OptSpan&&) = delete;

    // allow to reset
    void reset() { span_.reset(); }

    // direct accessors to the span
    Span& operator*() { return span_.value(); }
    const Span& operator*() const { return span_.value(); }

    Span* operator->() { return &span_.value(); }
    const Span* operator->() const { return &span_.value(); }

    explicit operator bool() const { return span_.has_value(); }

    // allow do disable injection on this span and child spans.
    // useful if we want to only trace if incoming headers are present.
    void set_no_inject(bool b) { no_inject = b; }
    bool get_no_inject() const { return no_inject; }

    // wrapping methods.  See span.h for details
    OptSpan create_child(const SpanConfig& config) const {
        if (span_.has_value()) {
            return {*this, span_->create_child(config)};
        }
        return {};
    }
    OptSpan create_child() const {
        if (span_.has_value()) {
            return { *this, span_->create_child() };
        }
        return {};
    }

    std::uint64_t id() const
    {
        return span_.has_value() ? span_->id() : 0;
    }
    
    TraceID trace_id() const {
        return span_.has_value() ? span_->trace_id() : TraceID();
    }

    Optional<std::uint64_t> parent_id() const {
        return span_.has_value() ? span_->parent_id() : Optional<std::uint64_t>();
    }

    TimePoint start_time() const {
        return span_.has_value() ? span_->start_time() : TimePoint();
    }
    
    bool error() const {
        return span_.has_value() ? span_->error() : false;
    }

    const std::string& service_name() const {
        return span_.has_value() ? span_->service_name() : std::string();
    }

    const std::string& service_type() const {
        return span_.has_value() ? span_->service_type() : std::string();
    }

    const std::string& name() const {
        return span_.has_value() ? span_->name() : std::string();
    }

    const std::string& resource_name() const {
        return span_.has_value() ? span_->resource_name() : std::string();
    }

    Optional<StringView> lookup_tag(StringView name) const {
        return span_.has_value() ? span_->lookup_tag(name) : Optional<StringView>();
    }

    Optional<double> lookup_metric(StringView name) const {
        return span_.has_value() ? span_->lookup_metric(name) : Optional<double>();
    }
    
    void set_tag(StringView name, StringView value) {
        if (span_.has_value()) {
            span_->set_tag(name, value);
        }
    }
    
    void set_metric(StringView name, double value) {
        if (span_.has_value()) {
            span_->set_metric(name, value);
        }
    }
    
    void remove_tag(StringView name) {
        if (span_.has_value()) {
            span_->remove_tag(name);
        }
    }
    
    void remove_metric(StringView name) {
        if (span_.has_value()) {
            span_->remove_metric(name);
        }
    }

    void set_service_name(StringView name) {
        if (span_.has_value()) {
            span_->set_service_name(name);
        }
    }
    void set_service_type(StringView name) {
        if (span_.has_value()) {
            span_->set_service_type(name);
        }
    }

    void set_name(StringView name) {
        if (span_.has_value()) {
            span_->set_name(name);
        }
    }

    void set_resource_name(StringView name) {
        if (span_.has_value()) {
            span_->set_resource_name(name);
        }
    }

    void set_error(bool is_error) {
        if (span_.has_value()) {
            span_->set_error(is_error);
        }
    }

    void set_error_message(StringView error_message) {
        if (span_.has_value()) {
            span_->set_error_message(error_message);
        }
    }

    void set_error_type(StringView error_type) {
        if (span_.has_value()) {
            span_->set_error_type(error_type);
        }
    }

    void set_error_stack(StringView error_stack) {
        if (span_.has_value()) {
            span_->set_error_stack(error_stack);
        }
    }

    void set_end_time(std::chrono::steady_clock::time_point end_time) {
        if (span_.has_value()) {
            span_->set_end_time(end_time);
        }
    }

    void inject(DictWriter& writer) const {
        if (span_.has_value() && !no_inject) {
            span_->inject(writer);
        }
    }

    void inject(DictWriter& writer, const InjectionOptions& options) const {
        if (span_.has_value() && !no_inject) {
            span_->inject(writer, options);
        }
    }

    Expected<void> read_sampling_delegation_response(const DictReader& reader)
    {
        if (span_.has_value()) {
            return span_->read_sampling_delegation_response(reader);
        }
        return {};
    }

    // return the segment pointer or null
    TraceSegment* trace_segment() {
        if (span_.has_value()) {
            return &span_->trace_segment();
        }
        return nullptr;
    }

    const TraceSegment* trace_segment() const
    {
        if (span_.has_value()) {
            return &span_->trace_segment();
        }
        return nullptr;
    }

 private:
    std::optional<datadog::tracing::Span> span_;  // The wrapped span
    // extra flags:  no_inject:  do not inject this span into the headers
    bool no_inject = false;
};


// Similarly, wrap the Tracer for the case when we don't manage to configure
// the tracer properly, or want to keep it disabled.  
class OptTracer
{
    using Tracer = datadog::tracing::Tracer;
    using SpanConfig = datadog::tracing::SpanConfig;
    using FinalizedTracerConfig = datadog::tracing::FinalizedTracerConfig;
    using IDGenerator = datadog::tracing::IDGenerator;
    using Clock = datadog::tracing::Clock;
    template <typename Value>
    using Expected = datadog::tracing::Expected<Value>;
    using DictReader = datadog::tracing::DictReader;

public:
    OptTracer() = default;
    explicit OptTracer(const FinalizedTracerConfig& config)
        : tracer_(Tracer(config)) {}

    OptTracer(const FinalizedTracerConfig& config, const std::shared_ptr<const IDGenerator>& generator)
        : tracer_(Tracer(config, generator))
    {}

    // explicitly create a null span, e.g. when extraction fails and one does not
    // want to do any tracing.
    OptSpan create_nullspan()
    {
        return {};
    }

    OptSpan create_span()
    {
        if (tracer_) {
            return tracer_->create_span();
        }
        return {};
    }
    OptSpan create_span(const SpanConfig& config)
    {
        if (tracer_) {
            return tracer_->create_span(config);
        }
        return {};
    }

    Expected<OptSpan> extract_span(const DictReader& reader)
    {
        if (tracer_) {
            auto maybe_span = tracer_->extract_span(reader);
            if (maybe_span) {
                return OptSpan(std::move(*maybe_span));
            }
            return maybe_span.error();
        }
        return {};
    }

    Expected<OptSpan> extract_span(const DictReader& reader, const SpanConfig& config)
    {
        if (tracer_) {
            auto maybe_span = tracer_->extract_span(reader, config);
            if (maybe_span) {
                return OptSpan(std::move(*maybe_span));
            }
            return maybe_span.error();
        }
        return {};
    }

    OptSpan extract_or_create_span(const DictReader& reader)
    {
        if (tracer_) {
            return tracer_->extract_or_create_span(reader);
        }
        return {};
    }
    
    OptSpan extract_or_create_span(const DictReader& reader, const SpanConfig& config)
    {
        if (tracer_) {
            return tracer_->extract_or_create_span(reader, config);
        }
        return {};
    }

    std::string config() const
    {
        if (tracer_) {
            return tracer_->config();
        }
        return "{}";
    }

private:
    std::optional<Tracer> tracer_;
};

}  // namespace DDTrace

[dmehala]

Hi @KenFigueiredo

Unfortunately, this work has been deprioritized for the quarter. That said, I’ll do my best to make progress and will keep you updated.