Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(csharp/src/Apache.Arrow.Adbc): add OpenTelemetry compatible tracing support #2559

Open
wants to merge 108 commits into
base: main
Choose a base branch
from

Conversation

birschick-bq
Copy link
Contributor

@birschick-bq birschick-bq commented Feb 25, 2025

This PR adds OpenTelemetry-compatible tracing support on the CSharp library.

Under the hood it uses Activity and ActivitySource from the net System.Diagnostics.DiagnosticSource package.

Create an instance of the ActivityTrace object.

public class YourDriver : AdbcDriver
{
    public YourDriver(string? activitySourceName = default, string? traceParent = default)
    {
        Trace = new ActivityTrace(activitySourceName, traceParent);
    }
    protected ActivityTrace Trace { get; }
}

By default, the activitySourceName will be created from the driver assembly name.

Then to instrument tracing, use one of the following overrides of

  • TraceActivity
  • TraceActivityAsync

Example:

 public void Connect(...)
 {
    Trace.TraceActivity((activity) =>
    {
        driver.OpenSession(...);
    });
 }

Each of these overrides starts a new Activity which will be non-null only if there is an active ActivityListener or OpenTelemetry exporter. The Activity is passed to the delegate Func/Action in case it need to add ActivityEvent, ActivityLink or Tags (KeyValuePair). When instrumenting tracing, you should always use the null-conditional operator (?. ) when accessing members on the passed delegate parameter, activity.

Example:

public void GetObjects(...)
{
    TraceActivity((activity) =>
    {
        activity?.AddEvent("db.operation.name.start", [new("db.operation.name", nameof(Client.GetCatalogs))]);
        var result = driver.GetCatalogs(...);
        var eventActivity = activity?.AddEvent("db.operation.name.end",
            [
                new("db.operation.name", nameof(Client.GetCatalogs)),
                new("db.response.status_code", getCatalogsResp.Status.StatusCode.ToString())
            ]);
    });
}

The default behavior is to invoke the delegate and if successful (i.e., no exception thrown), the Activity.SetStatus is set to Ok. If an exception is observed, then the Activity.SetStatus is set to Error and the exception is traced (Activity.AddException) as an event in the current Activity.

Callers can pass a traceparent string to any of the TraceActivity[Async] methods using the optional traceParent parameter. The parameter takes precedence over the property. The traceId from the traceParent parameter or TraceParent property will be adopted as the rootId for all trace Activity on that call or object. If TraceParent is null (initial or set later), then the Activity creates a new rootId for the beginning of the initial Activity in the stack.

Example:

public void Connect(..., string? traceParent = default)
{
    TraceActivity((activity) =>
    {
        activity?.AddEvent("opensession.start");
        var result = driver.OpenSession(...);
        activity?.AddEvent("opensession.end", ["status", result.Status]);
    }, traceParent: traceParent);
}

Identifiers used for events and tags should follow the OpenTelemetry semantic guidance ..
https://opentelemetry.io/docs/specs/semconv/database/database-spans/
https://opentelemetry.io/docs/specs/semconv/database/database-metrics/

@birschick-bq birschick-bq changed the title Dev/birschick bq/tracing alternative feat(csharp/src/Apache.Arrow.Adbc): Draft: Tracing alternative Feb 25, 2025
@birschick-bq birschick-bq changed the title feat(csharp/src/Apache.Arrow.Adbc): Draft: Tracing alternative feat(csharp/src/Apache.Arrow.Adbc): add OpenTelemetry compatible tracing support Feb 25, 2025
/// <summary>
/// Simplified version of <see cref="Activity"/> that excludes some properties, etc.
/// </summary>
internal class SerializableActivity
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is explicitly necessary, but helps with testing. It also "could" be useful a serialized version of Activity

public override AdbcDatabase Open(IReadOnlyDictionary<string, string> parameters)
{
return new HiveServer2Database(parameters);
return new HiveServer2Database(parameters, Trace);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing reference to Trace object to child object for consistent activity source name and handling of trace parent.

activity?.AddEvent("db.operation.name.start", [new("db.operation.name", nameof(Client.FetchResults))]);
TFetchResultsReq request = new TFetchResultsReq(this.statement.OperationHandle, TFetchOrientation.FETCH_NEXT, this.statement.BatchSize);
TFetchResultsResp response = await this.statement.Connection.Client!.FetchResults(request, cancellationToken);
activity?.AddEvent("db.operation.name.end",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Helps to trace the begin/end of fetching results from the server so the duration can be calculated from the difference.

TGetCatalogsResp getCatalogsResp = Client.GetCatalogs(getCatalogsReq, cancellationToken).Result;
// Since this API only allows non-null values, we'll treat empty string as null to allow the TraceParent to be unset.
case HiveServer2Parameters.TraceParent:
Trace.TraceParent = !string.IsNullOrWhiteSpace(value)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allow the caller to set the trace_parent on the connection.

@birschick-bq birschick-bq marked this pull request as ready for review March 4, 2025 23:40
@github-actions github-actions bot added this to the ADBC Libraries 17 milestone Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants