Skip to content

Commit 38a1987

Browse files
lloekiekump
andauthored
feat(data-pipeline): add meta struct blob setter (#2303)
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated. # What does this PR do? Adds an opaque per-key `meta_struct` blob setter to `TracerSpan`. Keys are validated as UTF-8, while values are copied as uninterpreted binary data. This transitional API lets consumers pass their existing per-key MessagePack blobs without changing the Rust trace model. # Motivation Native exporter consumers cannot currently populate `meta_struct`, which blocks payload parity tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). The opaque setter provides the safe first stage before the language-neutral structured-value APIs. # Additional Notes **Stack position: L1, current PR: [DataDog/libdatadog PR 2303](#2303 These open L0-L5 changes are not included in [libdatadog v38.0.0](https://github.com/DataDog/libdatadog/releases/tag/v38.0.0). They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). Complete expected libdatadog review and merge order for that single v39 release: | Level | libdatadog PR | Matching dd-trace-rb consumer | | --- | --- | --- | | L0 | [DataDog/libdatadog PR 2302](#2302) | Release prerequisite; no payload API consumer | | **L1 (current)** | **[DataDog/libdatadog PR 2303](#2303 | **[R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130 | | L2 | [DataDog/libdatadog PR 2305](#2305) | [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129) | | L3 | [DataDog/libdatadog PR 2301](#2301) | [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134) | | L4 | [DataDog/libdatadog PR 2304](#2304) | [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132) | | L5 | [DataDog/libdatadog PR 2300](#2300) | [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133) | L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. The matching payload consumers are [R4 DataDog/dd-trace-rb PR 6130](DataDog/dd-trace-rb#6130), [R5 DataDog/dd-trace-rb PR 6129](DataDog/dd-trace-rb#6129), [R6 DataDog/dd-trace-rb PR 6134](DataDog/dd-trace-rb#6134), [R7 DataDog/dd-trace-rb PR 6132](DataDog/dd-trace-rb#6132), and [R8 DataDog/dd-trace-rb PR 6133](DataDog/dd-trace-rb#6133). All depend through the common v39 uptake in [R3 DataDog/dd-trace-rb PR 6143](DataDog/dd-trace-rb#6143). The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7. [DataDog/libdatadog-rb v37 PR 65](DataDog/libdatadog-rb#65) and [DataDog/libdatadog-rb v38 PR 66](DataDog/libdatadog-rb#66) are separate ordered prerequisite releases only for [R2 DataDog/dd-trace-rb PR 6131](DataDog/dd-trace-rb#6131) and its `ForkSafeRuntime` uptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake. L0 tracks [APMSP-3830](https://datadoghq.atlassian.net/browse/APMSP-3830); L1-L5 implement the payload work tracked by [APMSP-3219](https://datadoghq.atlassian.net/browse/APMSP-3219). # How to test the change? Prior local validation completed successfully: - `nix develop -c cargo test -p libdd-data-pipeline-ffi` passed, including arbitrary binary values, overwrite semantics, null handles, and invalid UTF-8 keys. - `nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings` passed. - `nix develop .#nightly -c cargo fmt --all -- --check` passed. - The generated C header was rebuilt through `nix develop -c cargo run --bin release -- --out /tmp/opencode/native-meta-struct-libdatadog/x86_64-linux` and consumed successfully by the matching dd-trace-rb branch. - `nix develop .#ruby34 --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/appsec/actions_handler/serializable_backtrace_spec.rb` passed 116 examples, including AppSec and AI Guard-shaped wire conformance. - Targeted Standard and Steep checks for the native transport passed. [APMSP-3219]: https://datadoghq.atlassian.net/browse/APMSP-3219?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [APMSP-3830]: https://datadoghq.atlassian.net/browse/APMSP-3830?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: ekump <edmund.kump@datadoghq.com> Co-authored-by: loic.nageleisen <loic.nageleisen@datadoghq.com>
1 parent e53d1cf commit 38a1987

1 file changed

Lines changed: 212 additions & 34 deletions

File tree

libdd-data-pipeline-ffi/src/tracer.rs

Lines changed: 212 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,9 @@ use crate::error::{ExporterError, ExporterErrorCode as ErrorCode};
1313
use crate::response::ExporterResponse;
1414
use crate::trace_exporter::TraceExporter;
1515
use crate::{catch_panic, gen_error};
16-
use libdd_common_ffi::slice::AsBytes;
16+
use libdd_common_ffi::slice::{AsBytes, ByteSlice};
1717
use libdd_common_ffi::CharSlice;
18-
use libdd_tinybytes::BytesString;
18+
use libdd_tinybytes::{Bytes, BytesString};
1919
use libdd_trace_utils::span::v04::SpanBytes;
2020
use std::ptr::NonNull;
2121

@@ -27,10 +27,17 @@ type TokioCancellationToken = tokio_util::sync::CancellationToken;
2727

2828
/// Convert a [`CharSlice`] to a [`BytesString`], copying the bytes.
2929
///
30-
/// Returns an error if the slice is not valid UTF-8.
30+
/// Returns [`ErrorCode::InvalidArgument`] if the slice is malformed and [`ErrorCode::InvalidInput`]
31+
/// if the bytes are not valid UTF-8.
3132
#[inline]
3233
fn charslice_to_bytesstring(s: CharSlice) -> Result<BytesString, Box<ExporterError>> {
33-
BytesString::from_slice(s.as_bytes()).map_err(|_| {
34+
let bytes = s.try_as_bytes().map_err(|_| {
35+
Box::new(ExporterError::new(
36+
ErrorCode::InvalidArgument,
37+
&ErrorCode::InvalidArgument.to_string(),
38+
))
39+
})?;
40+
BytesString::from_slice(bytes).map_err(|_| {
3441
Box::new(ExporterError::new(
3542
ErrorCode::InvalidInput,
3643
&ErrorCode::InvalidInput.to_string(),
@@ -67,9 +74,12 @@ pub struct TracerSpanFields<'a> {
6774

6875
/// Create a new span with all scalar fields set.
6976
///
70-
/// String fields are copied from the provided slices. The `meta` and
71-
/// `metrics` maps start empty; use [`ddog_tracer_span_set_meta`] and
72-
/// [`ddog_tracer_span_set_metric`] to populate them.
77+
/// String fields are copied from the provided slices. The `meta`, `metrics` and `meta_struct`
78+
/// maps start empty; use [`ddog_tracer_span_set_meta`], [`ddog_tracer_span_set_metric`] and
79+
/// [`ddog_tracer_span_set_meta_struct_blob`] to populate them.
80+
///
81+
/// Returns an error if `fields` is null, if any string field is not valid UTF-8, or if any of
82+
/// its slices is malformed.
7383
///
7484
/// # Safety
7585
///
@@ -134,10 +144,13 @@ pub unsafe extern "C" fn ddog_tracer_span_free(handle: Box<TracerSpan>) {
134144
///
135145
/// Both `key` and `value` are copied into the span.
136146
///
147+
/// Returns an error if `handle` is null, if `key` or `value` is not valid UTF-8, or if either
148+
/// slice is malformed.
149+
///
137150
/// # Safety
138151
///
139-
/// `handle` must be a valid pointer to a `TracerSpan`.
140-
/// `key` and `value` must point to valid UTF-8 memory.
152+
/// `handle` must be a valid pointer to a `TracerSpan`. `key` and `value` must point to valid
153+
/// memory for their stated lengths.
141154
#[no_mangle]
142155
pub unsafe extern "C" fn ddog_tracer_span_set_meta(
143156
handle: Option<&mut TracerSpan>,
@@ -167,10 +180,13 @@ pub unsafe extern "C" fn ddog_tracer_span_set_meta(
167180
///
168181
/// The `key` is copied into the span.
169182
///
183+
/// Returns an error if `handle` is null, if `key` is not valid UTF-8, or if the slice is
184+
/// malformed.
185+
///
170186
/// # Safety
171187
///
172-
/// `handle` must be a valid pointer to a `TracerSpan`.
173-
/// `key` must point to valid UTF-8 memory.
188+
/// `handle` must be a valid pointer to a `TracerSpan`. `key` must point to valid memory for its
189+
/// stated length.
174190
#[no_mangle]
175191
pub unsafe extern "C" fn ddog_tracer_span_set_metric(
176192
handle: Option<&mut TracerSpan>,
@@ -192,6 +208,45 @@ pub unsafe extern "C" fn ddog_tracer_span_set_metric(
192208
)
193209
}
194210

211+
/// Add or overwrite a structured metadata entry (`meta_struct`) on the span.
212+
///
213+
/// The `key` and opaque binary `value` are copied into the span. The value is
214+
/// not interpreted or validated as MessagePack.
215+
///
216+
/// Returns an error if `handle` is null, if `key` is not valid UTF-8, or if either slice is
217+
/// malformed.
218+
///
219+
/// # Safety
220+
///
221+
/// `handle` must be a valid pointer to a `TracerSpan`. `key` and `value` must point to valid
222+
/// memory for their stated lengths.
223+
#[no_mangle]
224+
pub unsafe extern "C" fn ddog_tracer_span_set_meta_struct_blob(
225+
handle: Option<&mut TracerSpan>,
226+
key: CharSlice,
227+
value: ByteSlice,
228+
) -> Option<Box<ExporterError>> {
229+
catch_panic!(
230+
if let Some(span) = handle {
231+
let key = match charslice_to_bytesstring(key) {
232+
Ok(s) => s,
233+
Err(e) => return Some(e),
234+
};
235+
let value = match value.try_as_bytes() {
236+
Ok(v) => v,
237+
Err(_) => return gen_error!(ErrorCode::InvalidArgument),
238+
};
239+
span.0
240+
.meta_struct
241+
.insert(key, Bytes::copy_from_slice(value));
242+
None
243+
} else {
244+
gen_error!(ErrorCode::InvalidArgument)
245+
},
246+
gen_error!(ErrorCode::Panic)
247+
)
248+
}
249+
195250
// ---------------------------------------------------------------------------
196251
// TracerTraceChunks
197252
// ---------------------------------------------------------------------------
@@ -312,17 +367,14 @@ pub extern "C" fn ddog_trace_exporter_cancel_token_new() -> Box<TokioCancellatio
312367

313368
/// Cancel a cancellation token.
314369
///
315-
/// All clones of the same token observe the cancellation. If a
316-
/// [`ddog_trace_exporter_send_trace_chunks`] call is using this token at the
317-
/// time of cancellation, that send stops waiting for the agent at its next
318-
/// await point and returns an error; the trace chunks it was sending may be
319-
/// lost.
320-
///
321-
/// Cancellation only affects a send that is in progress. If no send is using
322-
/// the token, cancelling it has no immediate effect: a send started afterwards
323-
/// with an already-cancelled token returns an error without contacting the
324-
/// agent, and a token cancelled after its send has already finished does
325-
/// nothing.
370+
/// All clones of the same token observe the cancellation. Cancellation is cooperative and only
371+
/// affects a [`ddog_trace_exporter_send_trace_chunks`] call that is in flight: that send stops
372+
/// waiting for the agent at its next await point and fails with
373+
/// [`ExporterErrorCode::IoError`], and the chunks it was sending may be lost.
374+
///
375+
/// Cancelling while no send is using the token has no immediate effect. A send started later with
376+
/// an already-cancelled token fails the same way without contacting the agent, and cancelling
377+
/// after a send has finished does nothing.
326378
#[no_mangle]
327379
pub extern "C" fn ddog_trace_exporter_cancel_token_cancel(token: Option<&TokioCancellationToken>) {
328380
if let Some(token) = token {
@@ -346,19 +398,11 @@ pub extern "C" fn ddog_trace_exporter_cancel_token_drop(
346398

347399
/// Send trace chunks through a [`TraceExporter`], consuming the chunks.
348400
///
349-
/// This calls `TraceExporter::send_trace_chunks` which processes stats,
350-
/// serializes in the configured output format, and sends to the agent
351-
/// with retry logic.
401+
/// Computes stats, serializes in the configured output format, and sends to the agent with
402+
/// retries.
352403
///
353-
/// When `cancel` is non-null, cancelling that token (via
354-
/// [`ddog_trace_exporter_cancel_token_cancel`]) while the send is in progress
355-
/// aborts the in-flight request and returns an error with code
356-
/// [`ExporterErrorCode::IoError`]. Cancellation is cooperative: it only takes
357-
/// effect while a request is actually in flight. A token that is already
358-
/// cancelled when the send starts makes this function return that error
359-
/// immediately, and cancelling after the send has finished has no effect.
360-
/// Cancelling an in-flight send may cause the trace chunks being sent to be
361-
/// lost.
404+
/// When `cancel` is non-null, cancelling that token aborts the in-flight request; see
405+
/// [`ddog_trace_exporter_cancel_token_cancel`].
362406
///
363407
/// On success, if `response_out` is non-null, a heap-allocated
364408
/// [`ExporterResponse`] is written there. The caller owns it and must
@@ -407,6 +451,10 @@ mod tests {
407451
CharSlice::from_bytes(s.as_bytes())
408452
}
409453

454+
fn bs(bytes: &[u8]) -> ByteSlice<'_> {
455+
ByteSlice::from(bytes)
456+
}
457+
410458
fn make_minimal_span() -> Box<TracerSpan> {
411459
unsafe {
412460
let mut handle = MaybeUninit::<Box<TracerSpan>>::uninit();
@@ -465,6 +513,7 @@ mod tests {
465513
assert_eq!(span.0.error, 0);
466514
assert!(span.0.meta.is_empty());
467515
assert!(span.0.metrics.is_empty());
516+
assert!(span.0.meta_struct.is_empty());
468517
assert!(span.0.span_links.is_empty());
469518
assert!(span.0.span_events.is_empty());
470519

@@ -527,6 +576,39 @@ mod tests {
527576
}
528577
}
529578

579+
#[test]
580+
fn set_meta_struct_blob_inserts_binary_entries() {
581+
unsafe {
582+
let mut span = make_minimal_span();
583+
let value = b"\x82\xa6nested\x92\xc3\xc0\xa3raw\xc4\x03\x00\xff\x80";
584+
585+
let err =
586+
ddog_tracer_span_set_meta_struct_blob(Some(&mut *span), cs("_dd.stack"), bs(value));
587+
assert!(err.is_none());
588+
589+
assert_eq!(span.0.meta_struct.get("_dd.stack").unwrap().as_ref(), value);
590+
591+
ddog_tracer_span_free(span);
592+
}
593+
}
594+
595+
// Repeated keys are appended, not replaced: `VecMap` defers deduplication to encode time,
596+
// so both entries are retained and the last one wins on read.
597+
#[test]
598+
fn set_meta_struct_blob_last_write_wins() {
599+
unsafe {
600+
let mut span = make_minimal_span();
601+
602+
ddog_tracer_span_set_meta_struct_blob(Some(&mut *span), cs("k"), bs(b"first"));
603+
ddog_tracer_span_set_meta_struct_blob(Some(&mut *span), cs("k"), bs(b"second"));
604+
605+
assert_eq!(span.0.meta_struct.get("k").unwrap().as_ref(), b"second");
606+
assert_eq!(span.0.meta_struct.len(), 2);
607+
608+
ddog_tracer_span_free(span);
609+
}
610+
}
611+
530612
#[test]
531613
fn set_meta_null_handle_returns_error() {
532614
unsafe {
@@ -545,6 +627,102 @@ mod tests {
545627
}
546628
}
547629

630+
#[test]
631+
fn set_meta_struct_blob_null_handle_returns_error() {
632+
unsafe {
633+
let err = ddog_tracer_span_set_meta_struct_blob(None, cs("k"), bs(b"value"));
634+
assert!(err.is_some());
635+
ddog_trace_exporter_error_free(err);
636+
}
637+
}
638+
639+
#[test]
640+
fn set_meta_struct_blob_invalid_key_returns_error() {
641+
unsafe {
642+
let mut span = make_minimal_span();
643+
let key = CharSlice::from_bytes(&[0xff]);
644+
645+
let err = ddog_tracer_span_set_meta_struct_blob(Some(&mut *span), key, bs(b"value"));
646+
assert!(err.is_some());
647+
assert!(span.0.meta_struct.is_empty());
648+
ddog_trace_exporter_error_free(err);
649+
650+
ddog_tracer_span_free(span);
651+
}
652+
}
653+
654+
#[test]
655+
fn set_meta_struct_blob_null_value_returns_error() {
656+
unsafe {
657+
let mut span = make_minimal_span();
658+
let value = ByteSlice::from_raw_parts(std::ptr::null(), 5);
659+
660+
let err = ddog_tracer_span_set_meta_struct_blob(Some(&mut *span), cs("k"), value);
661+
assert_eq!(err.as_ref().unwrap().code, ErrorCode::InvalidArgument);
662+
assert!(span.0.meta_struct.is_empty());
663+
ddog_trace_exporter_error_free(err);
664+
665+
ddog_tracer_span_free(span);
666+
}
667+
}
668+
669+
#[test]
670+
fn set_meta_struct_blob_null_key_returns_error() {
671+
unsafe {
672+
let mut span = make_minimal_span();
673+
let key = CharSlice::from_raw_parts(std::ptr::null(), 5);
674+
675+
let err = ddog_tracer_span_set_meta_struct_blob(Some(&mut *span), key, bs(b"value"));
676+
assert_eq!(err.as_ref().unwrap().code, ErrorCode::InvalidArgument);
677+
assert!(span.0.meta_struct.is_empty());
678+
ddog_trace_exporter_error_free(err);
679+
680+
ddog_tracer_span_free(span);
681+
}
682+
}
683+
684+
#[test]
685+
fn set_meta_struct_blob_accepts_empty_value() {
686+
unsafe {
687+
let mut span = make_minimal_span();
688+
689+
let err = ddog_tracer_span_set_meta_struct_blob(Some(&mut *span), cs("k"), bs(b""));
690+
assert!(err.is_none());
691+
assert_eq!(span.0.meta_struct.get("k").unwrap().as_ref(), b"");
692+
693+
ddog_tracer_span_free(span);
694+
}
695+
}
696+
697+
// An empty key is valid UTF-8 and is technically accepted.
698+
#[test]
699+
fn set_meta_struct_blob_accepts_empty_key() {
700+
unsafe {
701+
let mut span = make_minimal_span();
702+
703+
let err = ddog_tracer_span_set_meta_struct_blob(Some(&mut *span), cs(""), bs(b"value"));
704+
assert!(err.is_none());
705+
assert_eq!(span.0.meta_struct.get("").unwrap().as_ref(), b"value");
706+
707+
ddog_tracer_span_free(span);
708+
}
709+
}
710+
711+
#[test]
712+
fn set_meta_null_value_returns_error() {
713+
unsafe {
714+
let mut span = make_minimal_span();
715+
let value = CharSlice::from_raw_parts(std::ptr::null(), 5);
716+
717+
let err = ddog_tracer_span_set_meta(Some(&mut *span), cs("k"), value);
718+
assert_eq!(err.as_ref().unwrap().code, ErrorCode::InvalidArgument);
719+
assert!(span.0.meta.is_empty());
720+
ddog_trace_exporter_error_free(err);
721+
722+
ddog_tracer_span_free(span);
723+
}
724+
}
725+
548726
#[test]
549727
fn new_with_empty_strings_succeeds() {
550728
unsafe {

0 commit comments

Comments
 (0)