Skip to content

Commit 05d785e

Browse files
leoromanovskypr-shepherd-6ad11f[bot]vjfridgeclaude
authored
feat(datadog-ffe): server-side EVP flagevaluation payload + bincode-safe sidecar delivery (#2117)
## Motivation PHP relies on libdatadog and the sidecar for server-side flag-evaluation delivery, so the shared sidecar path must preserve the same worker-facing EVP contract as the direct SDK implementations. This contribution gives PHP a reusable, backend-verifiable flagevaluation transport without adding a one-off EVP writer in the PHP extension, which keeps cross-SDK adoption smoother for SDK owners and APM review. ## Changes - Keeps the sidecar action enum bincode-compatible by appending `FfeFlagEvaluationBatch` after existing variants. - Keeps flagevaluation IPC structs bincode-safe; outbound placeholder stripping happens only in the sidecar flusher before the EVP POST. - Shares the common Agent EVP proxy endpoint/header/request path between FFE exposures and flagevaluation delivery. - Carries `context.evaluation` over IPC as a JSON-object string and re-expands it to an object for outbound EVP JSON. - Preserves valid context values such as `false`, `""`, `{}`, and `[]` in user context. - Adds the flagevaluation C ABI path without a `reason` argument. - Supports optional real `targeting_rule.key`. - Retries reliable enqueue using retained encoded bytes so reconnect retry does not consume the payload. - Coalesces PHP flagevaluation batches in the sidecar by schema-visible dimensions and folds overflow into degraded buckets. - Splits sidecar EVP POST bodies by encoded uncompressed JSON bytes under the 5 MiB EVP limit, keeping the existing 512-event chunking as a secondary guard. - Degrades an oversized full-tier row by omitting `targeting_key` and `context`; drops and logs only if the degraded row still cannot fit. ## Decisions - OpenFeature `reason` is not part of the native ABI, sidecar event, outbound EVP payload, or aggregation contract. - Bincode wire compatibility is part of the PHP product path because PHP reaches EVP through worker-to-sidecar IPC before HTTP. - FFE exposures and flagevaluation share Agent EVP proxy transport mechanics; schema-specific payload shaping remains in each flusher because exposure deduplication and flagevaluation coalescing have different contracts. - The flagevaluation sidecar flusher owns worker-facing JSON cleanup so shared wire structs remain plain enough for positional bincode serialization. - Degraded events omit targeting key and context while retaining visible flag, variant, allocation, error, runtime-default, and targeting-rule fields. - Payload splitting happens after sidecar JSON cleanup because that is where the final uncompressed POST body exists; count-based chunking alone is not sufficient for the EVP 5 MiB limit. - PHP consumes this flagevaluation endpoint through the C ABI; the companion PHP draft points its libdatadog submodule at this PR head. ```mermaid flowchart TD A[sidecar receives/coalesces batches] --> B[clean and encode event JSON] B --> C{candidate POST <= 5 MiB?} C -- yes --> D[POST through Agent EVP proxy] C -- no --> E{single full row can degrade?} E -- yes --> F[omit targeting_key and context] F --> B E -- no --> G[drop and log/count] ``` ## Validation Evidence ### Dogfooding App - Companion PHP draft: DataDog/dd-trace-php#3984 - `ffe-dogfooding` `app-php7` and `app-php8-openfeature` were run with local PHP artifacts using this libdatadog sidecar path. - Dogfooding ran with the real-backend EVP override: Agent `DD_EVP_PROXY_CONFIG_ADDITIONAL_ENDPOINTS={}`, `DD_SKIP_SSL_VALIDATION=false`, and `DD_REMOTE_CONFIGURATION_NO_TLS_VALIDATION=false`. - Sent 12 identical evaluations per targeting key for `ffe-dogfooding-string-flag`: - `libdd-php7-batch-20260623T020845Z-alpha` - `libdd-php7-batch-20260623T020845Z-bravo` - `libdd-php8of-batch-20260623T020845Z-alpha` - `libdd-php8of-batch-20260623T020845Z-bravo` - App-side responses returned `variant_2`. ### System Tests - Companion draft PR: DataDog/system-tests#7187 ### Staging End-To-End - Retriever staging query used `--datacenter us1.staging.dog --customer-auth=skip` against the `flagevaluation` track for the exact targeting keys above. - Retriever returned one backend row per targeting key, each with `flag.key=ffe-dogfooding-string-flag`, `variant.key=variant_2`, `allocation.key=allocation-override-392dd7c149f8`, and `evaluation_count=12`: - `libdd-php7-batch-20260623T020845Z-alpha`: `first_evaluation=1782180525878`, `last_evaluation=1782180525985`, `timestamp=1782180525985`, `evaluation_count=12` - `libdd-php7-batch-20260623T020845Z-bravo`: `first_evaluation=1782180525997`, `last_evaluation=1782180526103`, `timestamp=1782180526103`, `evaluation_count=12` - `libdd-php8of-batch-20260623T020845Z-alpha`: `first_evaluation=1782180526245`, `last_evaluation=1782180526344`, `timestamp=1782180526344`, `evaluation_count=12` - `libdd-php8of-batch-20260623T020845Z-bravo`: `first_evaluation=1782180526356`, `last_evaluation=1782180526451`, `timestamp=1782180526451`, `evaluation_count=12` --------- Co-authored-by: pr-shepherd-6ad11f[bot] <305617008+pr-shepherd-6ad11f[bot]@users.noreply.github.com> Co-authored-by: vickie.fridge <vickie.fridge@datadoghq.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent 5876427 commit 05d785e

20 files changed

Lines changed: 3274 additions & 101 deletions

File tree

.github/workflows/release-proposal-dispatch.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -433,7 +433,7 @@ jobs:
433433
run: |
434434
# Get commits since release for each crate and save to file
435435
"${WORKFLOW_SCRIPTS_ROOT}/commits-since-release.sh" "$(cat /tmp/crates.json)" > /tmp/commits-by-crate.json
436-
436+
437437
# Capture ephemeral release branch tip now. Use this in Release version bumps
438438
# so tag/merge-base resolution uses the same ref the script used.
439439
git rev-parse HEAD > /tmp/release_head_sha
@@ -1011,7 +1011,6 @@ jobs:
10111011
if [ "${#PR_TITLE}" -gt 100 ]; then
10121012
PR_TITLE="${PR_TITLE:0:97}..."
10131013
fi
1014-
10151014
if gh pr create \
10161015
--head "$BRANCH_NAME" \
10171016
--title "$PR_TITLE" \

Cargo.lock

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

datadog-sidecar-ffi/src/lib.rs

Lines changed: 297 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,15 @@ use datadog_sidecar::service::agent_info::AgentInfoReader;
2121
use datadog_sidecar::service::telemetry::InternalTelemetryAction;
2222
use datadog_sidecar::service::{
2323
blocking::{self, SidecarTransport},
24-
DynamicInstrumentationConfigState, FfeEvaluationMetric as SidecarFfeEvaluationMetric,
25-
FfeExposure as SidecarFfeExposure, FfeExposureBatch as SidecarFfeExposureBatch,
26-
FfeTelemetryContext as SidecarFfeTelemetryContext, InstanceId, QueueId, RuntimeMetadata,
27-
SerializedTracerHeaderTags, SessionConfig, SidecarAction, SidecarFlushOptions,
24+
AllocationKey, ContextDD, DynamicInstrumentationConfigState, EvalError,
25+
FfeEvaluationMetric as SidecarFfeEvaluationMetric, FfeExposure as SidecarFfeExposure,
26+
FfeExposureBatch as SidecarFfeExposureBatch,
27+
FfeFlagEvaluationBatch as SidecarFfeFlagEvaluationBatch,
28+
FfeFlagEvaluationEvent as SidecarFfeFlagEvaluationEvent,
29+
FfeTelemetryContext as SidecarFfeTelemetryContext, FlagEvalEventContext, FlagKey, InstanceId,
30+
QueueId, RuntimeMetadata, SerializedTracerHeaderTags, SessionConfig, SidecarAction,
31+
SidecarFlushOptions, TargetingRuleKey, VariantKey, MAX_CONTEXT_DEPTH, MAX_CONTEXT_FIELDS,
32+
MAX_FIELD_LENGTH,
2833
};
2934
use datadog_sidecar::service::{get_telemetry_action_sender, InternalTelemetryActions};
3035
use datadog_sidecar::shm_remote_config::{path_for_remote_config, RemoteConfigReader};
@@ -1216,6 +1221,25 @@ pub struct FfeEvaluationMetric<'a> {
12161221
pub allocation_key: CharSlice<'a>,
12171222
}
12181223

1224+
#[repr(C)]
1225+
pub struct FfeFlagEvaluation<'a> {
1226+
pub timestamp_ms: i64,
1227+
pub flag_key: CharSlice<'a>,
1228+
pub first_evaluation_ms: i64,
1229+
pub last_evaluation_ms: i64,
1230+
pub evaluation_count: u64,
1231+
pub variant: CharSlice<'a>,
1232+
pub allocation_key: CharSlice<'a>,
1233+
pub targeting_rule_key: CharSlice<'a>,
1234+
pub targeting_key: CharSlice<'a>,
1235+
/// UTF-8 JSON object. Empty, invalid, or non-object JSON is omitted. Object
1236+
/// values are pruned to 256 leaf fields, 256-byte string values, and four
1237+
/// levels of nested context depth.
1238+
pub evaluation_context_json: CharSlice<'a>,
1239+
pub error_message: CharSlice<'a>,
1240+
pub runtime_default_used: bool,
1241+
}
1242+
12191243
/// Send structured FFE exposure events to the sidecar. The sidecar owns
12201244
/// deduplication, JSON serialization, and Agent EVP delivery. This function is
12211245
/// caller-driven; shared libdatadog evaluator calls do not log unless an SDK
@@ -1287,6 +1311,78 @@ fn ddog_sidecar_send_ffe_exposure_batch_impl(
12871311
MaybeError::None
12881312
}
12891313

1314+
/// Send structured FFE flag evaluation events to the sidecar. The sidecar owns
1315+
/// JSON serialization and Agent EVP delivery. This function is caller-driven;
1316+
/// callers must aggregate and bound event cardinality before passing a batch.
1317+
///
1318+
/// # Safety
1319+
/// `context` and every element in `flag_evaluations` must contain valid UTF-8
1320+
/// `CharSlice` values. Empty `flag_evaluations` is a no-op.
1321+
#[no_mangle]
1322+
#[allow(clippy::missing_safety_doc)]
1323+
pub unsafe extern "C" fn ddog_sidecar_send_ffe_flag_evaluation_batch(
1324+
transport: &mut Box<SidecarTransport>,
1325+
instance_id: &InstanceId,
1326+
queue_id: &QueueId,
1327+
context: &FfeTelemetryContext<'_>,
1328+
flag_evaluations: Slice<FfeFlagEvaluation<'_>>,
1329+
) -> MaybeError {
1330+
std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
1331+
ddog_sidecar_send_ffe_flag_evaluation_batch_impl(
1332+
transport,
1333+
instance_id,
1334+
queue_id,
1335+
context,
1336+
flag_evaluations,
1337+
)
1338+
}))
1339+
.unwrap_or_else(|panic| {
1340+
MaybeError::Some(libdd_common_ffi::utils::handle_panic_error(
1341+
panic,
1342+
"ddog_sidecar_send_ffe_flag_evaluation_batch",
1343+
))
1344+
})
1345+
}
1346+
1347+
fn ddog_sidecar_send_ffe_flag_evaluation_batch_impl(
1348+
transport: &mut Box<SidecarTransport>,
1349+
instance_id: &InstanceId,
1350+
queue_id: &QueueId,
1351+
context: &FfeTelemetryContext<'_>,
1352+
flag_evaluations: Slice<FfeFlagEvaluation<'_>>,
1353+
) -> MaybeError {
1354+
let flag_evaluations = try_c!(flag_evaluations
1355+
.try_as_slice()
1356+
.map_err(|e| format!("Invalid flag evaluation slice: {e}")));
1357+
1358+
if flag_evaluations.is_empty() {
1359+
return MaybeError::None;
1360+
}
1361+
1362+
let context = try_c!(ffe_context_from_ffi(context));
1363+
let flag_evaluations = try_c!(flag_evaluations
1364+
.iter()
1365+
.map(|event| ffe_flag_evaluation_from_ffi(event, &context.service))
1366+
.collect::<Result<Vec<_>, _>>());
1367+
1368+
if flag_evaluations.is_empty() {
1369+
return MaybeError::None;
1370+
}
1371+
1372+
try_c!(blocking::enqueue_actions(
1373+
transport,
1374+
instance_id,
1375+
queue_id,
1376+
vec![SidecarAction::FfeFlagEvaluationBatch(
1377+
SidecarFfeFlagEvaluationBatch {
1378+
context,
1379+
flag_evaluations,
1380+
}
1381+
)],
1382+
));
1383+
MaybeError::None
1384+
}
1385+
12901386
/// Send structured FFE evaluation metric events to the sidecar. The sidecar
12911387
/// owns aggregation, OTLP/protobuf serialization, and OTLP HTTP delivery. This
12921388
/// function is caller-driven so SDKs with existing host-language hooks can
@@ -1351,6 +1447,107 @@ fn ffe_exposure_from_ffi(exposure: &FfeExposure<'_>) -> Result<SidecarFfeExposur
13511447
})
13521448
}
13531449

1450+
fn ffe_flag_evaluation_from_ffi(
1451+
event: &FfeFlagEvaluation<'_>,
1452+
service: &str,
1453+
) -> Result<SidecarFfeFlagEvaluationEvent, String> {
1454+
let evaluation = optional_json_object_string(event.evaluation_context_json)?;
1455+
let dd = (!service.is_empty()).then(|| ContextDD {
1456+
service: service.to_owned(),
1457+
});
1458+
let context = if evaluation.is_some() || dd.is_some() {
1459+
Some(FlagEvalEventContext { evaluation, dd })
1460+
} else {
1461+
None
1462+
};
1463+
1464+
Ok(SidecarFfeFlagEvaluationEvent {
1465+
timestamp: event.timestamp_ms,
1466+
flag: FlagKey {
1467+
key: char_slice_to_string(event.flag_key)?,
1468+
},
1469+
first_evaluation: event.first_evaluation_ms,
1470+
last_evaluation: event.last_evaluation_ms,
1471+
evaluation_count: event.evaluation_count,
1472+
variant: optional_string(event.variant)?.map(|key| VariantKey { key }),
1473+
allocation: optional_string(event.allocation_key)?.map(|key| AllocationKey { key }),
1474+
targeting_rule: optional_string(event.targeting_rule_key)?
1475+
.map(|key| TargetingRuleKey { key }),
1476+
targeting_key: optional_string(event.targeting_key)?,
1477+
context,
1478+
error: optional_string(event.error_message)?.map(|message| EvalError { message }),
1479+
runtime_default_used: event.runtime_default_used,
1480+
})
1481+
}
1482+
1483+
fn prune_evaluation_context_json(value: serde_json::Value) -> Option<String> {
1484+
let serde_json::Value::Object(attrs) = value else {
1485+
return None;
1486+
};
1487+
1488+
let mut remaining_fields = MAX_CONTEXT_FIELDS;
1489+
let pruned = prune_context_object(&attrs, 1, &mut remaining_fields);
1490+
Some(serde_json::Value::Object(pruned).to_string())
1491+
}
1492+
1493+
fn prune_context_object(
1494+
attrs: &serde_json::Map<String, serde_json::Value>,
1495+
depth: usize,
1496+
remaining_fields: &mut usize,
1497+
) -> serde_json::Map<String, serde_json::Value> {
1498+
let mut keys: Vec<_> = attrs.keys().collect();
1499+
keys.sort();
1500+
1501+
let mut pruned = serde_json::Map::new();
1502+
for key in keys {
1503+
if *remaining_fields == 0 {
1504+
break;
1505+
}
1506+
let Some(value) = attrs
1507+
.get(key)
1508+
.and_then(|value| prune_context_value(value, depth, remaining_fields))
1509+
else {
1510+
continue;
1511+
};
1512+
pruned.insert(key.clone(), value);
1513+
}
1514+
pruned
1515+
}
1516+
1517+
fn prune_context_value(
1518+
value: &serde_json::Value,
1519+
depth: usize,
1520+
remaining_fields: &mut usize,
1521+
) -> Option<serde_json::Value> {
1522+
match value {
1523+
serde_json::Value::String(s) if s.len() > MAX_FIELD_LENGTH => None,
1524+
serde_json::Value::Object(attrs) => {
1525+
if depth >= MAX_CONTEXT_DEPTH {
1526+
return None;
1527+
}
1528+
let pruned = prune_context_object(attrs, depth + 1, remaining_fields);
1529+
(!pruned.is_empty()).then_some(serde_json::Value::Object(pruned))
1530+
}
1531+
serde_json::Value::Array(values) => {
1532+
if depth >= MAX_CONTEXT_DEPTH {
1533+
return None;
1534+
}
1535+
let pruned: Vec<_> = values
1536+
.iter()
1537+
.filter_map(|value| prune_context_value(value, depth + 1, remaining_fields))
1538+
.collect();
1539+
(!pruned.is_empty()).then_some(serde_json::Value::Array(pruned))
1540+
}
1541+
_ => {
1542+
if *remaining_fields == 0 {
1543+
return None;
1544+
}
1545+
*remaining_fields -= 1;
1546+
Some(value.clone())
1547+
}
1548+
}
1549+
}
1550+
13541551
fn ffe_metric_from_ffi(
13551552
metric: &FfeEvaluationMetric<'_>,
13561553
) -> Result<SidecarFfeEvaluationMetric, String> {
@@ -1371,6 +1568,17 @@ fn optional_string(slice: CharSlice) -> Result<Option<String>, String> {
13711568
}
13721569
}
13731570

1571+
fn optional_json_object_string(slice: CharSlice) -> Result<Option<String>, String> {
1572+
let Some(raw) = optional_string(slice)? else {
1573+
return Ok(None);
1574+
};
1575+
let value = match serde_json::from_str::<serde_json::Value>(&raw) {
1576+
Ok(value) => value,
1577+
Err(_) => return Ok(None),
1578+
};
1579+
Ok(prune_evaluation_context_json(value))
1580+
}
1581+
13741582
#[no_mangle]
13751583
#[allow(clippy::missing_safety_doc)]
13761584
#[allow(improper_ctypes_definitions)] // DebuggerPayload is just a pointer, we hide its internals
@@ -1792,6 +2000,23 @@ mod tests {
17922000
use super::*;
17932001
use std::borrow::Cow;
17942002

2003+
fn ffi_flag_evaluation<'a>(evaluation_context_json: &'a str) -> FfeFlagEvaluation<'a> {
2004+
FfeFlagEvaluation {
2005+
timestamp_ms: 1_700_000_000_000,
2006+
flag_key: CharSlice::from("flag-a"),
2007+
first_evaluation_ms: 1_699_999_000_000,
2008+
last_evaluation_ms: 1_700_000_000_000,
2009+
evaluation_count: 7,
2010+
variant: CharSlice::empty(),
2011+
allocation_key: CharSlice::empty(),
2012+
targeting_rule_key: CharSlice::empty(),
2013+
targeting_key: CharSlice::empty(),
2014+
evaluation_context_json: CharSlice::from(evaluation_context_json),
2015+
error_message: CharSlice::empty(),
2016+
runtime_default_used: false,
2017+
}
2018+
}
2019+
17952020
#[test]
17962021
fn otlp_metrics_endpoint_inherits_agent_test_token_when_missing() {
17972022
let agent_endpoint = Endpoint {
@@ -1825,4 +2050,72 @@ mod tests {
18252050

18262051
assert_eq!(endpoint.test_token.as_deref(), Some("metrics-token"));
18272052
}
2053+
2054+
#[test]
2055+
fn ffe_flag_evaluation_preserves_service_without_evaluation_context() {
2056+
let event = ffi_flag_evaluation("");
2057+
2058+
let converted = ffe_flag_evaluation_from_ffi(&event, "checkout").unwrap();
2059+
let context = converted.context.expect("service attribution must remain");
2060+
2061+
assert!(context.evaluation.is_none());
2062+
assert_eq!(
2063+
context.dd.expect("dd context must be present").service,
2064+
"checkout"
2065+
);
2066+
}
2067+
2068+
#[test]
2069+
fn ffe_flag_evaluation_prunes_context_field_count_and_long_strings() {
2070+
let mut attrs = serde_json::Map::new();
2071+
attrs.insert(
2072+
"aaa_long".to_owned(),
2073+
serde_json::Value::String("x".repeat(MAX_FIELD_LENGTH + 1)),
2074+
);
2075+
for index in 0..=MAX_CONTEXT_FIELDS {
2076+
attrs.insert(format!("field_{index:03}"), serde_json::json!(index));
2077+
}
2078+
let raw = serde_json::Value::Object(attrs).to_string();
2079+
let event = ffi_flag_evaluation(&raw);
2080+
2081+
let converted = ffe_flag_evaluation_from_ffi(&event, "checkout").unwrap();
2082+
let context = converted.context.expect("context must be present");
2083+
let evaluation = context.evaluation.expect("evaluation context must remain");
2084+
let value: serde_json::Value = serde_json::from_str(&evaluation).unwrap();
2085+
let attrs = value.as_object().unwrap();
2086+
2087+
assert_eq!(attrs.len(), MAX_CONTEXT_FIELDS);
2088+
assert!(!attrs.contains_key("aaa_long"));
2089+
assert!(attrs.contains_key("field_000"));
2090+
assert!(attrs.contains_key(&format!("field_{:03}", MAX_CONTEXT_FIELDS - 1)));
2091+
assert!(!attrs.contains_key(&format!("field_{MAX_CONTEXT_FIELDS:03}")));
2092+
}
2093+
2094+
#[test]
2095+
fn ffe_flag_evaluation_prunes_context_beyond_depth_four() {
2096+
let raw = serde_json::json!({
2097+
"a": {
2098+
"b": {
2099+
"c": {
2100+
"d": "kept",
2101+
"too_deep": {
2102+
"e": "dropped"
2103+
}
2104+
}
2105+
}
2106+
},
2107+
"top": true
2108+
})
2109+
.to_string();
2110+
let event = ffi_flag_evaluation(&raw);
2111+
2112+
let converted = ffe_flag_evaluation_from_ffi(&event, "checkout").unwrap();
2113+
let context = converted.context.expect("context must be present");
2114+
let evaluation = context.evaluation.expect("evaluation context must remain");
2115+
let value: serde_json::Value = serde_json::from_str(&evaluation).unwrap();
2116+
2117+
assert_eq!(value["a"]["b"]["c"]["d"], "kept");
2118+
assert!(value["a"]["b"]["c"].get("too_deep").is_none());
2119+
assert_eq!(value["top"], true);
2120+
}
18282121
}

datadog-sidecar/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ libdd-trace-utils = { path = "../libdd-trace-utils" }
2929
libdd-trace-stats = { path = "../libdd-trace-stats", features = ["telemetry", "dogstatsd"] }
3030
libdd-remote-config = { path = "../libdd-remote-config" }
3131
datadog-live-debugger = { path = "../datadog-live-debugger" }
32-
libdd-ffe = { path = "../libdd-ffe", features = ["exposure-events", "evaluation-metrics"] }
32+
libdd-ffe = { path = "../libdd-ffe", features = ["exposure-events", "evaluation-metrics", "flagevaluation-evp"] }
3333
libdd-crashtracker = { path = "../libdd-crashtracker" }
3434
libdd-dogstatsd-client = { path = "../libdd-dogstatsd-client" }
3535
libdd-tinybytes = { path = "../libdd-tinybytes" }

0 commit comments

Comments
 (0)