Skip to content

feat(openfeature): route agentless exposure events - #9527

Draft
leoromanovsky wants to merge 3 commits into
masterfrom
ffl-2697-nodejs-serverless-evp-routing
Draft

feat(openfeature): route agentless exposure events#9527
leoromanovsky wants to merge 3 commits into
masterfrom
ffl-2697-nodejs-serverless-evp-routing

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Motivation

  • Business goal: standalone exposure delivery for the Datadog OpenFeature client.
  • Existing behavior: OpenFeature uses /info and Agent EVP proxy v2.
  • Current gap: no local receiver disables exposure delivery.
  • Required behavior: optional discovery with authenticated direct EVP fallback.
  • Scope: exposure events only. serverless-init remains optional until trace delivery requires it.
flowchart TB
  Evaluation["OpenFeature evaluation"] --> Cache["Exposure cache"]
  Cache --> Writer["Exposure writer"]

  Writer -->|"Remote Configuration"| Required["Required /info discovery"]
  Required --> Agent["Agent EVP proxy v2"]

  Writer -->|"Agentless"| Optional["Optional /info discovery"]
  Optional -->|"EVP v4 or v2 found"| Receiver["Local receiver<br/>serverless-init or Agent"]
  Optional -. "No listener or compatible route" .-> Direct["Direct HTTPS EVP<br/>with DD_API_KEY"]
  Receiver -. "Refusal, 403, 404, or 405" .-> Direct

  Agent --> Intake["Exposure intake"]
  Receiver --> Intake
  Direct --> Intake
Loading

Changes and Decisions

  • Use the shared EVP route contract from refactor(exporters): add shared EVP proxy utilities #9738.
  • Remote Configuration: preserve Agent EVP proxy v2.
  • Agentless local route: prefer EVP v4, then EVP v2.
  • Agentless direct route: use DD_API_KEY when discovery fails or finds no compatible route.
  • Safe fallback: retry the same batch after refusal, 403, 404, or 405.
  • Ambiguous result: do not replay after timeout, reset, 429, or 5xx.
  • Routing header: do not require the Agent-consumed subdomain header in evp_proxy_allowed_headers.
  • Direct proxy support: honor standard HTTPS proxy environment variables.
  • Delivery behavior: preserve the startup buffer and exposure cache.

Validation

  • System-test PR #7478 is restacked on #7477.
  • Shared contract at 88ec90d5e ran locally against this PR at aa02379d90.
  • Agent: TEST_LIBRARY=nodejs ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_exposure_egress.py passed one test.
  • Direct: TEST_LIBRARY=nodejs ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS_DIRECT tests/ffe/test_exposure_egress.py passed one test.
  • Sidecar: TEST_LIBRARY=nodejs ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS_SERVERLESS tests/ffe/test_exposure_egress.py passed one test.
  • Each route ran the same inherited test. Five identical evaluations produced one exposure.
  • The sidecar route used serverless-init:1.9.13 and received HTTP 202.
  • The direct route had no local receiver and received HTTP 202.
  • Each agentless route produced no exposure on the unused route.
  • The no-Agent ffe-dogfooding profile reached CDN readiness and sent one direct exposure.

@dd-octo-sts

dd-octo-sts Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 7.99 MB
Deduped: 8.77 MB
No deduping: 8.77 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | https-proxy-agent | 7.0.6 | 27.71 kB | 101.18 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB | | proxy-from-env | 2.1.0 | 15.64 kB | 15.64 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jul 25, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 98.56% (+0.02%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: aa02379 | Docs | View more details | Give us feedback!

@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.56%. Comparing base (d820eae) to head (aa02379).
⚠️ Report is 83 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9527      +/-   ##
==========================================
+ Coverage   98.55%   98.56%   +0.01%     
==========================================
  Files         966      970       +4     
  Lines      139382   139902     +520     
  Branches    12055    12492     +437     
==========================================
+ Hits       137369   137896     +527     
+ Misses       2013     2006       -7     
Flag Coverage Δ
aiguard 57.00% <100.00%> (+0.03%) ⬆️
aiguard-integration 55.85% <100.00%> (-0.02%) ⬇️
apm-bucket-0 57.17% <100.00%> (-0.04%) ⬇️
apm-bucket-1 63.31% <100.00%> (-0.04%) ⬇️
apm-bucket-2 62.16% <100.00%> (-0.04%) ⬇️
apm-bucket-3 59.74% <100.00%> (-0.04%) ⬇️
apm-capabilities-tracing 62.31% <100.00%> (-0.01%) ⬇️
apm-integrations-aerospike 56.21% <100.00%> (-0.04%) ⬇️
apm-integrations-confluentinc-kafka-javascript 61.14% <100.00%> (-0.04%) ⬇️
apm-integrations-couchbase 56.65% <100.00%> (-0.04%) ⬇️
apm-integrations-http 61.85% <100.00%> (-0.04%) ⬇️
apm-integrations-kafkajs 61.66% <100.00%> (-0.04%) ⬇️
apm-integrations-next 59.35% <100.00%> (-0.04%) ⬇️
apm-integrations-prisma 58.46% <100.00%> (-0.04%) ⬇️
appsec 72.07% <100.00%> (-0.06%) ⬇️
appsec-express_fastify_graphql 69.40% <100.00%> (-0.03%) ⬇️
appsec-integration 50.16% <100.00%> (-0.01%) ⬇️
appsec-kafka_ldapjs_lodash 63.37% <100.00%> (-0.04%) ⬇️
appsec-mongodb-core_mongoose_mysql 66.84% <100.00%> (-0.03%) ⬇️
appsec-next 56.61% <100.00%> (-0.04%) ⬇️
appsec-node-serialize_passport_postgres 66.25% <100.00%> (-0.03%) ⬇️
appsec-sourcing_stripe_template 64.68% <100.00%> (-0.03%) ⬇️
debugger 64.22% <100.00%> (-0.03%) ⬇️
instrumentations-bucket-0 51.64% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-1 59.34% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-10 60.83% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-11 61.56% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-12 51.65% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-13 52.39% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-14 51.60% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-2 53.09% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-3 53.45% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-4 58.71% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-5 50.26% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-6 60.45% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-7 58.05% <100.00%> (-0.04%) ⬇️
instrumentations-bucket-8 59.03% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-9 57.39% <100.00%> (-0.04%) ⬇️
instrumentations-instrumentation-couchbase 50.90% <100.00%> (-0.05%) ⬇️
instrumentations-integration-esbuild 34.21% <0.00%> (-0.03%) ⬇️
llmobs-ai_anthropic_bedrock 62.80% <100.00%> (-0.03%) ⬇️
llmobs-bucket-1 61.30% <100.00%> (-0.03%) ⬇️
llmobs-openai 61.70% <100.00%> (-0.03%) ⬇️
llmobs-openai-agents_vertex-ai 59.98% <100.00%> (-0.03%) ⬇️
llmobs-sdk 66.74% <100.00%> (-0.04%) ⬇️
openfeature 56.00% <81.60%> (+0.33%) ⬆️
openfeature-unit 53.92% <100.00%> (+0.71%) ⬆️
platform-core_esbuild_instrumentations-misc 41.22% <100.00%> (-0.03%) ⬇️
platform-integration 60.40% <100.00%> (-0.02%) ⬇️
platform-shimmer_unit-guardrails_webpack 38.88% <100.00%> (-0.03%) ⬇️
plugins-bucket-0 56.87% <100.00%> (-0.04%) ⬇️
plugins-bucket-1 54.00% <100.00%> (-0.02%) ⬇️
plugins-bucket-11 61.42% <100.00%> (-0.04%) ⬇️
plugins-bucket-17 61.24% <100.00%> (-0.04%) ⬇️
plugins-bucket-18 61.88% <100.00%> (-0.04%) ⬇️
plugins-bucket-19 61.26% <100.00%> (-0.04%) ⬇️
plugins-bucket-20 63.68% <100.00%> (-0.04%) ⬇️
plugins-bucket-4 58.26% <100.00%> (-0.04%) ⬇️
plugins-bullmq_cassandra_cookie 61.32% <100.00%> (-0.04%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 56.30% <100.00%> (-0.04%) ⬇️
plugins-fetch_fs_generic-pool 58.16% <100.00%> (-0.08%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 64.10% <100.00%> (-0.04%) ⬇️
plugins-hapi_hono_ioredis 59.85% <100.00%> (-0.04%) ⬇️
plugins-knex_langgraph_ldapjs 55.01% <100.00%> (-0.04%) ⬇️
plugins-light-my-request_limitd-client_lodash 58.31% <100.00%> (-0.05%) ⬇️
plugins-mariadb_memcached_mercurius 61.25% <100.00%> (-0.04%) ⬇️
plugins-mongodb_mongodb-core_mongoose 59.21% <100.00%> (-0.04%) ⬇️
plugins-multer_mysql_mysql2 58.79% <100.00%> (-0.04%) ⬇️
plugins-nats_node-serialize_opensearch 60.34% <100.00%> (-0.04%) ⬇️
plugins-passport-http_pino_postgres 58.54% <100.00%> (-0.04%) ⬇️
plugins-process_pug_redis 57.34% <100.00%> (-0.04%) ⬇️
plugins-undici_url_valkey 57.97% <100.00%> (-0.04%) ⬇️
plugins-vm_winston_ws 59.54% <100.00%> (-0.04%) ⬇️
profiling 61.46% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-aws-sdk 55.08% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-base-inject-field 50.88% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-bedrockruntime 54.61% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-client 56.17% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-dynamodb 55.44% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-eventbridge 49.68% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-kinesis 59.03% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-lambda 57.18% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-s3 55.54% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-serverless-peer-service 59.29% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-sns 59.84% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-sqs 60.25% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-stepfunctions 55.37% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-util 51.41% <100.00%> (-0.05%) ⬇️
serverless-bucket-0 54.05% <100.00%> (-0.02%) ⬇️
serverless-bucket-1 58.80% <100.00%> (-0.04%) ⬇️
test-optimization-cucumber 71.00% <100.00%> (-0.03%) ⬇️
test-optimization-cypress 64.84% <100.00%> (+0.09%) ⬆️
test-optimization-jest 72.41% <100.00%> (-0.01%) ⬇️
test-optimization-mocha 72.08% <100.00%> (+0.04%) ⬆️
test-optimization-playwright-playwright-atr 59.87% <100.00%> (+<0.01%) ⬆️
test-optimization-playwright-playwright-efd 60.00% <83.33%> (+<0.01%) ⬆️
test-optimization-playwright-playwright-final-status 60.24% <83.33%> (+0.07%) ⬆️
test-optimization-playwright-playwright-impacted-tests 59.70% <83.33%> (+0.15%) ⬆️
test-optimization-playwright-playwright-reporting 60.87% <100.00%> (-0.10%) ⬇️
test-optimization-playwright-playwright-test-management 60.68% <83.33%> (-0.10%) ⬇️
test-optimization-playwright-playwright-test-span 59.87% <100.00%> (-0.11%) ⬇️
test-optimization-selenium 59.06% <100.00%> (-0.14%) ⬇️
test-optimization-testopt 57.61% <100.00%> (+0.06%) ⬆️
test-optimization-vitest 73.29% <100.00%> (+0.04%) ⬆️
test-optimization-vitest-browser 58.94% <83.33%> (+<0.01%) ⬆️
test-optimization-webdriverio 65.42% <83.33%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@leoromanovsky
leoromanovsky changed the base branch from master to leo.romanovsky/ffl-2446-evp-flagevaluation-nodejs July 28, 2026 04:48
@leoromanovsky
leoromanovsky changed the base branch from leo.romanovsky/ffl-2446-evp-flagevaluation-nodejs to master July 28, 2026 04:49
@leoromanovsky
leoromanovsky force-pushed the ffl-2697-nodejs-serverless-evp-routing branch from 3915c1e to 0f2b357 Compare July 28, 2026 05:10
@pr-commenter

pr-commenter Bot commented Jul 28, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-08 15:44:54

Comparing candidate commit aa02379 in PR branch ffl-2697-nodejs-serverless-evp-routing with baseline commit 59ba684 in branch leo.romanovsky/evp-proxy-utilities.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2317 metrics, 41 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-214.634ms; +209.741ms] or [-7.943%; +7.762%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-234.129ms; +229.840ms] or [-9.015%; +8.850%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-162.084ms; +154.252ms] or [-5.201%; +4.950%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-188129.393µs; +186802.926µs] or [-6.399%; +6.354%]

scenario:appsec-control-20

  • unstable execution_time [-125755.147µs; +124564.547µs] or [-7.580%; +7.508%]

scenario:appsec-control-24

  • unstable execution_time [-115078.296µs; +116752.730µs] or [-9.175%; +9.308%]

scenario:appsec-control-26

  • unstable execution_time [-126678.364µs; +127103.264µs] or [-10.096%; +10.130%]

scenario:appsec-iast-no-vulnerability-control-20

  • unstable execution_time [-12626.362µs; +13561.429µs] or [-4.923%; +5.288%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-9.394ms; +19.390ms] or [-3.610%; +7.451%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-14.209ms; +16.956ms] or [-5.513%; +6.579%]

scenario:appsec-iast-with-vulnerability-control-20

  • unstable execution_time [-26.135ms; +30.160ms] or [-4.786%; +5.523%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-28.908ms; +26.606ms] or [-5.258%; +4.839%]

scenario:child_process-shell-string-24

  • unstable execution_time [-14.381ms; +19.816ms] or [-4.397%; +6.059%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-1598.671ms; +487.209ms] or [-19.795%; +6.033%]
  • unstable execution_time [-1619.460ms; +494.278ms] or [-18.454%; +5.633%]
  • unstable instructions [-13.7G instructions; +4.4G instructions] or [-20.787%; +6.639%]
  • unstable throughput [-139.874op/s; +472.618op/s] or [-3.731%; +12.605%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-3110.221ms; +2729.491ms] or [-34.531%; +30.304%]
  • unstable execution_time [-3135.715ms; +2751.432ms] or [-32.236%; +28.286%]
  • unstable instructions [-26.4G instructions; +23.1G instructions] or [-35.737%; +31.311%]
  • unstable max_rss_usage [-9698.766KB; +10975.566KB] or [-6.016%; +6.808%]
  • unstable throughput [-727.621op/s; +874.843op/s] or [-20.936%; +25.171%]

scenario:debugger-line-probe-without-snapshot-24

  • unstable cpu_user_time [-3741.098ms; +2505.911ms] or [-39.754%; +26.628%]
  • unstable execution_time [-3802.850ms; +2541.827ms] or [-37.580%; +25.118%]
  • unstable instructions [-32.1G instructions; +21.2G instructions] or [-41.559%; +27.497%]
  • unstable throughput [-670.147op/s; +1035.241op/s] or [-19.722%; +30.467%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-3726.612ms; +3701.716ms] or [-36.107%; +35.866%]
  • unstable execution_time [-3719.120ms; +3703.371ms] or [-33.593%; +33.450%]
  • unstable instructions [-32844.6M instructions; +32660.7M instructions] or [-37.946%; +37.733%]
  • unstable max_rss_usage [-12655.612KB; +12041.212KB] or [-7.837%; +7.457%]
  • unstable throughput [-721.682op/s; +729.379op/s] or [-23.488%; +23.739%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-328.671ms; +285.500ms] or [-6.792%; +5.900%]
  • unstable execution_time [-327.371ms; +280.277ms] or [-6.662%; +5.704%]
  • unstable throughput [-100656.287op/s; +112701.059op/s] or [-5.898%; +6.604%]

scenario:plugin-graphql-long-with-depth-off-26

  • unstable max_rss_usage [-12.551MB; +51.480MB] or [-6.985%; +28.649%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable execution_time [-618.959ms; +573.431ms] or [-5.226%; +4.842%]
  • unstable throughput [-3.336op/s; +3.593op/s] or [-4.911%; +5.289%]

scenario:plugin-graphql-long-with-depth-on-max-26

  • unstable max_rss_usage [-15.159MB; +8.564MB] or [-6.643%; +3.753%]

scenario:plugin-pg-service-26

  • unstable cpu_user_time [-23.738ms; +75.458ms] or [-2.664%; +8.467%]
  • unstable execution_time [-24.261ms; +75.230ms] or [-2.705%; +8.388%]
  • unstable throughput [-521405.813op/s; +161067.808op/s] or [-7.699%; +2.378%]

@leoromanovsky leoromanovsky changed the title feat(openfeature): route serverless Feature Flags EVP telemetry feat(openfeature): route agentless exposure events Aug 7, 2026
Add shared EVP protocol constants, route discovery, and path construction.\n\nRefactor the existing Feature Flags Agent path to use the shared utilities. Add migration pointers for CI Visibility and LLM Observability. Document the Agent-compatible receiver contract.
@leoromanovsky
leoromanovsky changed the base branch from master to leo.romanovsky/evp-proxy-utilities August 8, 2026 04:59
@leoromanovsky
leoromanovsky force-pushed the ffl-2697-nodejs-serverless-evp-routing branch from 7b40ff8 to 3b591e2 Compare August 8, 2026 05:32
Base automatically changed from leo.romanovsky/evp-proxy-utilities to master August 18, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant