Skip to content

fix(openfeature): isolate malformed flag configuration - #9775

Draft
leoromanovsky wants to merge 10 commits into
masterfrom
agent/isolate-malformed-flags
Draft

fix(openfeature): isolate malformed flag configuration#9775
leoromanovsky wants to merge 10 commits into
masterfrom
agent/isolate-malformed-flags

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Motivation

Malformed UFC flags must be isolated during ingestion so valid siblings still evaluate, while rejected keys return a parse error instead of looking absent.

Changes

  • integrate the canonical fixture submodule and pin it to ea8b5cc5ce335109f11f3efbc5fd608f98a3ca54
  • validate and isolate flags independently
  • return caller defaults with ERROR/PARSE_ERROR for rejected keys
  • preserve upstream FLAG_NOT_FOUND for absent keys
  • add canonical error-code and atomic refresh coverage

Decisions

  • preserve the upstream evaluator's allocation, metadata, and exposure behavior
  • transform SemVer rules at ingestion so canonical precedence ignores build metadata
  • replace active and rejected state together on refresh
  • keep the separate regex-fixture proposal in ffe-system-test-data#21 out of scope

Validation

  • full provider spec: 287 passing
  • targeted ESLint on edited JavaScript files
  • changed-file type filter has no introduced errors; two existing EvalMetricsHook/ConfigBase errors remain
  • git diff --check

@dd-octo-sts

dd-octo-sts Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 8 MB
Deduped: 8.66 MB
No deduping: 8.66 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Aug 11, 2026 •
edited by datadog-prod-us1-6 Bot
Loading

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 98.56% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a7df7df | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Aug 11, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-11 16:51:42

Comparing candidate commit a7df7df in PR branch agent/isolate-malformed-flags with baseline commit 77cd463 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2319 metrics, 39 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-206.862ms; +212.243ms] or [-7.629%; +7.827%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-245.541ms; +227.221ms] or [-9.412%; +8.710%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-167.049ms; +153.798ms] or [-5.350%; +4.925%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-182.483ms; +187.683ms] or [-6.171%; +6.347%]

scenario:appsec-control-20

  • unstable execution_time [-118.973ms; +126.388ms] or [-7.155%; +7.601%]

scenario:appsec-control-24

  • unstable execution_time [-114.121ms; +119.871ms] or [-9.114%; +9.573%]

scenario:appsec-control-26

  • unstable execution_time [-125.935ms; +122.456ms] or [-10.053%; +9.775%]

scenario:appsec-iast-no-vulnerability-control-20

  • unstable execution_time [-12.811ms; +16.712ms] or [-5.043%; +6.579%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-17.525ms; +12.570ms] or [-6.679%; +4.791%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-16.017ms; +13.597ms] or [-6.301%; +5.349%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-29.361ms; +27.273ms] or [-5.352%; +4.971%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-33.013ms; +36.870ms] or [-5.963%; +6.660%]

scenario:child_process-shell-string-24

  • unstable cpu_usage_percentage [-6.867%; +4.105%]
  • unstable execution_time [-17.602ms; +30.653ms] or [-5.450%; +9.490%]
  • unstable throughput [-264477.866op/s; +128645.667op/s] or [-7.297%; +3.549%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-2612.043ms; +2666.830ms] or [-29.979%; +30.607%]
  • unstable execution_time [-2606.666ms; +2664.656ms] or [-27.663%; +28.279%]
  • unstable instructions [-22368.3M instructions; +22748.7M instructions] or [-31.442%; +31.977%]
  • unstable max_rss_usage [-9911.608KB; +10313.208KB] or [-6.165%; +6.414%]
  • unstable throughput [-749.222op/s; +727.708op/s] or [-21.043%; +20.438%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-2073.403ms; +2998.315ms] or [-24.977%; +36.119%]
  • unstable execution_time [-2058.485ms; +2987.264ms] or [-22.829%; +33.129%]
  • unstable instructions [-18.1G instructions; +25.8G instructions] or [-26.712%; +38.173%]
  • unstable max_rss_usage [-8.111MB; +11.851MB] or [-5.111%; +7.467%]
  • unstable throughput [-814.594op/s; +588.059op/s] or [-22.278%; +16.083%]

scenario:debugger-line-probe-with-snapshot-minimal-26

  • unstable cpu_user_time [-2326.281ms; +783.376ms] or [-24.337%; +8.196%]
  • unstable execution_time [-2368.344ms; +848.515ms] or [-23.002%; +8.241%]
  • unstable instructions [-20.5G instructions; +6.8G instructions] or [-25.752%; +8.502%]
  • unstable throughput [-175.484op/s; +447.460op/s] or [-5.436%; +13.860%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-325.858ms; +396.472ms] or [-6.857%; +8.343%]
  • unstable execution_time [-326.365ms; +398.348ms] or [-6.762%; +8.254%]
  • unstable throughput [-141277.857op/s; +117961.624op/s] or [-8.115%; +6.776%]

scenario:llmobs-encode-unicode-ascii-24

  • unstable execution_time [-54.347ms; +82.217ms] or [-4.379%; +6.625%]

scenario:plugin-graphql-long-with-depth-and-collapse-off-20

  • unstable max_rss_usage [-23671.209KB; +22074.156KB] or [-5.987%; +5.583%]

scenario:plugin-graphql-long-with-depth-off-26

  • unstable max_rss_usage [-38.633MB; +28.753MB] or [-19.757%; +14.704%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable execution_time [-609.946ms; +572.423ms] or [-5.177%; +4.858%]
  • unstable throughput [-3.349op/s; +3.589op/s] or [-4.905%; +5.257%]

scenario:plugin-memcached-hashring-26

  • unstable cpu_user_time [-107.692ms; +165.423ms] or [-4.490%; +6.898%]
  • unstable execution_time [-109.410ms; +164.252ms] or [-4.438%; +6.662%]

@codecov

codecov Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.56%. Comparing base (77cd463) to head (a7df7df).

Additional details and impacted files 
@@           Coverage Diff            @@
##           master    #9775    +/-   ##
========================================
  Coverage   98.55%   98.56%            
========================================
  Files         966      967     +1     
  Lines      139420   139858   +438     
  Branches    12543    11913   -630     
========================================
+ Hits       137407   137846   +439     
+ Misses       2013     2012     -1
Flag Coverage Δ
aiguard 56.98% <ø> (-0.03%) ⬇️
aiguard-integration 55.88% <ø> (ø)
apm-bucket-0 57.23% <ø> (-0.03%) ⬇️
apm-bucket-1 63.36% <ø> (-0.02%) ⬇️
apm-bucket-2 62.20% <ø> (-0.03%) ⬇️
apm-bucket-3 59.79% <ø> (-0.03%) ⬇️
apm-capabilities-tracing 62.32% <ø> (ø)
apm-integrations-aerospike 56.27% <ø> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript 61.19% <ø> (-0.03%) ⬇️
apm-integrations-couchbase 56.70% <ø> (-0.03%) ⬇️
apm-integrations-http 61.90% <ø> (-0.02%) ⬇️
apm-integrations-kafkajs 61.71% <ø> (-0.03%) ⬇️
apm-integrations-next 59.40% <ø> (-0.03%) ⬇️
apm-integrations-prisma 58.51% <ø> (-0.02%) ⬇️
appsec 72.09% <ø> (-0.02%) ⬇️
appsec-express_fastify_graphql 69.42% <ø> (-0.02%) ⬇️
appsec-integration 50.17% <ø> (ø)
appsec-kafka_ldapjs_lodash 63.40% <ø> (-0.02%) ⬇️
appsec-mongodb-core_mongoose_mysql 66.86% <ø> (-0.02%) ⬇️
appsec-next 56.66% <ø> (-0.02%) ⬇️
appsec-node-serialize_passport_postgres 66.28% <ø> (-0.02%) ⬇️
appsec-sourcing_stripe_template 64.71% <ø> (-0.02%) ⬇️
debugger 64.26% <ø> (+<0.01%) ⬆️
instrumentations-bucket-0 51.70% <ø> (-0.03%) ⬇️
instrumentations-bucket-1 59.39% <ø> (-0.03%) ⬇️
instrumentations-bucket-10 60.87% <ø> (-0.02%) ⬇️
instrumentations-bucket-11 61.60% <ø> (-0.02%) ⬇️
instrumentations-bucket-12 51.72% <ø> (-0.03%) ⬇️
instrumentations-bucket-13 52.45% <ø> (-0.03%) ⬇️
instrumentations-bucket-14 51.67% <ø> (-0.03%) ⬇️
instrumentations-bucket-2 53.15% <ø> (-0.03%) ⬇️
instrumentations-bucket-3 53.50% <ø> (-0.03%) ⬇️
instrumentations-bucket-4 58.76% <ø> (-0.03%) ⬇️
instrumentations-bucket-5 50.32% <ø> (-0.02%) ⬇️
instrumentations-bucket-6 60.50% <ø> (-0.03%) ⬇️
instrumentations-bucket-7 58.10% <ø> (-0.03%) ⬇️
instrumentations-bucket-8 59.08% <ø> (-0.02%) ⬇️
instrumentations-bucket-9 57.44% <ø> (-0.03%) ⬇️
instrumentations-instrumentation-couchbase 50.97% <ø> (-0.03%) ⬇️
instrumentations-integration-esbuild 34.24% <ø> (ø)
llmobs-ai_anthropic_bedrock 62.84% <ø> (-0.02%) ⬇️
llmobs-bucket-1 61.35% <ø> (-0.02%) ⬇️
llmobs-openai 61.74% <ø> (-0.02%) ⬇️
llmobs-openai-agents_vertex-ai 60.03% <ø> (-0.02%) ⬇️
llmobs-sdk 66.78% <ø> (-0.03%) ⬇️
master-coverage 98.56% <100.00%> (?)
openfeature 55.70% <56.84%> (+0.01%) ⬆️
openfeature-unit 53.74% <100.00%> (+0.47%) ⬆️
platform-core_esbuild_instrumentations-misc 41.26% <ø> (-0.02%) ⬇️
platform-integration 60.43% <ø> (ø)
platform-shimmer_unit-guardrails_webpack 38.92% <ø> (-0.02%) ⬇️
plugins-bucket-0 56.93% <ø> (-0.02%) ⬇️
plugins-bucket-1 54.03% <ø> (ø)
plugins-bucket-11 61.47% <ø> (-0.03%) ⬇️
plugins-bucket-17 61.28% <ø> (-0.03%) ⬇️
plugins-bucket-18 61.92% <ø> (-0.02%) ⬇️
plugins-bucket-19 61.31% <ø> (-0.03%) ⬇️
plugins-bucket-20 63.72% <ø> (-0.03%) ⬇️
plugins-bucket-4 58.31% <ø> (-0.03%) ⬇️
plugins-bullmq_cassandra_cookie 61.37% <ø> (-0.03%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 56.35% <ø> (-0.03%) ⬇️
plugins-fetch_fs_generic-pool 58.22% <ø> (-0.03%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 64.14% <ø> (-0.03%) ⬇️
plugins-hapi_hono_ioredis 59.90% <ø> (-0.03%) ⬇️
plugins-knex_langgraph_ldapjs 55.07% <ø> (-0.03%) ⬇️
plugins-light-my-request_limitd-client_lodash 58.38% <ø> (-0.02%) ⬇️
plugins-mariadb_memcached_mercurius 61.29% <ø> (-0.02%) ⬇️
plugins-mongodb_mongodb-core_mongoose 59.26% <ø> (-0.03%) ⬇️
plugins-multer_mysql_mysql2 58.85% <ø> (-0.03%) ⬇️
plugins-nats_node-serialize_opensearch 60.39% <ø> (-0.03%) ⬇️
plugins-passport-http_pino_postgres 58.59% <ø> (-0.03%) ⬇️
plugins-process_pug_redis 57.39% <ø> (-0.03%) ⬇️
plugins-undici_url_valkey 58.02% <ø> (-0.03%) ⬇️
plugins-vm_winston_ws 59.59% <ø> (-0.03%) ⬇️
profiling 61.51% <ø> (-0.03%) ⬇️
serverless-aws-sdk-aws-sdk 55.13% <ø> (-0.02%) ⬇️
serverless-aws-sdk-base-inject-field 50.94% <ø> (-0.03%) ⬇️
serverless-aws-sdk-bedrockruntime 54.66% <ø> (-0.02%) ⬇️
serverless-aws-sdk-client 56.22% <ø> (-0.03%) ⬇️
serverless-aws-sdk-dynamodb 55.50% <ø> (-0.02%) ⬇️
serverless-aws-sdk-eventbridge 49.74% <ø> (-0.02%) ⬇️
serverless-aws-sdk-kinesis 59.08% <ø> (-0.02%) ⬇️
serverless-aws-sdk-lambda 57.23% <ø> (-0.02%) ⬇️
serverless-aws-sdk-s3 55.59% <ø> (-0.02%) ⬇️
serverless-aws-sdk-serverless-peer-service 59.34% <ø> (-0.02%) ⬇️
serverless-aws-sdk-sns 59.88% <ø> (-0.02%) ⬇️
serverless-aws-sdk-sqs 60.30% <ø> (-0.02%) ⬇️
serverless-aws-sdk-stepfunctions 55.42% <ø> (-0.02%) ⬇️
serverless-aws-sdk-util 51.47% <ø> (-0.03%) ⬇️
serverless-bucket-0 54.07% <ø> (ø)
serverless-bucket-1 58.85% <ø> (-0.03%) ⬇️
test-optimization-cucumber 71.03% <ø> (-0.04%) ⬇️
test-optimization-cypress 64.79% <ø> (+<0.01%) ⬆️
test-optimization-jest 72.44% <ø> (-0.01%) ⬇️
test-optimization-mocha 72.11% <ø> (+0.03%) ⬆️
test-optimization-playwright-playwright-atr 59.88% <ø> (-0.03%) ⬇️
test-optimization-playwright-playwright-efd 60.02% <ø> (-0.01%) ⬇️
test-optimization-playwright-playwright-final-status 60.18% <ø> (-0.02%) ⬇️
test-optimization-playwright-playwright-impacted-tests 59.72% <ø> (+0.13%) ⬆️
test-optimization-playwright-playwright-reporting 60.88% <ø> (-0.12%) ⬇️
test-optimization-playwright-playwright-test-management 60.70% <ø> (-0.27%) ⬇️
test-optimization-playwright-playwright-test-span 59.93% <ø> (-0.01%) ⬇️
test-optimization-selenium 59.08% <ø> (-0.16%) ⬇️
test-optimization-testopt 57.62% <ø> (+0.07%) ⬆️
test-optimization-vitest 73.33% <ø> (+0.03%) ⬆️
test-optimization-vitest-browser 58.96% <ø> (+0.04%) ⬆️
test-optimization-webdriverio 65.46% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

semver-patch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@leoromanovsky