Skip to content

test(appsec): deflake RASP SSRF "should not detect threat" express tests - #9417

Merged
IlyasShabi merged 2 commits into
masterfrom
ccapell/deflake-rasp-ssrf-express-test
Jul 20, 2026
Merged

test(appsec): deflake RASP SSRF "should not detect threat" express tests#9417
IlyasShabi merged 2 commits into
masterfrom
ccapell/deflake-rasp-ssrf-express-test

Conversation

@CarlesDD

@CarlesDD CarlesDD commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Points the SSRF RASP "should not detect threat" express tests at an unresolvable host (not-a-threat.invalid) instead of www.datadoghq.com, and ends the response on the outbound request's error event. The request now fails fast locally instead of depending on a live external connection.

Motivation

Flaky test.

Ending the response waited on a live request to www.datadoghq.com, so slow CI networking pushed the web span past the 1000 ms assertion window (No matching trace received). The SSRF check runs at request start, before DNS, so it never needed the network.

Additional Notes

Behaviour under assertion is unchanged: both hosts yield _dd.appsec.rasp.rule.eval = 1 with no threat; the fix is more deterministic.

@dd-octo-sts

dd-octo-sts Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 6.77 MB
Deduped: 7.43 MB
No deduping: 7.43 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 438.86 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@CarlesDD

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f8ec5796d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/test/appsec/rasp/ssrf.express.plugin.spec.js
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jul 17, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 98.34% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c768114 | Docs | Datadog PR Page | Give us feedback!

@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.34%. Comparing base (87b974e) to head (c768114).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9417      +/-   ##
==========================================
- Coverage   98.34%   98.34%   -0.01%     
==========================================
  Files         924      924              
  Lines      123056   123056              
  Branches    10869    11090     +221     
==========================================
- Hits       121024   121016       -8     
- Misses       2032     2040       +8     
Flag Coverage Δ
aiguard 58.22% <ø> (-0.03%) ⬇️
aiguard-integration 57.08% <ø> (ø)
apm-bucket-0 58.48% <ø> (-0.03%) ⬇️
apm-bucket-1 64.59% <ø> (-0.03%) ⬇️
apm-bucket-2 63.64% <ø> (-0.03%) ⬇️
apm-bucket-3 60.93% <ø> (-0.03%) ⬇️
apm-capabilities-tracing 62.31% <ø> (ø)
apm-integrations-aerospike 57.61% <ø> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript 62.53% <ø> (-0.03%) ⬇️
apm-integrations-couchbase 57.89% <ø> (-0.03%) ⬇️
apm-integrations-http 63.63% <ø> (-0.03%) ⬇️
apm-integrations-kafkajs 63.09% <ø> (-0.03%) ⬇️
apm-integrations-next 60.02% <ø> (-0.03%) ⬇️
apm-integrations-prisma 59.45% <ø> (-0.03%) ⬇️
appsec 73.94% <ø> (-0.02%) ⬇️
appsec-express_fastify_graphql 71.28% <ø> (-0.27%) ⬇️
appsec-integration 51.90% <ø> (ø)
appsec-kafka_ldapjs_lodash 64.82% <ø> (-0.03%) ⬇️
appsec-mongodb-core_mongoose_mysql 68.55% <ø> (-0.02%) ⬇️
appsec-next 58.19% <ø> (-0.03%) ⬇️
appsec-node-serialize_passport_postgres 68.24% <ø> (-0.02%) ⬇️
appsec-sourcing_stripe_template 66.54% <ø> (-0.02%) ⬇️
debugger 65.87% <ø> (-0.01%) ⬇️
instrumentations-bucket-0 52.39% <ø> (-0.03%) ⬇️
instrumentations-bucket-1 61.12% <ø> (-0.03%) ⬇️
instrumentations-bucket-10 62.98% <ø> (-0.03%) ⬇️
instrumentations-bucket-11 52.40% <ø> (-0.03%) ⬇️
instrumentations-bucket-12 52.85% <ø> (-0.03%) ⬇️
instrumentations-bucket-13 52.35% <ø> (-0.03%) ⬇️
instrumentations-bucket-2 54.32% <ø> (-0.03%) ⬇️
instrumentations-bucket-3 60.09% <ø> (-0.03%) ⬇️
instrumentations-bucket-4 52.98% <ø> (-0.03%) ⬇️
instrumentations-bucket-5 58.32% <ø> (-0.03%) ⬇️
instrumentations-bucket-6 61.68% <ø> (-0.03%) ⬇️
instrumentations-bucket-7 59.22% <ø> (-0.03%) ⬇️
instrumentations-bucket-8 60.49% <ø> (-0.03%) ⬇️
instrumentations-bucket-9 62.35% <ø> (-0.03%) ⬇️
instrumentations-instrumentation-couchbase 51.91% <ø> (-0.03%) ⬇️
instrumentations-integration-esbuild 34.05% <ø> (ø)
llmobs-ai_anthropic_bedrock 63.32% <ø> (-0.03%) ⬇️
llmobs-bucket-1 62.61% <ø> (-0.03%) ⬇️
llmobs-openai 63.29% <ø> (-0.03%) ⬇️
llmobs-sdk 65.41% <ø> (-0.03%) ⬇️
llmobs-vertex-ai 59.97% <ø> (-0.03%) ⬇️
master-coverage 98.34% <ø> (?)
openfeature 54.61% <ø> (-0.09%) ⬇️
openfeature-unit 53.43% <ø> (-0.03%) ⬇️
platform-core_esbuild_instrumentations-misc 40.67% <ø> (-0.02%) ⬇️
platform-integration 62.31% <ø> (ø)
platform-shimmer_unit-guardrails_webpack 39.34% <ø> (-0.02%) ⬇️
plugins-bucket-0 57.76% <ø> (-0.03%) ⬇️
plugins-bucket-1 55.17% <ø> (ø)
plugins-bucket-11 62.72% <ø> (-0.40%) ⬇️
plugins-bucket-17 62.77% <ø> (?)
plugins-bucket-18 63.42% <ø> (+0.49%) ⬆️
plugins-bucket-19 62.53% <ø> (+1.55%) ⬆️
plugins-bucket-20 65.27% <ø> (+2.34%) ⬆️
plugins-bucket-4 59.39% <ø> (-0.03%) ⬇️
plugins-bullmq_cassandra_cookie 62.68% <ø> (-0.03%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 57.56% <ø> (-0.03%) ⬇️
plugins-fetch_fs_generic-pool 59.67% <ø> (-0.03%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 65.67% <ø> (-0.03%) ⬇️
plugins-hapi_hono_ioredis 61.10% <ø> (-0.03%) ⬇️
plugins-jest_knex_langgraph ?
plugins-jest_langgraph_ldapjs 56.32% <ø> (?)
plugins-ldapjs_light-my-request_limitd-client ?
plugins-light-my-request_limitd-client_lodash 59.47% <ø> (?)
plugins-lodash_mariadb_memcached ?
plugins-mariadb_memcached_mercurius 62.40% <ø> (?)
plugins-moleculer_mongodb_mongodb-core ?
plugins-mongodb_mongodb-core_mongoose 60.54% <ø> (?)
plugins-mongoose_multer_mysql ?
plugins-multer_mysql_mysql2 59.26% <ø> (?)
plugins-mysql2_nats_node-serialize ?
plugins-nats_node-serialize_opensearch 61.69% <ø> (?)
plugins-opensearch_passport-http_pino ?
plugins-passport-http_pino_postgres 59.49% <ø> (?)
plugins-postgres_process_pug ?
plugins-process_pug_redis 58.51% <ø> (?)
plugins-redis_router_sequelize ?
plugins-test-and-upstream-rhea_undici_url ?
plugins-undici_url_valkey 59.37% <ø> (?)
plugins-valkey_vm_winston ?
plugins-vm_winston_ws 60.80% <ø> (?)
plugins-ws ?
profiling 63.07% <ø> (-0.03%) ⬇️
serverless-aws-sdk-aws-sdk 55.71% <ø> (-0.03%) ⬇️
serverless-aws-sdk-bedrockruntime 55.43% <ø> (-0.03%) ⬇️
serverless-aws-sdk-client 57.17% <ø> (-0.03%) ⬇️
serverless-aws-sdk-dynamodb 56.36% <ø> (-0.04%) ⬇️
serverless-aws-sdk-eventbridge 49.93% <ø> (-0.03%) ⬇️
serverless-aws-sdk-kinesis 60.15% <ø> (-0.03%) ⬇️
serverless-aws-sdk-lambda 58.14% <ø> (-0.03%) ⬇️
serverless-aws-sdk-s3 56.30% <ø> (-0.03%) ⬇️
serverless-aws-sdk-serverless-peer-service 60.56% <ø> (-0.03%) ⬇️
serverless-aws-sdk-sns 60.96% <ø> (-0.03%) ⬇️
serverless-aws-sdk-sqs 61.40% <ø> (-0.03%) ⬇️
serverless-aws-sdk-stepfunctions 56.29% <ø> (-0.03%) ⬇️
serverless-aws-sdk-util 52.19% <ø> (-0.03%) ⬇️
serverless-bucket-0 55.22% <ø> (ø)
serverless-bucket-1 60.09% <ø> (-0.03%) ⬇️
test-optimization-cucumber 72.99% <ø> (-0.03%) ⬇️
test-optimization-cypress 66.42% <ø> (+0.08%) ⬆️
test-optimization-jest 74.39% <ø> (-0.05%) ⬇️
test-optimization-mocha 74.73% <ø> (+0.04%) ⬆️
test-optimization-playwright-playwright-atr 61.43% <ø> (-0.01%) ⬇️
test-optimization-playwright-playwright-efd 61.62% <ø> (ø)
test-optimization-playwright-playwright-final-status 61.59% <ø> (-0.17%) ⬇️
test-optimization-playwright-playwright-impacted-tests 61.31% <ø> (+0.16%) ⬆️
test-optimization-playwright-playwright-reporting 61.28% <ø> (+0.06%) ⬆️
test-optimization-playwright-playwright-test-management 61.86% <ø> (-0.41%) ⬇️
test-optimization-playwright-playwright-test-span 61.34% <ø> (-0.06%) ⬇️
test-optimization-selenium 60.71% <ø> (-0.11%) ⬇️
test-optimization-testopt 58.86% <ø> (-0.27%) ⬇️
test-optimization-vitest 71.33% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter

pr-commenter Bot commented Jul 17, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-17 10:54:59

Comparing candidate commit c768114 in PR branch ccapell/deflake-rasp-ssrf-express-test with baseline commit 87b974e in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2315 metrics, 43 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-212.721ms; +216.097ms] or [-8.032%; +8.160%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-236.393ms; +231.480ms] or [-9.302%; +9.109%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-153.000ms; +156.018ms] or [-4.965%; +5.063%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-193.027ms; +176.408ms] or [-6.650%; +6.078%]

scenario:appsec-control-20

  • unstable execution_time [-95.837ms; +136.323ms] or [-5.910%; +8.407%]

scenario:appsec-control-24

  • unstable execution_time [-115.500ms; +111.392ms] or [-9.312%; +8.981%]

scenario:appsec-control-26

  • unstable execution_time [-124.389ms; +121.672ms] or [-10.077%; +9.856%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-13.885ms; +15.920ms] or [-5.423%; +6.218%]

scenario:appsec-iast-with-vulnerability-control-20

  • unstable execution_time [-24.708ms; +30.032ms] or [-4.528%; +5.503%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-27.125ms; +32.327ms] or [-4.905%; +5.846%]

scenario:child_process-file-args-24

  • unstable execution_time [-26.289ms; +31.457ms] or [-5.551%; +6.643%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-1925.710ms; +3087.443ms] or [-23.325%; +37.397%]
  • unstable execution_time [-1886.300ms; +3098.662ms] or [-21.025%; +34.538%]
  • unstable instructions [-16.7G instructions; +26.7G instructions] or [-24.777%; +39.664%]
  • unstable max_rss_usage [-8.031MB; +13.109MB] or [-5.119%; +8.356%]
  • unstable throughput [-841.711op/s; +502.492op/s] or [-22.960%; +13.707%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-3615.163ms; +5133.471ms] or [-32.554%; +46.226%]
  • unstable execution_time [-3716.341ms; +5280.795ms] or [-31.393%; +44.608%]
  • unstable instructions [-31.9G instructions; +45.6G instructions] or [-34.100%; +48.786%]
  • unstable max_rss_usage [-11.943MB; +15.996MB] or [-7.252%; +9.714%]
  • unstable throughput [-1024.370op/s; +729.179op/s] or [-35.007%; +24.919%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-2608.286ms; +2441.733ms] or [-29.622%; +27.730%]
  • unstable execution_time [-2604.753ms; +2448.159ms] or [-27.338%; +25.694%]
  • unstable instructions [-21818.2M instructions; +20791.9M instructions] or [-30.330%; +28.903%]
  • unstable max_rss_usage [-10607.215KB; +9678.415KB] or [-6.659%; +6.076%]
  • unstable throughput [-688.193op/s; +720.834op/s] or [-19.594%; +20.523%]

scenario:debugger-line-probe-with-snapshot-minimal-26

  • unstable cpu_user_time [-3571.944ms; +3613.687ms] or [-34.633%; +35.038%]
  • unstable execution_time [-3576.271ms; +3604.009ms] or [-32.459%; +32.710%]
  • unstable instructions [-32326.5M instructions; +32316.4M instructions] or [-37.376%; +37.364%]
  • unstable max_rss_usage [-10779.000KB; +10712.600KB] or [-6.610%; +6.569%]
  • unstable throughput [-716.709op/s; +699.031op/s] or [-23.256%; +22.682%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-2268.622ms; +736.019ms] or [-23.760%; +7.709%]
  • unstable execution_time [-2292.627ms; +721.785ms] or [-22.332%; +7.031%]
  • unstable instructions [-20.3G instructions; +6.6G instructions] or [-25.454%; +8.283%]
  • unstable throughput [-140.333op/s; +454.855op/s] or [-4.349%; +14.096%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-336.144ms; +347.308ms] or [-6.899%; +7.128%]
  • unstable execution_time [-337.316ms; +354.892ms] or [-6.820%; +7.175%]
  • unstable throughput [-126264.436op/s; +120281.310op/s] or [-7.437%; +7.085%]

scenario:plugin-graphql-long-with-depth-off-20

  • unstable max_rss_usage [-4.029MB; +10.056MB] or [-3.154%; +7.872%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable cpu_user_time [-613.181ms; +589.277ms] or [-5.315%; +5.108%]
  • unstable execution_time [-628.413ms; +600.935ms] or [-5.341%; +5.107%]
  • unstable throughput [-3.522op/s; +3.704op/s] or [-5.149%; +5.416%]

scenario:test-optimization-large-suite-20

  • unstable max_rss_usage [-4119343 bytes; +4119676 bytes] or [-5.199%; +5.200%]

@CarlesDD
CarlesDD marked this pull request as ready for review July 17, 2026 10:18
@CarlesDD
CarlesDD requested a review from a team as a code owner July 17, 2026 10:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

axiosToTest.get(`https://${req.query.host}`)

P2 Badge Disable proxies for the unresolvable-host request

When the test process has HTTPS_PROXY configured, Axios 1.16.1 uses its proxy-from-env dependency and sends this request to the proxy rather than resolving .invalid locally. A proxy that retries or waits while resolving the target leaves res.end() pending and can still exceed checkRaspExecutedAndNotThreat's 1000 ms window, so the intended deflake does not hold in proxied CI environments. Set proxy: false (and a local timeout) for this outbound test request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@IlyasShabi
IlyasShabi merged commit bab39ed into master Jul 20, 2026
726 of 729 checks passed
@IlyasShabi
IlyasShabi deleted the ccapell/deflake-rasp-ssrf-express-test branch July 20, 2026 08:23
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 20, 2026
…sts (#9417)

* test(appsec): deflake RASP SSRF "should not detect threat" express tests

* test(appsec): disable proxy on RASP SSRF outbound "not detect threat" requests
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Jul 20, 2026
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 20, 2026
…sts (#9417)

* test(appsec): deflake RASP SSRF "should not detect threat" express tests

* test(appsec): disable proxy on RASP SSRF outbound "not detect threat" requests
This was referenced Jul 20, 2026
leoromanovsky pushed a commit that referenced this pull request Jul 22, 2026
…sts (#9417)

* test(appsec): deflake RASP SSRF "should not detect threat" express tests

* test(appsec): disable proxy on RASP SSRF outbound "not detect threat" requests
leoromanovsky pushed a commit that referenced this pull request Jul 22, 2026
…sts (#9417)

* test(appsec): deflake RASP SSRF "should not detect threat" express tests

* test(appsec): disable proxy on RASP SSRF outbound "not detect threat" requests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants