test(ffe): add agentless serverless exposure contract

[leoromanovsky]

Motivation

Our business goal is to make the Datadog OpenFeature client simple to integrate.

Agentless delivery is the default. Customers can send exposures directly or add serverless-init later. All three paths must produce identical exposure behavior.

The existing system test covers only Remote Configuration with Agent egress. It does not cover either agentless exposure path.

flowchart LR
  Evaluation["Feature flag evaluation"] --> Cache["Exposure cache"]
  Cache --> Agent["Agent EVP proxy"]
  Agent --> Intake["Exposure intake"]

  Cache -. "Missing contract" .-> Sidecar["serverless-init EVP proxy"]
  Sidecar -.-> Intake

  Cache -. "Missing contract" .-> Direct["Direct HTTPS EVP"]
  Direct -.-> Intake

Loading

The solid path has system-test coverage before this PR. The dashed paths do not.

Changes and Decisions

• Add one scenario for agentless sidecar egress.
• Add one scenario for agentless direct egress.
• Define one exposure setup and one test method in ExposureEgressContract.
• Use three small adapters to select the scenario and capture interface.
• Use one test definition and identical semantic expectations across all three paths.
• Start serverless-init:1.9.13 only for the sidecar scenario.
• Run the direct scenario without a local receiver.
• Assert that the unused path receives no duplicate exposure.
• Gate unsupported scenario adapters in each SDK manifest.

flowchart TB
  RC["Remote Configuration"] --> SDK["Evaluation and exposure cache"]
  UFC["Agentless UFC"] --> SDK

  SDK --> Agent["Agent EVP proxy"]
  SDK --> Sidecar["serverless-init 1.9.13"]
  SDK --> Direct["Direct HTTPS EVP"]

  Agent --> AgentCapture["Agent capture"]
  Sidecar --> SidecarCapture["Sidecar capture"]
  Direct --> DirectCapture["Direct capture"]

  AgentCapture --> Contract["One shared side-effects contract"]
  SidecarCapture --> Contract
  DirectCapture --> Contract

Loading

Test matrix

Configuration delivery	Local receiver	Exposure egress	Scenario	Test file	Capture interface
Remote Configuration	Datadog Agent	Agent EVP proxy	FEATURE_FLAGGING_AND_EXPERIMENTATION	test_exposure_egress.py	interfaces.agent
Agentless UFC	serverless-init:1.9.13	Sidecar EVP proxy	FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS_SERVERLESS	test_exposure_egress.py	interfaces.datadog_sidecar
Agentless UFC	None	Direct HTTPS EVP	FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS_DIRECT	test_exposure_egress.py	interfaces.datadog_direct

Each adapter inherits the same contract. Five identical evaluations must produce one exposure with identical semantic fields.

Note to reviewers

Failures in test dd-gitlab/K8S_LIB_INJECTION_PROFILING_ENABLED: [dd-lib-java-init-test-app, ${PRIVATE_DOCKER_REGISTRY}/system-tests/dd-lib-java-init-test-app are on main branch and known, not related to this diff.

[github-actions]

CODEOWNERS have been resolved as:

tests/ffe/test_exposure_egress.py                                       @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/utils/exposures.py                                            @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
mirror_images.lock.yaml                                                 @DataDog/system-tests-core
mirror_images.yaml                                                      @DataDog/system-tests-core
tests/ffe/README.md                                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/test_the_test/scenarios.json                                      @DataDog/system-tests-core
tests/test_the_test/test_compute_libraries_and_scenarios.py             @DataDog/system-tests-core
tests/test_the_test/test_group_rules.py                                 @DataDog/system-tests-core
tests/test_the_test/test_mock_ffe_agentless_backend.py                  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_context/_scenarios/agentless_endtoend.py                         @DataDog/system-tests-core
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/interfaces/__init__.py                                            @DataDog/system-tests-core
utils/proxy/core.py                                                     @DataDog/system-tests-core
utils/proxy/ports.py                                                    @DataDog/system-tests-core
utils/scripts/libraries_and_scenarios_rules.yml                         @DataDog/system-tests-core
tests/ffe/test_exposures_datadog_agent.py                               @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

[datadog-official]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 437603f | Docs | Datadog PR Page | Give us feedback!}

[Copilot]

Pull request overview

Adds system-test coverage for Feature Flags & Experimentation (FFE) exposure delivery across three egress topologies (Agent EVP proxy, agentless direct HTTPS, and agentless serverless-init sidecar), aiming to enforce an identical exposure contract regardless of deployment path.

Changes:

• Introduces two new agentless end-to-end scenarios (direct + serverless-init sidecar) and wires them into CI selection/rules.
• Extends the proxy/interface layer with two new capture routes (datadog_direct, datadog_sidecar) and a serverless-init:1.9.13 helper container.
• Adds a shared exposure egress contract test (tests/ffe/test_exposure_egress.py) and a route-neutral helper module for exposure assertions.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/scripts/libraries_and_scenarios_rules.yml	Maps new scenarios/tests into the dynamic library/scenario selection rules.
utils/proxy/ports.py	Adds proxy ports for sidecar vs direct Datadog traffic capture.
utils/proxy/core.py	Routes new ports through mocked-backend handling and assigns new interface names.
utils/interfaces/init.py	Exposes new proxy-based interfaces (datadog_sidecar, datadog_direct).
utils/_context/containers.py	Adds ServerlessInitContainer definition for sidecar scenario.
utils/_context/_scenarios/endtoend.py	Extends agentless FFE scenario to support sidecar/direct exposure egress + capture interfaces.
utils/_context/_scenarios/init.py	Registers two new scenario names for direct + serverless agentless exposure egress.
tests/test_the_test/test_mock_ffe_agentless_backend.py	Adds unit-style checks for new agentless exposure scenario topology/env wiring.
tests/test_the_test/test_group_rules.py	Ensures new scenarios are excluded from tracer-release grouping and are in ffe group.
tests/test_the_test/test_compute_libraries_and_scenarios.py	Updates scenario computation expectations for new files/scenarios.
tests/ffe/utils/exposures.py	Introduces shared exposure matching + contract assertions used across egress paths.
tests/ffe/test_exposures_datadog_agent.py	Refactors to reuse shared exposure helpers and removes redundant same-subject caching test (replaced by new contract).
tests/ffe/test_exposure_egress.py	New shared contract test covering exposure semantics + “no duplicate egress” expectations.
tests/ffe/README.md	Updates FFE test index to reflect new/renamed exposure tests.
mirror_images.yaml	Adds datadog/serverless-init:1.9.13 to mirrored images list.
mirror_images.lock.yaml	Locks digest/target for the new mirrored serverless-init image.
manifests/rust.yml	Updates skips/versions for new exposure tests (agentless tests marked missing_feature).
manifests/ruby.yml	Updates skips/versions for new exposure tests (agentless tests marked missing_feature).
manifests/python.yml	Updates skips/versions for new exposure tests (agentless tests marked missing_feature).
manifests/php.yml	Updates skips/versions for new exposure tests (agentless tests marked missing_feature).
manifests/nodejs.yml	Updates skips/versions for new exposure tests (agentless tests marked missing_feature with issue link).
manifests/java.yml	Updates skips/versions for new exposure tests (agentless tests marked missing_feature).
manifests/golang.yml	Updates skips/versions for new exposure tests (agentless tests marked missing_feature).
manifests/dotnet.yml	Updates skips/versions for new exposure tests (agentless tests marked missing_feature).
manifests/cpp_nginx.yml	Updates skips/versions for new exposure tests (missing_feature).
manifests/cpp_httpd.yml	Updates skips/versions for new exposure tests (missing_feature).
.github/workflows/run-end-to-end.yml	Adds CI steps to run the two new agentless exposure scenarios.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (2)

utils/_context/containers.py:860

• This receiver has no health check, so TestedContainer.wait_for_health() marks it healthy immediately after Docker starts it. The weblog only depends on that start completion, not on port 8126 accepting requests, which leaves the sidecar scenario vulnerable to sending its first exposure before serverless-init is ready. Add a readiness probe for the APM listener (consistent with the Agent receiver in this file) before dependent weblogs start.

        super().__init__(
            name="ffe-serverless-init",
            image_name="datadog/serverless-init:1.9.13",
            environment={
                "DD_API_KEY": _FAKE_DD_API_KEY,
                "DD_SITE": "datad0g.com",
                "DD_SERVICE": "ffe-system-tests-serverless-init",
                "DD_ENV": "system-tests",
                "DD_APM_ENABLED": "true",
                "DD_APM_NON_LOCAL_TRAFFIC": "true",
                "DD_PROXY_HTTPS": f"http://proxy:{ProxyPorts.datadog_sidecar}",
                "DD_PROXY_HTTP": f"http://proxy:{ProxyPorts.datadog_sidecar}",
                "DD_SERVERLESS_FLUSH_STRATEGY": "periodically,100",
                "DD_SKIP_SSL_VALIDATION": "true",
            },
        )

utils/_context/_scenarios/agentless_endtoend.py:167

• The direct adapter is not actually configured with “no local receiver.” Passing use_proxy_for_weblog=True makes WeblogContainer inject DD_AGENT_HOST=proxy and DD_TRACE_AGENT_PORT=8126 (utils/_context/containers.py:970-980); only the sidecar branch overrides those values. This can cause SDKs that resolve an Agent endpoint from host/port to select or attempt the Agent EVP route instead of proving direct HTTPS egress. Please separate “start the proxy for HTTPS capture” from “route tracer traffic through the proxy,” so the direct topology has no Agent endpoint variables.

            use_proxy_for_weblog=exposure_egress is not None,

[sameerank]

Two suggestions for improvement. Otherwise looks good!

resolved