[ACIX-1891] Disable internet in all host tests by default, add an option to enable it - #54158
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🎯 Code Coverage (details) 🔗 Commit SHA: a83d830 | Docs | Datadog PR Page | Give us feedback! |
Files inventory check summaryFile checks results against ancestor 36c2cf97: Results for datadog-agent_7.83.0~devel.git.466.a83d830.pipeline.128051181-1_amd64.deb:No change detected |
Add two new VMOption functions to ec2/vmargs.go: - WithoutInternetAccess(): replaces the default security groups with the no-internet ones (existing feature, now with a proper VM-level option) - WithInternetAccess(): the explicit opt-in counterpart — overrides any no-internet default set by a provisioner Both options manipulate the same withoutInternetAccess bool on vmArgs. No behavior change: neither function is called by any provisioner default yet.
…27765169) All tests that use awshost.Provisioner (or variants) now opt-in to internet access via ec2.WithInternetAccess() / scenec2.WithInternetAccess(). This unblocks the 41 E2E test suites that were failing because the host provisioner now defaults to no-internet. Tests changed: - fleet/suite: fleet upgrade/config/extensions suites - installer/script, installer/unix: installer script/package suites - agent-platform/tests: install-script, ddot, package-signing suites - apm/vm_test: VM fakeintake suite (pulls docker images from ghcr.io) - ndm/snmp: snmp VM and autodiscovery suites - cws/ec2_test: CWS EC2 agent suite - cws/windows_test: CWS Windows agent suite - cspm/host_benchmarks_nix_test: CSPM host benchmark suites (RHEL/AlmaLinux repos) - remote-config/tracer_test: RC tracer suite - agent-subcommands: linux + windows subcommand suites - agent-configuration: linux + windows configuration suites - agent-runtimes: Windows agent runtime suites - netpath/netpath_int_win: Windows network path integration suite - agent-log-pipelines/file-tailing: Windows file tailing suite
This is the behavioral change: the Host provisioner now defaults to no-internet access unless the test explicitly calls ec2.WithInternetAccess(). Changes: - resources/aws/ec2/vm.go: security group selection based on WithoutInternetAccess - resources/aws/environment.go: add NoInternetSecurityGroupNames() helper - resources/aws/environmentDefaults.go: set no-internet for agentQa environment - scenarios/aws/ec2/vm.go: pass withoutInternetAccess through to InstanceArgs - testing/provisioners/aws/host/host.go: prepend WithoutInternetAccess as the provisioner default (caller-provided opts including WithInternetAccess() run after and can override it) - components/datadog/agent/host_linuxos.go: use S3 URL (no internet needed) - components/datadog/updater/install_script.sh: use S3 URL - tests/agent-platform/install/install.go: use S3/internal URL - tests/fleet/agent/install.go: use S3/internal URL - examples: add WithoutInternetAccess() to examples that need it - remote-config/tracer_test.go: switch to Provisioner (fakeintake) so the test no longer needs internet at all
df28d9b to
26f32ac
Compare
github-actions
Bot
added
long review
dd-octo-sts
Bot
added
team/fleet-automation
team/agent-runtimes
team/fleet-remediation
team/windows-products
team/agent-security
team/agent-integrations
team/network-device-monitoring-core
team/agent-log-pipelines
team/agent-apm
…/acix-1891-disable-vm-internet-access
94c0ed6 to
3a328e1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a83d830db8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| TESTING_YUM_URL="yumtesting.datad0g.com" \ | ||
| TESTING_YUM_VERSION_PATH="testing/pipeline-${DD_PIPELINE_ID}-a7/7" \ | ||
| bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" | ||
| bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)" |
There was a problem hiding this comment.
When an AWS host enables the updater while using the new default no-internet security group, changing only the initial script URL is insufficient: this embedded script still passes keys.datadoghq.com, apttesting.datad0g.com, and yumtesting.datad0g.com to the downloaded installer, which subsequently contacts those public endpoints. They are not reachable through the S3 VPC gateway, so updater provisioning fails during repository/key setup; use the corresponding s3.amazonaws.com/... paths here as the normal host installer now does.
Useful? React with 👍 / 👎.
| sgNames := e.NoInternetSecurityGroupNames() | ||
| if len(sgNames) == 0 { | ||
| return nil, fmt.Errorf("no security groups configured to block internet access, set the %s config parameter", aws.DDInfraNoInternetSecurityGroupNamesParamName) |
There was a problem hiding this comment.
When a host suite selects the supported aws/sandbox or aws/tse-playground environment without explicitly opting into internet access, the provisioner now always reaches this error because both corresponding defaults leave noInternetSecurityGroupNames empty. Although the earlier agent-qa gap is fixed in the current code, these two environments remain unset, so their host suites abort before creating an EC2 instance unless callers provide an undocumented per-stack override; seed their defaults or avoid enabling the restriction where no group is configured.
Useful? React with 👍 / 👎.
![datadog-datadog-prod-us1-2[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The new host default blocks two supported AWS environments before VM creation and leaves the NPM external-traffic suite unable to run; updater installs also retain internet-only package endpoints, so the restricted-host rollout will break multiple real E2E paths.
📊 Validated against 5 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit a83d830 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| // We ALWAYS need to make a deep copy of `params`, as the provisioner can be called multiple times. | ||
| // and it's easy to forget about it, leading to hard to debug issues. | ||
| // Prepend the no-internet default so that caller-provided opts (e.g. ec2.WithInternetAccess()) can override it. | ||
| opts = append([]ProvisionerOption{WithRunOptions(ec2.WithEC2InstanceOptions(ec2.WithoutInternetAccess()))}, opts...) |
There was a problem hiding this comment.
Every stock host E2E using either supported AWS environment fails during provisioning.
Assertion details
- Input: Run the stock AWS host provisioner in the supported aws/sandbox or aws/tse-playground environment without an explicit aws/noInternetSecurityGroupNames override.
- Expected:
The provisioner should attach a configured restrictive security group, or preserve the previous usable host behavior until that environment has such a group. - Actual: The new universal default enables WithoutInternetAccess, but sandboxDefault() and tsePlaygroundDefault() leave noInternetSecurityGroupNames empty. NewInstance therefore returns
no security groups configured to block internet accessbefore creating a VM.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| // We ALWAYS need to make a deep copy of `params`, as the provisioner can be called multiple times. | ||
| // and it's easy to forget about it, leading to hard to debug issues. | ||
| // Prepend the no-internet default so that caller-provided opts (e.g. ec2.WithInternetAccess()) can override it. | ||
| opts = append([]ProvisionerOption{WithRunOptions(ec2.WithEC2InstanceOptions(ec2.WithoutInternetAccess()))}, opts...) |
There was a problem hiding this comment.
The existing agent-host NPM E2E becomes a deterministic failure rather than validating network monitoring.
Assertion details
- Input: Run TestVMSuiteEx6 in test/new-e2e/tests/npm/agentenv_npm_test.go through the stock AWS host provisioner.
- Expected:
The suite should curl www.datadoghq.com to create an observed network connection and then assert the NPM payload in fakeintake. - Actual: The changed provisioner now attaches the no-internet group, while this caller has no WithInternetAccess option. Its first MustExecute(
curl http://www.datadoghq.com) fails before payload assertions run.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| TESTING_YUM_URL="yumtesting.datad0g.com" \ | ||
| TESTING_YUM_VERSION_PATH="testing/pipeline-${DD_PIPELINE_ID}-a7/7" \ | ||
| bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" | ||
| bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)" |
There was a problem hiding this comment.
Updater-based host provisioning cannot install the Agent once internet access is disabled by default.
Assertion details
- Input: Provision a default restricted AWS host with the updater enabled for pipeline 127765169 on either x86_64 or arm64.
- Expected:
The installer script and every key/APT/YUM package download should use direct S3 URLs reachable through the VPC S3 gateway endpoint. - Actual:
Executing the embedded script with a fake installer payload showed that only the installer URL changed to S3; it still exports keys.datadoghq.com, apttesting.datad0g.com, and yumtesting.datad0g.com. The fetched installer then cannot reach those package endpoints from the new security group.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)" | |
| TESTING_KEYS_URL="s3.amazonaws.com/apttesting.datad0g.com/test-keys" \ | |
| TESTING_APT_URL="s3.amazonaws.com/apttesting.datad0g.com/datadog-agent/pipeline-${DD_PIPELINE_ID}-a7" \ | |
| TESTING_YUM_URL="s3.amazonaws.com/yumtesting.datad0g.com" \ | |
| bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)" |
There was a problem hiding this comment.
Pull request overview
This PR changes the AWS host E2E infrastructure so EC2 instances run without public internet access by default (via a dedicated “no-internet” security group set), and introduces an explicit opt-in (WithInternetAccess) for tests that still require outbound internet. It also updates many existing host-based tests to explicitly enable internet access where needed, and adjusts install-script/repo URLs to S3 endpoints to work better with S3 VPC gateway access.
Changes:
- Default AWS host provisioner instances to no internet access, with an explicit
WithInternetAccess()override. - Add EC2 VM options
WithoutInternetAccess()/WithInternetAccess()and plumb the flag through to instance provisioning (security-group selection). - Update a large set of new-e2e suites to explicitly opt into internet access where required; switch several installer URLs to
s3.amazonaws.com/....
Reviewed changes
Copilot reviewed 72 out of 72 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/new-e2e/tests/remote-config/tracer_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/netpath/network-path-integration/remote_config_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/netpath/network-path-integration/netpath_int_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/ndm/snmp/snmp_vm_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/ndm/snmp/autodiscovery_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/installer/unix/all_packages_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/installer/script/all_scripts_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/fleet/suite/suite.go | Enables internet access for fleet host runs |
| test/new-e2e/tests/fleet/agent/install.go | Switches installer script URLs to S3 endpoints |
| test/new-e2e/tests/discovery/process_autodiscovery_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/cws/windows_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/cws/ec2_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/cspm/host_benchmarks_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/apm/vm_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/status_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/secret_win_test.go | Opts suites into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/secret_nix_test.go | Opts suites into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/run_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/run_nix_test.go | Adds scenec2 import + opts suite into internet access |
| test/new-e2e/tests/agent-subcommands/hostname_ec2_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/hostname_ec2_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/health_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/flare_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/flare_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/diagnose_win_test.go | Opts suite(s) into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/diagnose_nix_test.go | Opts suite(s) into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/configcheck_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/configcheck_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/config_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-subcommands/check_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/shared_library_common_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/rtloader_multiprocessing_lib_common_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/procmgr/procmgr_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/procmgr/par_procmgr_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/ipc_security_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/infra_eudm_common_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/infra_basic_common_test.go | Opts suite/update path into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/hostname_drift_common_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-runtimes/auth_artifact_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-platform/tests/upgrade_test.go | Defaults vmOpts to internet access |
| test/new-e2e/tests/agent-platform/tests/step_by_step_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-platform/tests/persisting_integrations_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-platform/tests/package_signing_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-platform/tests/macos_install_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-platform/tests/install_script_test.go | Opts suite(s) into internet access explicitly |
| test/new-e2e/tests/agent-platform/tests/ddot_install_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-platform/install/install.go | Switches install-script download URL to S3 |
| test/new-e2e/tests/agent-log-pipelines/windows-log/file-tailing/file_tailing_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/sgc-embedded/sgc_multi_backend_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/sgc_win_test.go | Opts suite/update path into internet access explicitly |
| test/new-e2e/tests/agent-configuration/sgc_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/sgc_aws_win_test.go | Switches to NoFakeIntake + opts into internet access |
| test/new-e2e/tests/agent-configuration/sgc_aws_nix_test.go | Opts update path into internet access explicitly |
| test/new-e2e/tests/agent-configuration/secret_win_test.go | Opts suite/update path into internet access explicitly |
| test/new-e2e/tests/agent-configuration/secret_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/inventory_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/gui_win_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/gui_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/configrefresh_win_test.go | Opts suite/update path into internet access explicitly |
| test/new-e2e/tests/agent-configuration/configrefresh_nix_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/configrefresh_api_key_refresh_test.go | Opts suite into internet access explicitly |
| test/new-e2e/tests/agent-configuration/api_test.go | Opts suite into internet access explicitly |
| test/new-e2e/examples/vm_test.go | Updates example to explicitly configure no-internet |
| test/new-e2e/examples/agentenv_logs_test.go | Updates example to explicitly configure no-internet |
| test/e2e-framework/testing/provisioners/aws/host/host.go | Makes no-internet the default for host provisioners |
| test/e2e-framework/scenarios/aws/ec2/vmargs.go | Adds VM options for internet access control |
| test/e2e-framework/scenarios/aws/ec2/vm.go | Plumbs flag to resource-layer instance args |
| test/e2e-framework/resources/aws/environmentDefaults.go | Adds no-internet SG names to selected env defaults |
| test/e2e-framework/resources/aws/environment.go | Adds env accessor for no-internet SG names |
| test/e2e-framework/resources/aws/ec2/vm.go | Selects security groups based on no-internet flag |
| test/e2e-framework/components/datadog/updater/install_script.sh | Switches updater install script URL to S3 |
| test/e2e-framework/components/datadog/agent/host_linuxos.go | Switches repo/script URLs to S3 endpoints for installs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // We ALWAYS need to make a deep copy of `params`, as the provisioner can be called multiple times. | ||
| // and it's easy to forget about it, leading to hard to debug issues. | ||
| // Prepend the no-internet default so that caller-provided opts (e.g. ec2.WithInternetAccess()) can override it. | ||
| opts = append([]ProvisionerOption{WithRunOptions(ec2.WithEC2InstanceOptions(ec2.WithoutInternetAccess()))}, opts...) | ||
| params := getProvisionerParams(opts...) |
| // - [WithHostID] | ||
| // - [WithTenancy] | ||
| // - [WithVolumeThroughput] | ||
| // - [WithoutInternetAccess] | ||
| // - [WithPulumiResourceOptions] |
| defaultVPCID string | ||
| defaultSubnets []SubnetConfig | ||
| defaultSecurityGroups []string | ||
| noInternetSecurityGroupNames []string | ||
| defaultInstanceType string |
There was a problem hiding this comment.
Do all these tests really need internet ?
There was a problem hiding this comment.
We will try removing the flag progressively on all tests
Objective is first to add it, second to progressively remove it (and refactor tests when needed)
There was a problem hiding this comment.
Ah ok from the description I thought it was being set on the proper tests directly, but that makes sense
I think most tests owned by my team don't need internet
tests that still need internet must call it explicitly in the test definition.
There was a problem hiding this comment.
Could be worth asking some AI tool to check which tests might not need it (keeping it if there is any doubt) to reduce changes/reviewers in this PR
CI would show directly if a test fails anyway
There was a problem hiding this comment.
we plan at having some agents progressively doing it yep
likely split in multiple PRs so we can isolate failures + accelerate reviews
691a63b
into
main
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: e98625f Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_metrics_logs | memory utilization | +0.57 | [+0.33, +0.82] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_private_action_runner | memory utilization | +0.21 | [+0.09, +0.33] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | +0.04 | [+0.01, +0.08] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_no_fs_load | memory utilization | -0.02 | [-0.11, +0.07] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | -0.02 | [-0.08, +0.03] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | -0.10 | [-0.15, -0.06] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.14 | [-0.18, -0.11] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | -3.98 | [-4.84, -3.11] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 148.41MiB ≤ 154MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 731.04KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 487.51MiB ≤ 512MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.13MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 17 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 183.14MiB ≤ 195MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 264.17MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 356.39 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 18 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 403.11MiB ≤ 430MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.94GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_private_action_runner | memory_usage | 10/10 | 71.61MiB ≤ 75MiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 26.42 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 302.87MiB ≤ 330MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 58.16 ≤ 200 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 282.55MiB ≤ 310MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 20.48 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 289.43MiB ≤ 320MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
Replicate Execution Details
We run multiple replicates for each experiment/variant. However, we allow replicates to be automatically retried if there are any failures, up to 8 times, at which point the replicate is marked dead and we are unable to run analysis for the entire experiment. We call each of these attempts at running replicates a replicate execution. This section lists all replicate executions that failed due to the target crashing or being oom killed.
Note: In the below tables we bucket failures by experiment, variant, and failure type. For each of these buckets we list out the replicate indexes that failed with an annotation signifying how many times said replicate failed with the given failure mode. In the below example the baseline variant of the experiment named experiment_with_failures had two replicates that failed by oom kills. Replicate 0, which failed 8 executions, and replicate 1 which failed 6 executions, all with the same failure mode.
| Experiment | Variant | Replicates | Failure | Logs | Debug Dashboard |
|---|---|---|---|---|---|
| experiment_with_failures | baseline | 0 (x8) 1 (x6) | Oom killed | Debug Dashboard |
The debug dashboard links will take you to a debugging dashboard specifically designed to investigate replicate execution failures.
❌ Retried Profiling Replicate Execution Failures (ddprof)
Note: Profiling replicas may still be executing. See the debug dashboard for up to date status.
| Experiment | Variant | Replicates | Failure | Debug Dashboard |
|---|---|---|---|---|
| quality_gate_logs | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_logs | comparison | 10 | Oom killed | Debug Dashboard |
| quality_gate_security_idle | comparison | 10 | Oom killed | Debug Dashboard |
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
What does this PR do?
Use a dedicated security group for hosts, that will not allow reaching internet by default, tests that still need internet must call it explicitly in the test definition. Long term goal is to reduce the number of tests that need to reach internet, to improve reliability of those tests
Motivation
Improve reliability enforce best practices
Describe how you validated your changes
Additional Notes