Skip to content

add logs codeowners experiments - #54934

Draft
cmetz100 wants to merge 1 commit into
cmetz/smp_experiment_selection_basefrom
cmetz/smp_experiment_selection_logs_codeowners_experiments
Draft

add logs codeowners experiments#54934
cmetz100 wants to merge 1 commit into
cmetz/smp_experiment_selection_basefrom
cmetz/smp_experiment_selection_logs_codeowners_experiments

Conversation

@cmetz100

@cmetz100 cmetz100 commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

adds the log-pipelines file_to_blackhole experiments in the codeowners bucket so they auto-run whenever log pipelines code changes. plus the logs/ codeowners entry.

Motivation

Describe how you validated your changes

Additional Notes

@cmetz100
add logs codeowners experiments
88934b7 
Signed-off-by: Caleb Metz <caleb.metz@datadoghq.com>
@cmetz100 Graphite App

cmetz100 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

This was referenced Aug 14, 2026
Draft
Draft
@dd-octo-sts

dd-octo-sts Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Gitlab CI Configuration Changes

Updated: .gitlab-ci.yml

Added Jobs

single_machine_performance-experiments-validate 
single_machine_performance-experiments-validate:
  image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
  needs: []
  rules:
  - if: $E2E_COVERAGE_PIPELINE == "true"
    when: never
  - changes:
      compare_to: $COMPARE_TO_BRANCH
      paths:
      - test/regression/**/*
  script:
  - SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)
  - SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)
  - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
  - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
  - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
  - aws --profile ${AWS_NAMED_PROFILE} s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp
    smp
  - chmod +x smp
  - ./smp experiments validate --target-config-dir ${CONFIG_DIR} --manifest ${CONFIG_DIR}/selection.yaml
    --label-prefix smp/ --exclude-path ebpf --in-ci
  stage: lint
  tags:
  - arch:amd64
  - specific:true
  variables:
    AWS_NAMED_PROFILE: single-machine-performance
    BOT_LOGIN: bot_login
    BOT_TOKEN: bot_token
    CONFIG_DIR: test/regression
    SMP_VERSION: dev-pr4702-6ac924e95

Changes Summary

Removed Modified Added Renamed
0 0 1 0

Updated: .gitlab/childs/smp-regression-child-pipeline.yml

Modified Jobs

.single-machine-performance-run_regression_detector 
  .single-machine-performance-run_regression_detector:
    artifacts:
      expire_in: 1 weeks
      paths:
      - submission_metadata
      - outputs/report.md
      - outputs/regression_signal.json
      - outputs/bounds_check_signal.json
      - outputs/junit.xml
      - outputs/report.json
      - outputs/decision_record.md
      when: always
    before_script:
    - echo "BASELINE_SHA=${BASELINE_SHA}" > regression_detector.env
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    script:
    - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DATADOG_API_KEY
    - datadog-ci tag --level job --tags smp_failure_mode:"timeout"
    - mkdir outputs
    - AWS_NAMED_PROFILE="single-machine-performance"
    - "SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
    - "SMP_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT \"${SMP_TEAM_NAME}_team_id\"\
      ) || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $SMP_API_URL)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
    - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
    - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
    - "aws --profile single-machine-performance s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp\
      \ smp || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-fetch-cli\"\n  exit $exit_code\n}\n"
    - chmod +x smp
    - source regression_detector.env
    - echo "Baseline SHA is ${BASELINE_SHA}"
    - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
    - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
      "s3://${SMP_TEAM_ID}-smp-artifacts/information/"
    - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${BASELINE_SHA}-7-full-amd64
    - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
    - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-full-amd64
    - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
    - SMP_TAGS="ci_pipeline_id=${PARENT_PIPELINE_ID},ci_job_id=${CI_JOB_ID},ci_commit_branch=${CI_COMMIT_BRANCH},purpose=agent_ci"
    - echo "Tags passed through SMP are ${SMP_TAGS}"
    - RUST_LOG="info,aws_config::profile::credentials=error"
    - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
+   - "if [ \"${SELECT_EXPERIMENTS}\" == \"true\" ]; then\n  export GITHUB_TOKEN=$(dd-octo-sts\
+     \ token --scope DataDog/datadog-agent --policy self.gitlab.comment-pr || true)\n\
+     \  # Resolve the PR's labels + changed files. The GitHub API calls live in the\
+     \ dda task (via\n  # GithubAPI), not as curl/jq here. Empty outputs on no-PR/failure\
+     \ -> the resolver defaults.\n  dda inv owners.smp-pr-context --branch \"${CI_COMMIT_BRANCH}\"\
+     \ --changed-files-out changed_files.txt --labels-out pr_labels.txt </dev/null\
+     \ \\\n    || { echo \"owners.smp-pr-context failed; resolving default set.\";\
+     \ : > changed_files.txt; : > pr_labels.txt; }\n  LABELS=$(cat pr_labels.txt 2>/dev/null\
+     \ || true)\n  # Resolve the run set. On ANY failure or empty result, fall back\
+     \ to the always-run quality\n  # gates so a selection-tooling problem (missing\
+     \ dda, resolve error, network) never silently\n  # drops the gates -- the worst\
+     \ case is \"only gates run\", never \"nothing runs\".\n  dda inv owners.smp-resolve\
+     \ --config-dir \"${CONFIG_DIR}\" --manifest \"${CONFIG_DIR}/selection.yaml\" --smp-bin\
+     \ ./smp --changed-files changed_files.txt --labels \"${LABELS}\" --runner container\
+     \ --exclude ebpf --out experiment_path_filter.txt </dev/null || echo \"owners.smp-resolve\
+     \ failed; falling back to quality_gates\"\n  EXPERIMENT_PATH_FILTER=$(cat experiment_path_filter.txt\
+     \ 2>/dev/null || true)\n  if [ -z \"${EXPERIMENT_PATH_FILTER}\" ]; then\n    echo\
+     \ \"Resolve produced no filter (tooling failure or empty); falling back to quality_gates.\"\
+     \n    EXPERIMENT_PATH_FILTER=\"quality_gates\"\n  fi\n  echo \"Experiment path\
+     \ filter: '${EXPERIMENT_PATH_FILTER}'\"\nfi\n"
    - if [ "$RUN_CONFIG_ONLY" == "true" ]; then
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only\
      \ \\\n--target-image ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n\
      --target-config-dir ${CONFIG_DIR} \\\n--submission-metadata submission_metadata\
      \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command\
      \ failed with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      job-submission\"\n  exit $exit_code\n} \n"
    - else
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
      \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
      \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
+     \ ${CONFIG_DIR} \\\n${EXPERIMENT_PATH_FILTER:+--experiment-path-filter \"${EXPERIMENT_PATH_FILTER}\"\
-     \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^                                                          ------------
+     \ --runner container} \\\n--submission-metadata submission_metadata \\\n--tags\
?       ^^^^^^^^^^^^^^^^^^
-     \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
?                                                                        ---------------- --
+     \ ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command failed with\
?      ++++++++++++
-     \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
?                                                                               --------
+     \ code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\
?     +++++++++++++++++++
-     \ $exit_code\n} \n"
+     \n  exit $exit_code\n} \n"
?      +++++++
    - fi
    - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
    - echo "SMP Job Id is ${SMP_JOB_ID}"
    - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
      \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
      \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
      \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
      \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
      smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
      \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
    - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
    - datadog-ci junit upload --service datadog-agent outputs/junit.xml
    - datadog-ci tag --level job --tags smp_failure_mode:"none"
    - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
      \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
      \ job --tags smp_optimization_goal:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_bounds_check:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
      \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
      \ job --tags smp_bounds_check:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"failed"
    - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
      \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
      Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
      \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
      \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
      \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
      \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
      \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
      \ {})\n            comparison = results.get('comparison', [])\n            num_total\
      \ = len(comparison)\n            failed_replicates = [\n                replicate\
      \ for replicate in comparison if not replicate.get('passed', False)\n        \
      \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
      \ = num_total - num_failed\n            if failed_replicates:\n              \
      \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
      \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
      \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
      \ = True\n            else:\n                decision_record.append(\n       \
      \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
      \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
      \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
      \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
      \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
      \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
      n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
      Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
      Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"passed"
    stage: functional_test
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h10m
.smp_job_submit 
  .smp_job_submit:
  - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API} --aws-named-profile\
    \ ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image ${BASELINE_IMAGE} \\\n\
    --comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha ${BASELINE_SHA} \\\n--comparison-sha\
-   \ ${CI_COMMIT_SHA} \\\n--target-config-dir ${CONFIG_DIR} \\\n--submission-metadata\
-   \ submission_metadata \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp\
-   \ job submit command failed with code $exit_code\"\n  datadog-ci tag --level job\
+   \ ${CI_COMMIT_SHA} \\\n--target-config-dir ${CONFIG_DIR} \\\n${EXPERIMENT_PATH_FILTER:+--experiment-path-filter\
+   \ \"${EXPERIMENT_PATH_FILTER}\" --runner container} \\\n--submission-metadata submission_metadata\
+   \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command failed\
+   \ with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
-   \ --tags smp_failure_mode:\"job-submission\"\n  exit $exit_code\n} \n"
?   ----------------------------
+   job-submission\"\n  exit $exit_code\n} \n"
single-machine-performance-metal-runners-regression_detector 
  single-machine-performance-metal-runners-regression_detector:
    allow_failure: true
    artifacts:
      expire_in: 1 weeks
      paths:
      - submission_metadata
      - outputs/report.md
      - outputs/regression_signal.json
      - outputs/bounds_check_signal.json
      - outputs/junit.xml
      - outputs/report.json
      - outputs/decision_record.md
      when: always
    before_script:
    - echo "BASELINE_SHA=${BASELINE_SHA}" > regression_detector.env
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    script:
    - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DATADOG_API_KEY
    - datadog-ci tag --level job --tags smp_failure_mode:"timeout"
    - mkdir outputs
    - AWS_NAMED_PROFILE="single-machine-performance"
    - "SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
    - "SMP_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT \"${SMP_TEAM_NAME}_team_id\"\
      ) || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $SMP_API_URL)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
    - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
    - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
    - "aws --profile single-machine-performance s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp\
      \ smp || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-fetch-cli\"\n  exit $exit_code\n}\n"
    - chmod +x smp
    - source regression_detector.env
    - echo "Baseline SHA is ${BASELINE_SHA}"
    - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
    - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
      "s3://${SMP_TEAM_ID}-smp-artifacts/information/"
    - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${BASELINE_SHA}-7-full-amd64
    - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
    - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-full-amd64
    - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
    - SMP_TAGS="ci_pipeline_id=${PARENT_PIPELINE_ID},ci_job_id=${CI_JOB_ID},ci_commit_branch=${CI_COMMIT_BRANCH},purpose=agent_ci"
    - echo "Tags passed through SMP are ${SMP_TAGS}"
    - RUST_LOG="info,aws_config::profile::credentials=error"
    - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
+   - "if [ \"${SELECT_EXPERIMENTS}\" == \"true\" ]; then\n  export GITHUB_TOKEN=$(dd-octo-sts\
+     \ token --scope DataDog/datadog-agent --policy self.gitlab.comment-pr || true)\n\
+     \  # Resolve the PR's labels + changed files. The GitHub API calls live in the\
+     \ dda task (via\n  # GithubAPI), not as curl/jq here. Empty outputs on no-PR/failure\
+     \ -> the resolver defaults.\n  dda inv owners.smp-pr-context --branch \"${CI_COMMIT_BRANCH}\"\
+     \ --changed-files-out changed_files.txt --labels-out pr_labels.txt </dev/null\
+     \ \\\n    || { echo \"owners.smp-pr-context failed; resolving default set.\";\
+     \ : > changed_files.txt; : > pr_labels.txt; }\n  LABELS=$(cat pr_labels.txt 2>/dev/null\
+     \ || true)\n  # Resolve the run set. On ANY failure or empty result, fall back\
+     \ to the always-run quality\n  # gates so a selection-tooling problem (missing\
+     \ dda, resolve error, network) never silently\n  # drops the gates -- the worst\
+     \ case is \"only gates run\", never \"nothing runs\".\n  dda inv owners.smp-resolve\
+     \ --config-dir \"${CONFIG_DIR}\" --manifest \"${CONFIG_DIR}/selection.yaml\" --smp-bin\
+     \ ./smp --changed-files changed_files.txt --labels \"${LABELS}\" --runner container\
+     \ --exclude ebpf --out experiment_path_filter.txt </dev/null || echo \"owners.smp-resolve\
+     \ failed; falling back to quality_gates\"\n  EXPERIMENT_PATH_FILTER=$(cat experiment_path_filter.txt\
+     \ 2>/dev/null || true)\n  if [ -z \"${EXPERIMENT_PATH_FILTER}\" ]; then\n    echo\
+     \ \"Resolve produced no filter (tooling failure or empty); falling back to quality_gates.\"\
+     \n    EXPERIMENT_PATH_FILTER=\"quality_gates\"\n  fi\n  echo \"Experiment path\
+     \ filter: '${EXPERIMENT_PATH_FILTER}'\"\nfi\n"
    - if [ "$RUN_CONFIG_ONLY" == "true" ]; then
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only\
      \ \\\n--target-image ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n\
      --target-config-dir ${CONFIG_DIR} \\\n--submission-metadata submission_metadata\
      \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command\
      \ failed with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      job-submission\"\n  exit $exit_code\n} \n"
    - else
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
      \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
      \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
+     \ ${CONFIG_DIR} \\\n${EXPERIMENT_PATH_FILTER:+--experiment-path-filter \"${EXPERIMENT_PATH_FILTER}\"\
-     \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^                                                          ------------
+     \ --runner container} \\\n--submission-metadata submission_metadata \\\n--tags\
?       ^^^^^^^^^^^^^^^^^^
-     \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
?                                                                        ---------------- --
+     \ ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command failed with\
?      ++++++++++++
-     \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
?                                                                               --------
+     \ code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\
?     +++++++++++++++++++
-     \ $exit_code\n} \n"
+     \n  exit $exit_code\n} \n"
?      +++++++
    - fi
    - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
    - echo "SMP Job Id is ${SMP_JOB_ID}"
    - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
      \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
      \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
      \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
      \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
      smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
      \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
    - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
    - datadog-ci junit upload --service datadog-agent outputs/junit.xml
    - datadog-ci tag --level job --tags smp_failure_mode:"none"
    - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
      \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
      \ job --tags smp_optimization_goal:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_bounds_check:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
      \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
      \ job --tags smp_bounds_check:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"failed"
    - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
      \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
      Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
      \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
      \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
      \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
      \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
      \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
      \ {})\n            comparison = results.get('comparison', [])\n            num_total\
      \ = len(comparison)\n            failed_replicates = [\n                replicate\
      \ for replicate in comparison if not replicate.get('passed', False)\n        \
      \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
      \ = num_total - num_failed\n            if failed_replicates:\n              \
      \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
      \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
      \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
      \ = True\n            else:\n                decision_record.append(\n       \
      \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
      \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
      \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
      \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
      \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
      \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
      n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
      Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
      Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"passed"
    stage: functional_test
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h10m
    variables:
      BOT_LOGIN: bot_login_metal-runners
      BOT_TOKEN: bot_token_metal-runners
      CONFIG_DIR: test/regression/ebpf
      SMP_API_URL: metal-runners_api_url
      SMP_TEAM_NAME: metal-runners
      SMP_VERSION: v0.27.0-nightly.20260423T145046Z
    when: manual
single-machine-performance-metal-runners-regression_detector_cfg 
  single-machine-performance-metal-runners-regression_detector_cfg:
    allow_failure: true
    artifacts:
      expire_in: 1 weeks
      paths:
      - submission_metadata
      - outputs/report.md
      - outputs/regression_signal.json
      - outputs/bounds_check_signal.json
      - outputs/junit.xml
      - outputs/report.json
      - outputs/decision_record.md
      when: always
    before_script:
    - echo "BASELINE_SHA=${BASELINE_SHA}" > regression_detector.env
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    script:
    - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DATADOG_API_KEY
    - datadog-ci tag --level job --tags smp_failure_mode:"timeout"
    - mkdir outputs
    - AWS_NAMED_PROFILE="single-machine-performance"
    - "SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
    - "SMP_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT \"${SMP_TEAM_NAME}_team_id\"\
      ) || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $SMP_API_URL)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
    - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
    - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
    - "aws --profile single-machine-performance s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp\
      \ smp || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-fetch-cli\"\n  exit $exit_code\n}\n"
    - chmod +x smp
    - source regression_detector.env
    - echo "Baseline SHA is ${BASELINE_SHA}"
    - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
    - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
      "s3://${SMP_TEAM_ID}-smp-artifacts/information/"
    - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${BASELINE_SHA}-7-full-amd64
    - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
    - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-full-amd64
    - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
    - SMP_TAGS="ci_pipeline_id=${PARENT_PIPELINE_ID},ci_job_id=${CI_JOB_ID},ci_commit_branch=${CI_COMMIT_BRANCH},purpose=agent_ci"
    - echo "Tags passed through SMP are ${SMP_TAGS}"
    - RUST_LOG="info,aws_config::profile::credentials=error"
    - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
+   - "if [ \"${SELECT_EXPERIMENTS}\" == \"true\" ]; then\n  export GITHUB_TOKEN=$(dd-octo-sts\
+     \ token --scope DataDog/datadog-agent --policy self.gitlab.comment-pr || true)\n\
+     \  # Resolve the PR's labels + changed files. The GitHub API calls live in the\
+     \ dda task (via\n  # GithubAPI), not as curl/jq here. Empty outputs on no-PR/failure\
+     \ -> the resolver defaults.\n  dda inv owners.smp-pr-context --branch \"${CI_COMMIT_BRANCH}\"\
+     \ --changed-files-out changed_files.txt --labels-out pr_labels.txt </dev/null\
+     \ \\\n    || { echo \"owners.smp-pr-context failed; resolving default set.\";\
+     \ : > changed_files.txt; : > pr_labels.txt; }\n  LABELS=$(cat pr_labels.txt 2>/dev/null\
+     \ || true)\n  # Resolve the run set. On ANY failure or empty result, fall back\
+     \ to the always-run quality\n  # gates so a selection-tooling problem (missing\
+     \ dda, resolve error, network) never silently\n  # drops the gates -- the worst\
+     \ case is \"only gates run\", never \"nothing runs\".\n  dda inv owners.smp-resolve\
+     \ --config-dir \"${CONFIG_DIR}\" --manifest \"${CONFIG_DIR}/selection.yaml\" --smp-bin\
+     \ ./smp --changed-files changed_files.txt --labels \"${LABELS}\" --runner container\
+     \ --exclude ebpf --out experiment_path_filter.txt </dev/null || echo \"owners.smp-resolve\
+     \ failed; falling back to quality_gates\"\n  EXPERIMENT_PATH_FILTER=$(cat experiment_path_filter.txt\
+     \ 2>/dev/null || true)\n  if [ -z \"${EXPERIMENT_PATH_FILTER}\" ]; then\n    echo\
+     \ \"Resolve produced no filter (tooling failure or empty); falling back to quality_gates.\"\
+     \n    EXPERIMENT_PATH_FILTER=\"quality_gates\"\n  fi\n  echo \"Experiment path\
+     \ filter: '${EXPERIMENT_PATH_FILTER}'\"\nfi\n"
    - if [ "$RUN_CONFIG_ONLY" == "true" ]; then
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only\
      \ \\\n--target-image ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n\
      --target-config-dir ${CONFIG_DIR} \\\n--submission-metadata submission_metadata\
      \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command\
      \ failed with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      job-submission\"\n  exit $exit_code\n} \n"
    - else
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
      \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
      \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
+     \ ${CONFIG_DIR} \\\n${EXPERIMENT_PATH_FILTER:+--experiment-path-filter \"${EXPERIMENT_PATH_FILTER}\"\
-     \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^                                                          ------------
+     \ --runner container} \\\n--submission-metadata submission_metadata \\\n--tags\
?       ^^^^^^^^^^^^^^^^^^
-     \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
?                                                                        ---------------- --
+     \ ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command failed with\
?      ++++++++++++
-     \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
?                                                                               --------
+     \ code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\
?     +++++++++++++++++++
-     \ $exit_code\n} \n"
+     \n  exit $exit_code\n} \n"
?      +++++++
    - fi
    - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
    - echo "SMP Job Id is ${SMP_JOB_ID}"
    - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
      \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
      \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
      \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
      \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
      smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
      \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
    - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
    - datadog-ci junit upload --service datadog-agent outputs/junit.xml
    - datadog-ci tag --level job --tags smp_failure_mode:"none"
    - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
      \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
      \ job --tags smp_optimization_goal:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_bounds_check:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
      \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
      \ job --tags smp_bounds_check:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"failed"
    - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
      \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
      Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
      \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
      \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
      \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
      \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
      \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
      \ {})\n            comparison = results.get('comparison', [])\n            num_total\
      \ = len(comparison)\n            failed_replicates = [\n                replicate\
      \ for replicate in comparison if not replicate.get('passed', False)\n        \
      \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
      \ = num_total - num_failed\n            if failed_replicates:\n              \
      \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
      \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
      \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
      \ = True\n            else:\n                decision_record.append(\n       \
      \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
      \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
      \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
      \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
      \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
      \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
      n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
      Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
      Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"passed"
    stage: functional_test
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h10m
    variables:
      BOT_LOGIN: bot_login_metal-runners
      BOT_TOKEN: bot_token_metal-runners
      CONFIG_DIR: test/regression/ebpf/config-only
      RUN_CONFIG_ONLY: 'true'
      SMP_API_URL: metal-runners_api_url
      SMP_TEAM_NAME: metal-runners
      SMP_VERSION: v0.27.0-nightly.20260423T145046Z
    when: manual
single-machine-performance-regression_detector 
  single-machine-performance-regression_detector:
    artifacts:
      expire_in: 1 weeks
      paths:
      - submission_metadata
      - outputs/report.md
      - outputs/regression_signal.json
      - outputs/bounds_check_signal.json
      - outputs/junit.xml
      - outputs/report.json
      - outputs/decision_record.md
      when: always
    before_script:
    - echo "BASELINE_SHA=${BASELINE_SHA}" > regression_detector.env
+   id_tokens:
+     DDOCTOSTS_ID_TOKEN:
+       aud: dd-octo-sts
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - allow_failure: false
      if: $SMP_ALLOW_FAILURE == "false"
    - allow_failure: true
      when: on_success
    script:
    - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DATADOG_API_KEY
    - datadog-ci tag --level job --tags smp_failure_mode:"timeout"
    - mkdir outputs
    - AWS_NAMED_PROFILE="single-machine-performance"
    - "SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
    - "SMP_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT \"${SMP_TEAM_NAME}_team_id\"\
      ) || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $SMP_API_URL)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
    - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
    - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
    - "aws --profile single-machine-performance s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp\
      \ smp || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-fetch-cli\"\n  exit $exit_code\n}\n"
    - chmod +x smp
    - source regression_detector.env
    - echo "Baseline SHA is ${BASELINE_SHA}"
    - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
    - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
      "s3://${SMP_TEAM_ID}-smp-artifacts/information/"
    - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${BASELINE_SHA}-7-full-amd64
    - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
    - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-full-amd64
    - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
    - SMP_TAGS="ci_pipeline_id=${PARENT_PIPELINE_ID},ci_job_id=${CI_JOB_ID},ci_commit_branch=${CI_COMMIT_BRANCH},purpose=agent_ci"
    - echo "Tags passed through SMP are ${SMP_TAGS}"
    - RUST_LOG="info,aws_config::profile::credentials=error"
    - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
+   - "if [ \"${SELECT_EXPERIMENTS}\" == \"true\" ]; then\n  export GITHUB_TOKEN=$(dd-octo-sts\
+     \ token --scope DataDog/datadog-agent --policy self.gitlab.comment-pr || true)\n\
+     \  # Resolve the PR's labels + changed files. The GitHub API calls live in the\
+     \ dda task (via\n  # GithubAPI), not as curl/jq here. Empty outputs on no-PR/failure\
+     \ -> the resolver defaults.\n  dda inv owners.smp-pr-context --branch \"${CI_COMMIT_BRANCH}\"\
+     \ --changed-files-out changed_files.txt --labels-out pr_labels.txt </dev/null\
+     \ \\\n    || { echo \"owners.smp-pr-context failed; resolving default set.\";\
+     \ : > changed_files.txt; : > pr_labels.txt; }\n  LABELS=$(cat pr_labels.txt 2>/dev/null\
+     \ || true)\n  # Resolve the run set. On ANY failure or empty result, fall back\
+     \ to the always-run quality\n  # gates so a selection-tooling problem (missing\
+     \ dda, resolve error, network) never silently\n  # drops the gates -- the worst\
+     \ case is \"only gates run\", never \"nothing runs\".\n  dda inv owners.smp-resolve\
+     \ --config-dir \"${CONFIG_DIR}\" --manifest \"${CONFIG_DIR}/selection.yaml\" --smp-bin\
+     \ ./smp --changed-files changed_files.txt --labels \"${LABELS}\" --runner container\
+     \ --exclude ebpf --out experiment_path_filter.txt </dev/null || echo \"owners.smp-resolve\
+     \ failed; falling back to quality_gates\"\n  EXPERIMENT_PATH_FILTER=$(cat experiment_path_filter.txt\
+     \ 2>/dev/null || true)\n  if [ -z \"${EXPERIMENT_PATH_FILTER}\" ]; then\n    echo\
+     \ \"Resolve produced no filter (tooling failure or empty); falling back to quality_gates.\"\
+     \n    EXPERIMENT_PATH_FILTER=\"quality_gates\"\n  fi\n  echo \"Experiment path\
+     \ filter: '${EXPERIMENT_PATH_FILTER}'\"\nfi\n"
    - if [ "$RUN_CONFIG_ONLY" == "true" ]; then
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only\
      \ \\\n--target-image ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n\
      --target-config-dir ${CONFIG_DIR} \\\n--submission-metadata submission_metadata\
      \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command\
      \ failed with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      job-submission\"\n  exit $exit_code\n} \n"
    - else
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
      \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
      \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
+     \ ${CONFIG_DIR} \\\n${EXPERIMENT_PATH_FILTER:+--experiment-path-filter \"${EXPERIMENT_PATH_FILTER}\"\
-     \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^                                                          ------------
+     \ --runner container} \\\n--submission-metadata submission_metadata \\\n--tags\
?       ^^^^^^^^^^^^^^^^^^
-     \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
?                                                                        ---------------- --
+     \ ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command failed with\
?      ++++++++++++
-     \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
?                                                                               --------
+     \ code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\
?     +++++++++++++++++++
-     \ $exit_code\n} \n"
+     \n  exit $exit_code\n} \n"
?      +++++++
    - fi
    - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
    - echo "SMP Job Id is ${SMP_JOB_ID}"
    - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
      \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
      \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
      \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
      \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
      smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
      \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
    - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
    - datadog-ci junit upload --service datadog-agent outputs/junit.xml
    - datadog-ci tag --level job --tags smp_failure_mode:"none"
    - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
      \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
      \ job --tags smp_optimization_goal:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_bounds_check:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
      \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
      \ job --tags smp_bounds_check:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"failed"
    - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
      \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
      Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
      \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
      \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
      \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
      \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
      \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
      \ {})\n            comparison = results.get('comparison', [])\n            num_total\
      \ = len(comparison)\n            failed_replicates = [\n                replicate\
      \ for replicate in comparison if not replicate.get('passed', False)\n        \
      \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
      \ = num_total - num_failed\n            if failed_replicates:\n              \
      \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
      \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
      \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
      \ = True\n            else:\n                decision_record.append(\n       \
      \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
      \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
      \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
      \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
      \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
      \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
      n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
      Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
      Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"passed"
    stage: functional_test
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h10m
    variables:
      BOT_LOGIN: bot_login
      BOT_TOKEN: bot_token
      CONFIG_DIR: test/regression
+     SELECT_EXPERIMENTS: 'true'
      SMP_API_URL: api_url
      SMP_TEAM_NAME: agent
-     SMP_VERSION: v0.27.0
+     SMP_VERSION: dev-pr4702-6ac924e95

Changes Summary

Removed Modified Added Renamed
0 5 0 0

ℹ️ Diff available in the job log.

@dd-octo-sts

dd-octo-sts Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Files inventory check summary

File checks results against ancestor 2d00390b:

Results for datadog-agent_7.84.0~devel.git.214.88934b7.pipeline.131107216-1_amd64.deb:

No change detected

Results for datadog-iot-agent_7.84.0~devel.git.214.88934b7.pipeline.131107216-1_amd64.deb:

No change detected

@dd-octo-sts

dd-octo-sts Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor 2d00390
📊 Static Quality Gates Dashboard
🔗 SQG Job

33 successful checks with minimal change (< 2 KiB)
Quality gate Current Size
✅ agent_deb_amd64 759.674 MiB
✅ agent_deb_amd64_fips 712.535 MiB
✅ agent_heroku_amd64 312.013 MiB
✅ agent_msi 643.628 MiB
✅ agent_rpm_amd64 759.657 MiB
✅ agent_rpm_amd64_fips 712.518 MiB
✅ agent_rpm_arm64 735.422 MiB
✅ agent_rpm_arm64_fips 691.605 MiB
✅ agent_suse_amd64 759.657 MiB
✅ agent_suse_amd64_fips 712.518 MiB
✅ agent_suse_arm64 735.422 MiB
✅ agent_suse_arm64_fips 691.605 MiB
✅ docker_agent_amd64 818.289 MiB
✅ docker_agent_arm64 819.098 MiB
✅ docker_agent_jmx_amd64 1009.187 MiB
✅ docker_agent_jmx_arm64 998.648 MiB
✅ docker_cluster_agent_amd64 210.475 MiB
✅ docker_cluster_agent_arm64 223.456 MiB
✅ docker_cws_instrumentation_amd64 7.439 MiB
✅ docker_cws_instrumentation_arm64 6.877 MiB
✅ docker_dogstatsd_amd64 39.482 MiB
✅ docker_dogstatsd_arm64 37.560 MiB
✅ docker_host_profiler_amd64 305.814 MiB
✅ docker_host_profiler_arm64 317.125 MiB
✅ dogstatsd_deb_amd64 30.224 MiB
✅ dogstatsd_deb_arm64 28.247 MiB
✅ dogstatsd_rpm_amd64 30.224 MiB
✅ dogstatsd_suse_amd64 30.224 MiB
✅ iot_agent_deb_amd64 46.475 MiB
✅ iot_agent_deb_arm64 43.127 MiB
✅ iot_agent_deb_armhf 43.910 MiB
✅ iot_agent_rpm_amd64 46.476 MiB
✅ iot_agent_suse_amd64 46.475 MiB

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 14, 2026

Copy link
Copy Markdown

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Job ID: 835bbeda-e1f8-4ac3-9f5c-310b03477c3b

Baseline: 2d00390
Comparison: 88934b7
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
➖ quality_gate_logs % cpu utilization +1.58 [+0.69, +2.47] 1 Logs bounds checks dashboard
➖ quality_gate_metrics_logs memory utilization +0.42 [+0.16, +0.67] 1 Logs bounds checks dashboard
➖ quality_gate_security_mean_fs_load memory utilization +0.25 [+0.17, +0.32] 1 Logs bounds checks dashboard
➖ quality_gate_idle memory utilization +0.22 [+0.10, +0.34] 1 Logs bounds checks dashboard
➖ file_to_blackhole_1000ms_latency egress throughput +0.01 [-0.89, +0.91] 1 Logs
➖ file_to_blackhole_0ms_latency egress throughput +0.01 [-0.62, +0.64] 1 Logs
➖ quality_gate_idle_all_features memory utilization +0.00 [-0.04, +0.04] 1 Logs bounds checks dashboard
➖ file_to_blackhole_100ms_latency egress throughput -0.02 [-1.00, +0.95] 1 Logs
➖ file_to_blackhole_500ms_latency egress throughput -0.04 [-0.98, +0.90] 1 Logs
➖ quality_gate_security_no_fs_load memory utilization -0.40 [-0.54, -0.26] 1 Logs bounds checks dashboard
➖ quality_gate_security_idle memory utilization -0.61 [-0.71, -0.50] 1 Logs bounds checks dashboard
➖ quality_gate_private_action_runner memory utilization -0.76 [-0.88, -0.64] 1 Logs bounds checks dashboard

Bounds Checks: ❌ Failed

perf experiment bounds_check_name replicates_passed observed_value links
✅ file_to_blackhole_0ms_latency memory_usage 10/10 0.19GiB ≤ 1.20GiB
✅ file_to_blackhole_0ms_latency missed_bytes 10/10 0B = 0B
✅ file_to_blackhole_1000ms_latency memory_usage 10/10 0.21GiB ≤ 1.20GiB
✅ file_to_blackhole_1000ms_latency missed_bytes 10/10 0B = 0B
✅ file_to_blackhole_100ms_latency memory_usage 10/10 0.20GiB ≤ 1.20GiB
✅ file_to_blackhole_100ms_latency missed_bytes 10/10 0B = 0B
✅ file_to_blackhole_500ms_latency memory_usage 10/10 0.21GiB ≤ 1.20GiB
✅ file_to_blackhole_500ms_latency missed_bytes 10/10 0B = 0B
✅ quality_gate_idle intake_connections 10/10 4 = 4 bounds checks dashboard
❌ quality_gate_idle memory_usage 0/10 174.31MiB > 154MiB bounds checks dashboard
✅ quality_gate_idle total_bytes_received 10/10 745.67KiB ≤ 819.20KiB bounds checks dashboard
✅ quality_gate_idle_all_features intake_connections 10/10 4 = 4 bounds checks dashboard
✅ quality_gate_idle_all_features memory_usage 10/10 510.59MiB ≤ 512MiB bounds checks dashboard
✅ quality_gate_idle_all_features total_bytes_received 10/10 1.13MiB ≤ 1.25MiB bounds checks dashboard
✅ quality_gate_logs intake_connections 10/10 18 ≤ 40 bounds checks dashboard
❌ quality_gate_logs memory_usage 2/10 207.21MiB > 195MiB bounds checks dashboard
✅ quality_gate_logs missed_bytes 10/10 0B = 0B bounds checks dashboard
✅ quality_gate_logs total_bytes_received 10/10 264.27MiB ≤ 292MiB bounds checks dashboard
✅ quality_gate_metrics_logs cpu_usage 10/10 411.96 ≤ 2000 bounds checks dashboard
✅ quality_gate_metrics_logs intake_connections 10/10 19 ≤ 40 bounds checks dashboard
✅ quality_gate_metrics_logs memory_usage 10/10 402.92MiB ≤ 430MiB bounds checks dashboard
✅ quality_gate_metrics_logs missed_bytes 10/10 0B = 0B bounds checks dashboard
✅ quality_gate_metrics_logs total_bytes_received 10/10 0.94GiB ≤ 1.04GiB bounds checks dashboard
✅ quality_gate_private_action_runner memory_usage 10/10 71.83MiB ≤ 75MiB bounds checks dashboard
✅ quality_gate_security_idle cpu_usage 10/10 26.30 ≤ 100 bounds checks dashboard
✅ quality_gate_security_idle memory_usage 10/10 324.43MiB ≤ 330MiB bounds checks dashboard
✅ quality_gate_security_mean_fs_load cpu_usage 10/10 59.07 ≤ 200 bounds checks dashboard
✅ quality_gate_security_mean_fs_load memory_usage 10/10 307.49MiB ≤ 310MiB bounds checks dashboard
✅ quality_gate_security_no_fs_load cpu_usage 10/10 20.87 ≤ 100 bounds checks dashboard
✅ quality_gate_security_no_fs_load memory_usage 10/10 309.25MiB ≤ 320MiB bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

❌ Failed. Some Quality Gates were violated.

  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 2/10 replicas passed. Failed 8 which is > 0. Gate FAILED.
  • quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 0/10 replicas passed. Failed 10 which is > 0. Gate FAILED.
  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cmetz100