Skip to content

kmt: add ubuntu 25.10 - #42172

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 11 commits into
mainfrom
bryce.kahle/kmt-ubuntu-2510
Feb 3, 2026
Merged

kmt: add ubuntu 25.10#42172
gh-worker-dd-mergequeue-cf854d[bot] merged 11 commits into
mainfrom
bryce.kahle/kmt-ubuntu-2510

Conversation

@brycekahle

@brycekahle brycekahle commented Oct 20, 2025 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

  • Adds Ubuntu 25.10 to KMT
  • Removes Ubuntu 24.10
  • Fixes flake detection to support multiple levels of nesting
  • Adds a bunch of flakes for tests that newly fail on 25.10
  • Adds flakes for other failing tests

Motivation

Ensuring latest kernel compatibility

Describe how you validated your changes

Existing tests.

Additional Notes

@brycekahle brycekahle changed the title kmt: add ubuntu 25.04, eol 24.10 kmt: add ubuntu 25.10 Oct 20, 2025
@brycekahle brycekahle added changelog/no-changelog No changelog entry needed qa/done QA done before merge and regressions are covered by tests labels Oct 21, 2025
@brycekahle

brycekahle commented Oct 21, 2025

Copy link
Copy Markdown
Member Author

/trigger-ci

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Oct 21, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-10-21 18:45:51 UTC ℹ️ Start processing command /trigger-ci
If you need support, contact us on Slack #devflow!

2025-10-21 18:46:08 UTC ℹ️ Branch synced on Gitlab

Branch bryce.kahle/kmt-ubuntu-2510 was out-of-sync on Gitlab - it has been resynced

If you need support, contact us on Slack #devflow!

2025-10-21 18:49:58 UTC 🚨 Devflow

POST https://gitlab.ddbuild.io/api/v4/projects/DataDog/datadog-agent/pipeline: 400 {message: {base: [kmt_secagent_tests_join_arm64: one job can only need 50 others, but you have listed 53. See needs keyword documentation for more details]}}

Details 
child workflow execution error (type: devtools.Devtools_TriggerCiFromPr, workflowID: c6b8595d-62ae-484b-99b2-ba51c365679e_28, runID: 74967c66-099b-4946-95ac-d30064c9a947, initiatedEventID: 28, startedEventID: 29): child workflow execution error (type: devtools.Devtools_TriggerCi, workflowID: 74967c66-099b-4946-95ac-d30064c9a947_14, runID: b3222ecc-c1c1-4606-a250-ac9ff4eda927, initiatedEventID: 14, startedEventID: 15): activity error (type: gitlab.GitlabService_CreatePipeline, scheduledEventID: 55, startedEventID: 56, identity: ): activity ScheduleToClose timeout (type: ScheduleToClose): POST https://gitlab.ddbuild.io/api/v4/projects/DataDog/datadog-agent/pipeline: 400 {message: {base: [kmt_secagent_tests_join_arm64: one job can only need 50 others, but you have listed 53. See needs keyword documentation for more details]}} (type: ErrorResponse, retryable: true)

If you need support, contact us on Slack #devflow with those details!

@agent-platform-auto-pr

agent-platform-auto-pr Bot commented Oct 21, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Gitlab CI Configuration Changes

Modified Jobs
kmt_run_secagent_tests_arm64 
  kmt_run_secagent_tests_arm64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    allow_failure: true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_secagent_arm64
    - upload_dependencies_secagent_arm64
    - upload_secagent_tests_arm64
    parallel:
      matrix:
      - TAG:
        - ubuntu_22.04
        - ubuntu_24.04
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        - amazon_5.4
        - amazon_5.10
        - amazon_2023
        - fedora_37
        - fedora_38
        - debian_11
        - debian_12
        - oracle_8.9
        - oracle_9.3
        - rocky_8.5
        - rocky_9.3
        - rocky_9.4
        - opensuse_15.5
        TEST_SET:
        - cws_host
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - .gitlab/test/kernel_matrix_testing/security_agent.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/tests/cws/**/*
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_security_agent
    tags:
    - arch:arm64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: arm64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: security-agent
kmt_run_secagent_tests_arm64_docker 
  kmt_run_secagent_tests_arm64_docker:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    allow_failure: true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_secagent_arm64
    - upload_dependencies_secagent_arm64
    - upload_secagent_tests_arm64
    parallel:
      matrix:
      - TAG:
        - ubuntu_22.04
        - ubuntu_24.04
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        - amazon_5.4
        - amazon_5.10
        - amazon_2023
        - fedora_37
        - fedora_38
        - debian_11
        - debian_12
        - oracle_8.9
        - oracle_9.3
        - rocky_8.5
        - rocky_9.3
        - rocky_9.4
        TEST_SET:
        - cws_docker
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - .gitlab/test/kernel_matrix_testing/security_agent.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/tests/cws/**/*
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_security_agent
    tags:
    - arch:arm64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: arm64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: security-agent
kmt_run_secagent_tests_arm64_peds 
  kmt_run_secagent_tests_arm64_peds:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    allow_failure: false
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_secagent_arm64
    - upload_dependencies_secagent_arm64
    - upload_secagent_tests_arm64
    parallel:
      matrix:
      - TAG:
        - ubuntu_22.04
        - ubuntu_24.04
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        - amazon_5.4
        - amazon_5.10
        - amazon_2023
        - fedora_37
        - fedora_38
        - debian_11
        - debian_12
        - oracle_8.9
        - oracle_9.3
        - rocky_8.5
        - rocky_9.3
        - rocky_9.4
        - opensuse_15.5
        TEST_SET:
        - cws_peds
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - .gitlab/test/kernel_matrix_testing/security_agent.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/tests/cws/**/*
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_security_agent
    tags:
    - arch:arm64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: arm64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: security-agent
kmt_run_secagent_tests_x64 
  kmt_run_secagent_tests_x64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    allow_failure: true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_secagent_x64
    - upload_dependencies_secagent_x64
    - upload_secagent_tests_x64
    parallel:
      matrix:
      - TAG:
        - ubuntu_18.04
        - ubuntu_20.04
        - ubuntu_22.04
        - ubuntu_24.04
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        - amazon_4.14
        - amazon_5.4
        - amazon_5.10
        - amazon_2023
        - fedora_37
        - fedora_38
        - debian_10
        - debian_11
        - debian_12
        - centos_7.9
        - oracle_8.9
        - oracle_9.3
        - rocky_8.5
        - rocky_9.3
        - rocky_9.4
        - opensuse_15.3
        - opensuse_15.5
        - suse_12.5
        TEST_SET:
        - cws_host
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - .gitlab/test/kernel_matrix_testing/security_agent.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/tests/cws/**/*
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_security_agent
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: security-agent
kmt_run_secagent_tests_x64_docker 
  kmt_run_secagent_tests_x64_docker:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    allow_failure: true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_secagent_x64
    - upload_dependencies_secagent_x64
    - upload_secagent_tests_x64
    parallel:
      matrix:
      - TAG:
        - ubuntu_18.04
        - ubuntu_20.04
        - ubuntu_22.04
        - ubuntu_24.04
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        - amazon_4.14
        - amazon_5.4
        - amazon_5.10
        - amazon_2023
        - fedora_37
        - fedora_38
        - debian_10
        - debian_11
        - debian_12
        - centos_7.9
        - oracle_8.9
        - oracle_9.3
        - rocky_8.5
        - rocky_9.3
        - rocky_9.4
        TEST_SET:
        - cws_docker
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - .gitlab/test/kernel_matrix_testing/security_agent.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/tests/cws/**/*
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_security_agent
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: security-agent
kmt_run_secagent_tests_x64_peds 
  kmt_run_secagent_tests_x64_peds:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    allow_failure: false
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_secagent_x64
    - upload_dependencies_secagent_x64
    - upload_secagent_tests_x64
    parallel:
      matrix:
      - TAG:
        - ubuntu_18.04
        - ubuntu_20.04
        - ubuntu_22.04
        - ubuntu_24.04
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        - amazon_4.14
        - amazon_5.4
        - amazon_5.10
        - amazon_2023
        - fedora_37
        - fedora_38
        - debian_10
        - debian_11
        - debian_12
        - centos_7.9
        - oracle_8.9
        - oracle_9.3
        - rocky_8.5
        - rocky_9.3
        - rocky_9.4
        - opensuse_15.3
        - opensuse_15.5
        - suse_12.5
        TEST_SET:
        - cws_peds
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - .gitlab/test/kernel_matrix_testing/security_agent.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/tests/cws/**/*
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_security_agent
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: security-agent
kmt_run_secagent_tests_x64_required 
  kmt_run_secagent_tests_x64_required:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    allow_failure: false
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_secagent_x64
    - upload_dependencies_secagent_x64
    - upload_secagent_tests_x64
    parallel:
      matrix:
      - TAG:
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        TEST_SET:
        - cws_req
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - allow_failure: true
      if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - pkg/ebpf/**/*
        - pkg/security/**/*
        - pkg/eventmonitor/**/*
        - .gitlab/test/kernel_matrix_testing/security_agent.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/tests/cws/**/*
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/security_agent.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_security_agent
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: security-agent
kmt_run_sysprobe_tests_arm64 
  kmt_run_sysprobe_tests_arm64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_sysprobe_arm64
    - upload_dependencies_sysprobe_arm64
    - upload_sysprobe_tests_arm64
    - upload_minimized_btfs_sysprobe_arm64
    parallel:
      matrix:
      - TAG:
        - ubuntu_18.04
        - ubuntu_20.04
        - ubuntu_22.04
        - ubuntu_24.04
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        - amazon_4.14
        - amazon_5.4
        - amazon_5.10
        - amazon_2023
        - fedora_37
        - fedora_38
        - debian_10
        - debian_11
        - debian_12
        - centos_7.9
        - centos_8
        - rocky_9.4
        TEST_SET:
        - only_usm
        - no_usm
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - cmd/system-probe/**/*
        - pkg/collector/corechecks/ebpf/**/*
        - pkg/discovery/**/*
        - pkg/ebpf/**/*
        - pkg/network/**/*
        - pkg/process/monitor/*
        - pkg/security/**/*
        - pkg/util/kernel/**/*
        - pkg/dyninst/**/*
        - pkg/gpu/**/*
        - .gitlab/test/kernel_matrix_testing/system_probe.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/system_probe.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_system_probe
    tags:
    - arch:arm64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: arm64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: system-probe
kmt_run_sysprobe_tests_x64 
  kmt_run_sysprobe_tests_x64:
    after_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - mkdir -p $CI_PROJECT_DIR/logs
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" >
      $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true
    - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\""
      | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz"
      || true
    - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/
      || true
    - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      || true
    - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}"
      "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n  ssh metal_instance \"scp\
      \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ || true\n  scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\
      \ $DD_AGENT_TESTING_DIR/ || true\nfi\n"
    - dda inv -- -e kmt.tag-ci-job
    - tar -xzvf $DD_AGENT_TESTING_DIR/testjson-*.tar.gz
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz"
      out.json
    - "if [ \"$CI_JOB_STATUS\" != \"success\" ]; then\n  echo \"This test job failed.\
      \ KMT automatically retries tests, so we don't allow direct retries of this job.\"\
      \n  echo \"However, if you're confident that the problem is not related to the\
      \ tests, you can retry the job by running:\"\n  echo \"dda inv kmt.retry-failed-pipeline\
      \ --pipeline-id $CI_PIPELINE_ID\"\nfi\n"
    artifacts:
      expire_in: 2 weeks
      paths:
      - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz
      - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz
      - $CI_PROJECT_DIR/logs
      - $CI_PROJECT_DIR/pcaps
      - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token)
      || exit $?; export DD_API_KEY
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE
      || exit $?
    - echo "" >> $AWS_EC2_SSH_KEY_FILE
    - chmod 600 $AWS_EC2_SSH_KEY_FILE
    - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - PLATFORMS_FOR_COMPLEXITY_COLLECTION="debian_10 ubuntu_18.04 centos_8 opensuse_15.3
      suse_12.5 fedora_38"
    - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\
      \ | grep -qw \"${TAG}\" ; then\n  export COLLECT_COMPLEXITY=yes\nfi\n"
    - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
    needs:
    - kmt_setup_env_sysprobe_x64
    - upload_dependencies_sysprobe_x64
    - upload_sysprobe_tests_x64
    - upload_minimized_btfs_sysprobe_x64
    parallel:
      matrix:
      - TAG:
        - ubuntu_16.04
        - ubuntu_18.04
        - ubuntu_20.04
        - ubuntu_22.04
        - ubuntu_24.04
-       - ubuntu_24.10
?                 ^
+       - ubuntu_25.10
?                 ^
        - amazon_4.14
        - amazon_5.4
        - amazon_5.10
        - amazon_2023
        - fedora_37
        - fedora_38
        - debian_9
        - debian_10
        - debian_11
        - debian_12
        - centos_7.9
        - centos_8
        - rocky_9.4
        - rocky_8.4
        TEST_SET:
        - only_usm
        - no_usm
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - job_execution_timeout
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_KMT_TESTS == 'on'
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - cmd/system-probe/**/*
        - pkg/collector/corechecks/ebpf/**/*
        - pkg/discovery/**/*
        - pkg/ebpf/**/*
        - pkg/network/**/*
        - pkg/process/monitor/*
        - pkg/security/**/*
        - pkg/util/kernel/**/*
        - pkg/dyninst/**/*
        - pkg/gpu/**/*
        - .gitlab/test/kernel_matrix_testing/system_probe.yml
        - .gitlab/test/kernel_matrix_testing/common.yml
        - .gitlab/build/source_test/ebpf.yml
        - test/new-e2e/system-probe/**/*
        - test/new-e2e/scenarios/system-probe/**/*
        - test/e2e-framework/testing/runner/**/*
        - test/e2e-framework/testing/utils/**/*
        - test/new-e2e/go.mod
        - go.mod
        - tasks/system_probe.py
        - tasks/kmt.py
        - tasks/kernel_matrix_testing/*
    - allow_failure: true
      when: manual
    script:
    - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output)
    - FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
    - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
    - FILTER_STATE="Name=instance-state-name,Values=running"
    - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
    - FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
    - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
    - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
    - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
    - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
    - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
      $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP"
      --output text --query $QUERY_INSTANCE_IDS | wc -l )
    - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n  echo \"These jobs do not permit\
      \ retries. The go tests are retried a user-specified number of times automatically.\
      \ In order to re-run the tests, you must trigger the pipeline again\"\n  'false'\n\
      fi\n"
    - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output)
    - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET
      $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET)))
      | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output)
    - GO_VERSION=$(dda inv go-version)
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
      no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\nServerAliveInterval 10\nServerAliveCountMax
      180\n" | tee -a ~/.ssh/config
    - chmod 600 ~/.ssh/config
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt"
    - ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt
      ${MICRO_VM_IP}:/job_env.txt"
    - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file
      /home/kernel-version-testing/ddvm_rsa -vm-cmd 'mount -o remount,size=5G /opt/kmt-ramfs
      && CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}
      /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root
      /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'"
    - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE
      -vm-cmd "${NESTED_VM_CMD}" -send-env-vars CI_COMMIT_SHA,DD_API_KEY
    - NO_RETRY_EXIT_CODE=42 ${CI_PROJECT_DIR}/tools/ci/retry.sh ssh metal_instance "ssh
      ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml
      -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'"
    - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && ${CI_PROJECT_DIR}/tools/ci/retry.sh
      scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log'
    stage: kernel_matrix_testing_system_probe
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h 30m
    variables:
      ARCH: x86_64
      AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      RETRY: 2
      TEST_COMPONENT: system-probe

Changes Summary

Removed Modified Added Renamed
0 9 0 0

ℹ️ Diff available in the job log.

@brycekahle
brycekahle force-pushed the bryce.kahle/kmt-ubuntu-2510 branch from f5a2621 to 0865ee2 Compare October 21, 2025 19:11
@agent-platform-auto-pr

agent-platform-auto-pr Bot commented Oct 21, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor f3e194a
📊 Static Quality Gates Dashboard

31 successful checks with minimal change (< 2 KiB)
Quality gate Current Size
✅ agent_deb_amd64 752.283 MiB
✅ agent_deb_amd64_fips 700.968 MiB
✅ agent_heroku_amd64 326.610 MiB
✅ agent_msi 662.873 MiB
✅ agent_rpm_amd64 752.267 MiB
✅ agent_rpm_amd64_fips 700.952 MiB
✅ agent_rpm_arm64 731.023 MiB
✅ agent_rpm_arm64_fips 683.074 MiB
✅ agent_suse_amd64 752.267 MiB
✅ agent_suse_amd64_fips 700.952 MiB
✅ agent_suse_arm64 731.023 MiB
✅ agent_suse_arm64_fips 683.074 MiB
✅ docker_agent_amd64 814.405 MiB
✅ docker_agent_arm64 817.794 MiB
✅ docker_agent_jmx_amd64 1005.283 MiB
✅ docker_agent_jmx_arm64 997.392 MiB
✅ docker_cluster_agent_amd64 181.170 MiB
✅ docker_cluster_agent_arm64 196.983 MiB
✅ docker_cws_instrumentation_amd64 7.135 MiB
✅ docker_cws_instrumentation_arm64 6.689 MiB
✅ docker_dogstatsd_amd64 38.816 MiB
✅ docker_dogstatsd_arm64 37.127 MiB
✅ dogstatsd_deb_amd64 30.035 MiB
✅ dogstatsd_deb_arm64 28.184 MiB
✅ dogstatsd_rpm_amd64 30.035 MiB
✅ dogstatsd_suse_amd64 30.035 MiB
✅ iot_agent_deb_amd64 43.013 MiB
✅ iot_agent_deb_arm64 40.119 MiB
✅ iot_agent_deb_armhf 40.689 MiB
✅ iot_agent_rpm_amd64 43.014 MiB
✅ iot_agent_suse_amd64 43.014 MiB
On-wire sizes (compressed)
Quality gate Change Size (prev → curr → max)
✅ agent_deb_amd64 +22.38 KiB (0.01% increase) 183.846 → 183.868 → 184.810
✅ agent_deb_amd64_fips -38.57 KiB (0.02% reduction) 172.610 → 172.572 → 173.790
✅ agent_heroku_amd64 +9.18 KiB (0.01% increase) 87.115 → 87.124 → 88.450
✅ agent_msi neutral 143.102 MiB → 143.300
✅ agent_rpm_amd64 +19.54 KiB (0.01% increase) 186.804 → 186.823 → 188.160
✅ agent_rpm_amd64_fips +20.16 KiB (0.01% increase) 175.620 → 175.640 → 176.600
✅ agent_rpm_arm64 +15.27 KiB (0.01% increase) 168.485 → 168.500 → 169.930
✅ agent_rpm_arm64_fips +17.53 KiB (0.01% increase) 159.087 → 159.104 → 160.550
✅ agent_suse_amd64 +19.54 KiB (0.01% increase) 186.804 → 186.823 → 188.160
✅ agent_suse_amd64_fips +20.16 KiB (0.01% increase) 175.620 → 175.640 → 176.600
✅ agent_suse_arm64 +15.27 KiB (0.01% increase) 168.485 → 168.500 → 169.930
✅ agent_suse_arm64_fips +17.53 KiB (0.01% increase) 159.087 → 159.104 → 160.550
✅ docker_agent_amd64 neutral 276.527 MiB → 277.400
✅ docker_agent_arm64 neutral 263.961 MiB → 266.040
✅ docker_agent_jmx_amd64 neutral 345.159 MiB → 346.020
✅ docker_agent_jmx_arm64 -5.2 KiB (0.00% reduction) 328.585 → 328.580 → 330.660
✅ docker_cluster_agent_amd64 neutral 64.001 MiB → 64.510
✅ docker_cluster_agent_arm64 neutral 60.269 MiB → 61.170
✅ docker_cws_instrumentation_amd64 neutral 2.994 MiB → 3.330
✅ docker_cws_instrumentation_arm64 neutral 2.726 MiB → 3.090
✅ docker_dogstatsd_amd64 neutral 15.023 MiB → 15.820
✅ docker_dogstatsd_arm64 neutral 14.344 MiB → 14.830
✅ dogstatsd_deb_amd64 neutral 7.938 MiB → 8.790
✅ dogstatsd_deb_arm64 neutral 6.821 MiB → 7.710
✅ dogstatsd_rpm_amd64 neutral 7.949 MiB → 8.800
✅ dogstatsd_suse_amd64 neutral 7.949 MiB → 8.800
✅ iot_agent_deb_amd64 -2.11 KiB (0.02% reduction) 11.289 → 11.287 → 12.040
✅ iot_agent_deb_arm64 neutral 9.646 MiB → 10.450
✅ iot_agent_deb_armhf neutral 9.843 MiB → 10.620
✅ iot_agent_rpm_amd64 neutral 11.305 MiB → 12.060
✅ iot_agent_suse_amd64 neutral 11.305 MiB → 12.060

@cit-pr-commenter

cit-pr-commenter Bot commented Oct 21, 2025 •
edited by cit-pr-commenter-54b7da Bot
Loading

Copy link
Copy Markdown

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: c4924c93-89c3-40ae-8a4f-7e6335f7ac6a

Baseline: f3e194a
Comparison: 473356c
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
➖ docker_containers_cpu % cpu utilization +0.84 [-2.30, +3.98] 1 Logs

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
➖ quality_gate_logs % cpu utilization +2.42 [+0.90, +3.94] 1 Logs bounds checks dashboard
➖ docker_containers_cpu % cpu utilization +0.84 [-2.30, +3.98] 1 Logs
➖ quality_gate_metrics_logs memory utilization +0.53 [+0.32, +0.74] 1 Logs bounds checks dashboard
➖ file_tree memory utilization +0.51 [+0.46, +0.56] 1 Logs
➖ ddot_metrics_sum_cumulativetodelta_exporter memory utilization +0.37 [+0.15, +0.60] 1 Logs
➖ otlp_ingest_metrics memory utilization +0.26 [+0.10, +0.42] 1 Logs
➖ docker_containers_memory memory utilization +0.17 [+0.09, +0.25] 1 Logs
➖ tcp_syslog_to_blackhole ingress throughput +0.05 [-0.02, +0.13] 1 Logs
➖ ddot_metrics_sum_cumulative memory utilization +0.05 [-0.12, +0.21] 1 Logs
➖ file_to_blackhole_100ms_latency egress throughput +0.03 [-0.01, +0.07] 1 Logs
➖ file_to_blackhole_0ms_latency egress throughput +0.03 [-0.49, +0.54] 1 Logs
➖ file_to_blackhole_1000ms_latency egress throughput +0.02 [-0.40, +0.44] 1 Logs
➖ uds_dogstatsd_to_api ingress throughput +0.00 [-0.12, +0.12] 1 Logs
➖ tcp_dd_logs_filter_exclude ingress throughput -0.00 [-0.09, +0.09] 1 Logs
➖ uds_dogstatsd_to_api_v3 ingress throughput -0.01 [-0.14, +0.11] 1 Logs
➖ file_to_blackhole_500ms_latency egress throughput -0.04 [-0.42, +0.35] 1 Logs
➖ quality_gate_idle_all_features memory utilization -0.04 [-0.07, -0.00] 1 Logs bounds checks dashboard
➖ uds_dogstatsd_20mb_12k_contexts_20_senders memory utilization -0.08 [-0.13, -0.02] 1 Logs
➖ quality_gate_idle memory utilization -0.09 [-0.13, -0.04] 1 Logs bounds checks dashboard
➖ ddot_logs memory utilization -0.22 [-0.28, -0.16] 1 Logs
➖ otlp_ingest_logs memory utilization -0.27 [-0.37, -0.17] 1 Logs
➖ ddot_metrics memory utilization -0.49 [-0.70, -0.27] 1 Logs
➖ ddot_metrics_sum_delta memory utilization -0.95 [-1.15, -0.75] 1 Logs

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed links
✅ docker_containers_cpu simple_check_run 10/10
✅ docker_containers_memory memory_usage 10/10
✅ docker_containers_memory simple_check_run 10/10
✅ file_to_blackhole_0ms_latency lost_bytes 10/10
✅ file_to_blackhole_0ms_latency memory_usage 10/10
✅ file_to_blackhole_1000ms_latency lost_bytes 10/10
✅ file_to_blackhole_1000ms_latency memory_usage 10/10
✅ file_to_blackhole_100ms_latency lost_bytes 10/10
✅ file_to_blackhole_100ms_latency memory_usage 10/10
✅ file_to_blackhole_500ms_latency lost_bytes 10/10
✅ file_to_blackhole_500ms_latency memory_usage 10/10
✅ quality_gate_idle intake_connections 10/10 bounds checks dashboard
✅ quality_gate_idle memory_usage 10/10 bounds checks dashboard
✅ quality_gate_idle_all_features intake_connections 10/10 bounds checks dashboard
✅ quality_gate_idle_all_features memory_usage 10/10 bounds checks dashboard
✅ quality_gate_logs intake_connections 10/10 bounds checks dashboard
✅ quality_gate_logs lost_bytes 10/10 bounds checks dashboard
✅ quality_gate_logs memory_usage 10/10 bounds checks dashboard
✅ quality_gate_metrics_logs cpu_usage 10/10 bounds checks dashboard
✅ quality_gate_metrics_logs intake_connections 10/10 bounds checks dashboard
✅ quality_gate_metrics_logs lost_bytes 10/10 bounds checks dashboard
✅ quality_gate_metrics_logs memory_usage 10/10 bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.

@dd-octo-sts

dd-octo-sts Bot commented Nov 6, 2025

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had activity in the past 15 days.

It will be closed in 30 days if no further activity occurs. If this pull request is still relevant, adding a comment or pushing new commits will keep it open. Also, you can always reopen the pull request if you missed the window.

Thank you for your contributions!

@dd-octo-sts dd-octo-sts Bot added the stale label Nov 6, 2025
@brycekahle
brycekahle force-pushed the bryce.kahle/kmt-ubuntu-2510 branch from 33ea9e1 to 4b502c3 Compare December 15, 2025 21:31
@brycekahle
brycekahle force-pushed the bryce.kahle/kmt-ubuntu-2510 branch from 4b502c3 to d388d94 Compare January 8, 2026 19:19
@github-actions github-actions Bot added long review PR is complex, plan time to review it and removed medium review PR review might take time labels Jan 9, 2026
@brycekahle
brycekahle force-pushed the bryce.kahle/kmt-ubuntu-2510 branch from 1ec4789 to ba2e2ce Compare January 9, 2026 23:04
@brycekahle

brycekahle commented Jan 12, 2026

Copy link
Copy Markdown
Member Author

/trigger-ci

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Jan 12, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-01-12 19:30:54 UTC ℹ️ Start processing command /trigger-ci

2026-01-12 19:31:12 UTC ℹ️ Branch synced on Gitlab

Branch bryce.kahle/kmt-ubuntu-2510 was out-of-sync on Gitlab - it has been resynced

2026-01-12 19:32:26 UTC ℹ️ Gitlab pipeline started

Started pipeline #90278448

@brycekahle
brycekahle force-pushed the bryce.kahle/kmt-ubuntu-2510 branch from d69b388 to 82cacda Compare January 13, 2026 23:14
@brycekahle
brycekahle force-pushed the bryce.kahle/kmt-ubuntu-2510 branch from 82cacda to 473356c Compare January 27, 2026 20:44
@brycekahle
brycekahle marked this pull request as ready for review January 28, 2026 21:53
@brycekahle
brycekahle requested review from a team as code owners January 28, 2026 21:53
@brycekahle
brycekahle requested a review from AmitaiBl January 28, 2026 21:53
L3n41c
L3n41c approved these changes Jan 29, 2026
View reviewed changes

@L3n41c L3n41c left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the single file owned by @DataDog/container-integrations

@brycekahle

brycekahle commented Feb 3, 2026

Copy link
Copy Markdown
Member Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Feb 3, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-02-03 02:18:10 UTC ℹ️ Start processing command /merge

2026-02-03 02:18:15 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 1h (p90).

2026-02-03 02:40:10 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit c26195f into main Feb 3, 2026
479 of 480 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the bryce.kahle/kmt-ubuntu-2510 branch February 3, 2026 02:40
@github-actions github-actions Bot added this to the 7.77.0 milestone Feb 3, 2026
justin-lesko pushed a commit that referenced this pull request Feb 3, 2026
@brycekahle @justin-lesko
kmt: add ubuntu 25.10 (#42172)
1463d2e 
### What does this PR do?

- Adds Ubuntu 25.10 to KMT
- Removes Ubuntu 24.10
- Fixes flake detection to support multiple levels of nesting
- Adds a bunch of flakes for tests that newly fail on 25.10
- Adds flakes for other failing tests

### Motivation

Ensuring latest kernel compatibility

### Describe how you validated your changes

Existing tests.

### Additional Notes


Co-authored-by: bryce.kahle <bryce.kahle@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gjulianm gjulianm gjulianm approved these changes
@L3n41c L3n41c L3n41c approved these changes
@chouetz chouetz chouetz approved these changes
+4 more reviewers
@paulcacheux paulcacheux paulcacheux approved these changes
@pimlu pimlu pimlu approved these changes
@guyarb guyarb guyarb approved these changes
@AmitaiBl AmitaiBl AmitaiBl approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

changelog/no-changelog No changelog entry needed component/system-probe long review PR is complex, plan time to review it qa/done QA done before merge and regressions are covered by tests stale team/ebpf-platform

Projects

None yet

Milestone

7.77.0

Development

Successfully merging this pull request may close these issues.

8 participants

@brycekahle @gjulianm @L3n41c @paulcacheux @pimlu @guyarb @chouetz @AmitaiBl