Skip to content
Draft
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions .agents/skills/explain-lading-config/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,13 @@ Override the checkout location with `LADING_DIR` if needed.
## Step 2: Determine target file

Use `.agents/skills/explain-lading-config/scripts/resolve-lading-config.sh` to
avoid ad-hoc matching. The script enumerates experiments under
`test/regression/cases/` (active) and `test/regression/x-disabled-cases/`
(disabled). Each experiment is a `<case>/lading/lading.yaml` addressed by its
case-directory name; disabled rows are flagged with a trailing `(disabled)`
column in the listing. `ebpf/cases/` (split-mode) and
`ebpf/config-only/cases/` are intentionally out of scope; if a user asks about
avoid ad-hoc matching. The script enumerates experiments under any `cases/`
directory within `test/regression/` (e.g. `quality_gates/cases/`,
`logs/<group>/cases/`) and `test/regression/x-disabled-cases/` (disabled). Each
experiment is a `<case>/lading/lading.yaml` addressed by its case-directory
name; disabled rows are flagged with a trailing `(disabled)` column in the
listing. The `ebpf/` subtrees (split-mode and config-only) are intentionally
out of scope; if a user asks about
one, tell them this skill doesn't cover it yet.

The script handles path-like inputs, substring case names, and shell
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
#!/usr/bin/env bash
# Resolve the target lading.yaml for the explain-lading-config skill.
#
# Scope: experiments under `test/regression/cases/` and
# Scope: experiments under any `cases/` directory within `test/regression/`
# (e.g. `quality_gates/cases/`, `logs/<group>/cases/`) and
# `test/regression/x-disabled-cases/`.
# The skill deliberately does not enumerate `ebpf/cases/` (split-mode)
# or `ebpf/config-only/cases/` yet. They have different semantics.
# The skill deliberately does not enumerate the `ebpf/` subtrees (split-mode
# and config-only) yet. They have different semantics.
#
# Usage:
# resolve-lading-config.sh [ARG]
Expand All @@ -24,7 +25,7 @@
# - glob with '*' or '?' — matched against experiment names, not paths
#
# Experiment name = the case directory name, i.e. the parent of `lading/`
# in `test/regression/cases/<case>/lading/lading.yaml`.
# in `test/regression/<group>/cases/<case>/lading/lading.yaml`.

set -euo pipefail

Expand All @@ -38,9 +39,9 @@ repo_root() {
require_regression_dir() {
local root
root="$(repo_root)"
if [[ ! -d "$root/test/regression/cases" ]]; then
if [[ ! -d "$root/test/regression" ]]; then
cat >&2 <<EOF
no test/regression/cases/ directory under $root
no test/regression/ directory under $root

This script must run from inside the DataDog/datadog-agent repository.
\`cd\` into the repo (or a subdirectory of it) and re-run.
Expand All @@ -49,16 +50,15 @@ EOF
fi
}

# Emit NUL-delimited paths for all lading.yaml files under
# test/regression/cases (active) and test/regression/x-disabled-cases.
# Emit NUL-delimited paths for all lading.yaml files under test/regression,
# at any depth (e.g. quality_gates/cases/*, logs/<group>/cases/*,
# x-disabled-cases/*), excluding the ebpf/ subtrees.
find_configs() {
local root
root="$(repo_root)"
local d
for d in cases x-disabled-cases; do
[[ -d "$root/test/regression/$d" ]] || continue
find "$root/test/regression/$d" -type f -name lading.yaml -print0
done
[[ -d "$root/test/regression" ]] || return 0
find "$root/test/regression" -type f -name lading.yaml \
-not -path '*/ebpf/*' -print0
}

# Extract the display name for a lading.yaml path:
Expand Down
5 changes: 3 additions & 2 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -1025,8 +1025,9 @@
/test/benchmarks/apm_scripts/ @DataDog/agent-apm
/test/regression/ @DataDog/single-machine-performance
/test/regression/ebpf @DataDog/single-machine-performance @DataDog/ebpf-platform
/test/regression/cases/quality_gate_security_* @DataDog/single-machine-performance @DataDog/agent-security
/test/regression/cases/quality_gate_private_action_runner* @DataDog/single-machine-performance @DataDog/action-platform
/test/regression/logs/ @DataDog/agent-log-pipelines
/test/regression/quality_gates/cases/quality_gate_security_* @DataDog/single-machine-performance @DataDog/agent-security
/test/regression/quality_gates/cases/quality_gate_private_action_runner* @DataDog/single-machine-performance @DataDog/action-platform

/tools/ @DataDog/agent-devx
/tools/host-profiler/ @DataDog/profiling-full-host
Expand Down
54 changes: 54 additions & 0 deletions .github/workflows/smp-label-sync.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: "SMP label sync check"

# Keeps the SMP selection manifest (test/regression/selection.yaml) and the repo's `smp/*` labels
# in sync: the manifest's `labels:` keys are the registry, so every manifest label must exist as a
# repo label (so it can be applied), and every repo `smp/*` label must be declared in the manifest
# (no orphans). Runs when the manifest changes. (Out-of-band repo-label drift when the manifest is
# untouched would need a scheduled run — a follow-up.)

on:
pull_request:
paths:
- test/regression/selection.yaml
branches:
- main
- "[0-9]+.[0-9]+.x"

permissions: {}

jobs:
smp-label-sync:
if: github.event.pull_request.head.repo.full_name == github.repository # non-fork PRs only
runs-on: ubuntu-latest
permissions:
contents: read
Comment on lines +25 to +26

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Grant label-read permission to the sync job

With the workflow-level permissions: {} this job token only gets contents: read, but the step below calls gh label list; GitHub documents the repository-labels endpoint as requiring issues: read or pull-requests: read for GitHub App/fine-grained tokens (https://docs.github.com/rest/issues/labels#list-labels-for-a-repository). On any same-repo PR that changes selection.yaml, the label list call will 403 before the manifest can be checked, so grant one of those read scopes.

Useful? React with 👍 / 👎.

steps:
- name: Checkout selection manifest
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
sparse-checkout: test/regression/selection.yaml
persist-credentials: false
- name: Manifest labels must match repo smp/* labels
env:
GH_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
set -euo pipefail
# Labels declared in the manifest (the registry). yq is preinstalled on ubuntu-latest.
yq -r '.labels // {} | keys | .[]' test/regression/selection.yaml | sort > manifest_labels.txt
# Existing smp/* labels in the repo.
gh label list --repo "$GITHUB_REPOSITORY" --limit 500 --json name -q '.[].name' \
| grep '^smp/' | sort > repo_labels.txt || true
echo "== manifest labels =="; cat manifest_labels.txt
echo "== repo smp/* labels =="; cat repo_labels.txt
orphans=$(comm -13 manifest_labels.txt repo_labels.txt || true) # in repo, not in manifest
missing=$(comm -23 manifest_labels.txt repo_labels.txt || true) # in manifest, not in repo
rc=0
if [ -n "$orphans" ]; then
echo "::error::Repo smp/* labels not declared in the manifest (delete them or add to selection.yaml):"; echo "$orphans"; rc=1
fi
if [ -n "$missing" ]; then
echo "::error::Manifest labels not created in the repo (create them with 'gh label create'):"; echo "$missing"; rc=1
fi
[ "$rc" -eq 0 ] && echo "Manifest labels and repo smp/* labels are in sync."
exit "$rc"
51 changes: 50 additions & 1 deletion .gitlab/childs/smp-regression-child-pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ variables:
--baseline-sha ${BASELINE_SHA} \
--comparison-sha ${CI_COMMIT_SHA} \
--target-config-dir ${CONFIG_DIR} \
${EXPERIMENT_PATH_FILTER:+--experiment-path-filter "${EXPERIMENT_PATH_FILTER}" --runner container} \
--submission-metadata submission_metadata \
--tags ${SMP_TAGS} || {
exit_code=$?
Expand Down Expand Up @@ -151,6 +152,46 @@ variables:
- echo "Tags passed through SMP are ${SMP_TAGS}"
- RUST_LOG="info,aws_config::profile::credentials=error"
- RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
# PR-driven experiment selection (main container job only; gated by SELECT_EXPERIMENTS).
# Resolves CODEOWNERS involvement + applied PR labels into EXPERIMENT_PATH_FILTER, which
# .smp_job_submit forwards as --experiment-path-filter --runner container. On main / no PR the
# resolver returns the default (always-mode, e.g. quality_gates) set: no involved teams, no labels.
- |
if [ "${SELECT_EXPERIMENTS}" == "true" ]; then
export GITHUB_TOKEN=$(dd-octo-sts token --scope DataDog/datadog-agent --policy self.gitlab.comment-pr || true)
GH_API="https://api.github.com/repos/DataDog/datadog-agent"
PR_NUMBER=$(curl -sf -H "Authorization: Bearer ${GITHUB_TOKEN}" \
"${GH_API}/pulls?head=DataDog:${CI_COMMIT_BRANCH}&state=open" 2>/dev/null | jq -r '.[0].number // empty' || true)
LABELS=""
: > changed_files.txt
if [ -n "${PR_NUMBER}" ]; then
echo "Resolved PR #${PR_NUMBER} for branch ${CI_COMMIT_BRANCH}"
LABELS=$(curl -sf -H "Authorization: Bearer ${GITHUB_TOKEN}" "${GH_API}/pulls/${PR_NUMBER}" 2>/dev/null | jq -r '[.labels[].name] | join(",")' || true)
# Paginate changed files (a PR can have >100); cap at 30 pages (3000 files) and warn if hit.
page=1
while : ; do
resp=$(curl -sf -H "Authorization: Bearer ${GITHUB_TOKEN}" "${GH_API}/pulls/${PR_NUMBER}/files?per_page=100&page=${page}" 2>/dev/null || echo '[]')
names=$(echo "${resp}" | jq -r '.[].filename' 2>/dev/null || true)
[ -z "${names}" ] && break
echo "${names}" >> changed_files.txt
[ "$(echo "${resp}" | jq -r 'length' 2>/dev/null || echo 0)" -lt 100 ] && break
page=$((page+1))
if [ "${page}" -gt 30 ]; then echo "WARNING: changed-files pagination hit the 30-page cap; team involvement may be incomplete"; break; fi
done
else
echo "No open PR for branch ${CI_COMMIT_BRANCH}; resolving default set."
fi
# Resolve the run set. On ANY failure or empty result, fall back to the always-run quality
# gates so a selection-tooling problem (missing dda, resolve error, network) never silently
# drops the gates -- the worst case is "only gates run", never "nothing runs".
dda inv owners.smp-resolve --config-dir "${CONFIG_DIR}" --manifest "${CONFIG_DIR}/selection.yaml" --smp-bin ./smp --changed-files changed_files.txt --labels "${LABELS}" --runner container --exclude ebpf --out experiment_path_filter.txt </dev/null || echo "owners.smp-resolve failed; falling back to quality_gates"
EXPERIMENT_PATH_FILTER=$(cat experiment_path_filter.txt 2>/dev/null || true)
if [ -z "${EXPERIMENT_PATH_FILTER}" ]; then
echo "Resolve produced no filter (tooling failure or empty); falling back to quality_gates."
EXPERIMENT_PATH_FILTER="quality_gates"
fi
echo "Experiment path filter: '${EXPERIMENT_PATH_FILTER}'"
fi
- if [ "$RUN_CONFIG_ONLY" == "true" ]; then
- !reference [.smp_config_only_job_submit]
- else
Expand Down Expand Up @@ -288,6 +329,11 @@ variables:
single-machine-performance-regression_detector:
extends:
- .single-machine-performance-run_regression_detector
# id_tokens inlined (child pipelines don't inherit the parent's .dd_octo_sts anchor). Used by the
# SELECT_EXPERIMENTS block to mint a GitHub token for reading the PR's labels + changed files.
id_tokens:
DDOCTOSTS_ID_TOKEN:
aud: dd-octo-sts
# allow_failure is driven by SMP_ALLOW_FAILURE so the parent trigger can
# decide per-context without changing this shared child job.
rules:
Expand All @@ -298,10 +344,13 @@ single-machine-performance-regression_detector:
variables:
SMP_TEAM_NAME: agent
SMP_API_URL: api_url
SMP_VERSION: v0.27.0
SMP_VERSION: dev-pr4702-9b56514f7
BOT_LOGIN: bot_login
BOT_TOKEN: bot_token
CONFIG_DIR: test/regression
# Enables the PR-driven experiment-selection block in the shared run template. Only the container
# job selects; the metal/ebpf jobs (old SMP) leave this unset and run unchanged.
SELECT_EXPERIMENTS: "true"

single-machine-performance-metal-runners-regression_detector:
extends:
Expand Down
33 changes: 33 additions & 0 deletions .gitlab/test/functional_test/regression_detector.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,36 @@
# Config gate for the SMP experiment tree. Runs early (lint stage) so authors get fast feedback,
# and is BLOCKING: correctness problems (duplicate experiment names, malformed labels,
# Leaf-XOR-Dir violations, invalid runner) always fail; missing required metadata fails under
# --in-ci. Offline apart from downloading the smp binary. ebpf is excluded until it is modeled.
single-machine-performance-experiments-validate:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add JOBOWNERS coverage for the validation job

This new GitLab job name is not covered by .gitlab/JOBOWNERS: the SMP entry uses single_machine_performance* (underscore) and the only hyphenated SMP pattern is for the metal runner, while .gitlab/.ci-linters.yml does not allowlist this job. The jobowners-exist CI linter will therefore fail as soon as this config is linted; add a matching JOBOWNERS entry or rename the job under the existing pattern.

Useful? React with 👍 / 👎.

stage: lint
image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
tags: ["arch:amd64", "specific:true"]
needs: []
rules:
- !reference [.except_coverage_pipeline]
- changes:
paths:
- test/regression/**/*
compare_to: $COMPARE_TO_BRANCH
variables:
SMP_VERSION: dev-pr4702-9b56514f7
CONFIG_DIR: test/regression
AWS_NAMED_PROFILE: single-machine-performance
BOT_LOGIN: bot_login
BOT_TOKEN: bot_token
script:
# Fetch the SMP bot AWS creds and download the smp binary (same source as the run job).
- SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)
- SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)
- aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
- aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
- aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
- aws --profile ${AWS_NAMED_PROFILE} s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp smp
- chmod +x smp
# Correctness always fails; missing metadata fails under --in-ci.
- ./smp experiments validate --target-config-dir ${CONFIG_DIR} --manifest ${CONFIG_DIR}/selection.yaml --exclude-path ebpf --in-ci

single_machine_performance-regression_detector-merge_base_check:
stage: functional_test
timeout: 10m
Expand Down
Loading
Loading