diff --git a/.agents/skills/run-e2e/SKILL.md b/.agents/skills/run-e2e/SKILL.md index e4825cd40d36..b474a1e4f9a1 100644 --- a/.agents/skills/run-e2e/SKILL.md +++ b/.agents/skills/run-e2e/SKILL.md @@ -1,98 +1,246 @@ --- name: run-e2e -description: Run E2E tests locally using the new-e2e framework with Pulumi-based infrastructure -allowed-tools: Bash, Read, Glob, Grep -argument-hint: " [--run TestName] [--keep-stack] [--configparams key=value]" -model: sonnet +description: >- + Run one already-written new-e2e test locally and triage the setup failures that stop it — "run the + containers e2e tests", "my e2e run fails before any test starts". +allowed-tools: Bash, Read, AskUserQuestion +argument-hint: "[target, e.g. ./tests/agent-subcommands/flare] [TestName] [--host] [--keep-stack] [--stack-name-suffix ]" --- -Run E2E tests from `test/new-e2e/tests/` using `dda inv new-e2e-tests.run`. +Run a single new-e2e target with `dda inv -- new-e2e-tests.run`. Most targets provision real +infrastructure in a cloud account — usually AWS, sometimes GCP or Azure — though the framework also +has local provisioners that cost nothing but time. How long a run takes is a property of the target: +minutes for a single VM, considerably longer for a Kubernetes cluster. Either way the aim is one +correct run rather than a fast iteration loop. -## Instructions +## Reference material -1. **Parse `$ARGUMENTS`** to determine what to run. The user may provide: - - A test directory path (e.g., `windows/install-test`, `agent-platform/upgrade`, `containers`) - - A test function name (e.g., `TestInstall`, `TestUpgrade`) - - Flags to pass through (see below) - - A combination of the above +Read these when a step calls for them rather than up front. -2. **Resolve the test target**: - - The invoke task automatically prepends `test/new-e2e/` to targets, so targets must be relative to that directory (e.g., `./tests/agent-subcommands/flare`, NOT `./test/new-e2e/tests/...`) - - If the user gives a directory like `agent-subcommands/flare`, use `./tests/agent-subcommands/flare` as `--targets` - - If the user gives a partial name, search for matching directories under `test/new-e2e/tests/` using Glob, then strip the `test/new-e2e/` prefix for the target - - If the user gives a test function name (starts with `Test`), find which package contains it using Grep under `test/new-e2e/tests/`, then set `--targets` to the package path (relative to `test/new-e2e/`) and `--run` to the test name - - If ambiguous, list the matching options and ask the user to pick one +| File | Load when | +|---|---| +| `references/devenv.md` | `devenv_e2e.py` exits non-zero, or something behaves differently inside the dev env than on the host | +| `references/setup.md` | Before offering to run `dda inv -- e2e.setup` — it explains what that writes and why it belongs on the host | +| `references/troubleshooting.md` | On the first failure that is not a test assertion, including anything that fails before the first `--- PASS`/`--- FAIL` line | +| `references/flags.md` | The request needs more than a target and a test name — a local package or image, dev mode, config-map overrides, retries | -3. **Build the command**: - ``` - dda inv new-e2e-tests.run --targets=./tests/ [flags] - ``` - IMPORTANT: `--targets` paths are relative to `test/new-e2e/`. Do NOT include `test/new-e2e/` in the target path. +## Step 1 — Get the target from the request, or ask for it -4. **Supported flags** (pass through from `$ARGUMENTS`): - - `--run ` — Only run tests matching this regex - - `--skip ` — Skip tests matching this regex - - `--keep-stack` — Keep infrastructure up after test (for debugging) - - `--configparams ` — Override Pulumi ConfigMap parameters - - `--agent-image ` — Use a specific agent image - - `--cluster-agent-image ` — Use a specific cluster agent image - - `--stack-name-suffix ` — Add suffix to stack name (useful for stuck stacks) - - `--verbose` / `--no-verbose` — Toggle verbose output (default: verbose) - - `--max-retries ` — Retry failed tests up to n times - - `--flavor ` — Package flavor (e.g., "datadog-agent") - - `--cache` — Enable test cache (disabled by default) +A target is a package path relative to `/test/new-e2e/`, such as `./tests/agent-subcommands/flare`. +Repeating that prefix inside the target is wrong, because the invoke task already resolves against +that module. -5. **Before running**, confirm the full command with the user. +Requests usually name a test rather than a package — "run the flare e2e test" is the normal shape, not +a malformed one. Use the target when the request carries one, and otherwise ask, offering any +candidates you already know of. What to avoid is the third option: searching the tree for a likely +match. A guessed target provisions the wrong thing, and you only find out once it has finished doing +so — after paying for it. -6. **Run the command** with a 60-minute timeout (infrastructure provisioning can take a while). Use `run_in_background` for the Bash tool since e2e tests are long-running. +Anchor a supplied test name — `--run '^TestFlareSuite$'`. Unanchored, `TestFlare` also selects +`TestFlareOpts` and anything else sharing the prefix. -7. **After completion**, summarize the results: which tests passed, which failed, and any useful error output. +## Step 2 — Decide where the test runs -## Prerequisites +```bash +test -f /.started && echo IN_DEVENV || echo ON_HOST +``` -The following must be configured before running e2e tests: -- `pulumi` CLI installed -- `~/.test_infra_config.yaml` exists with proper configuration +The dev env entrypoint creates `/.started` once the container is set up, so its presence means this +session is already inside one. `ON_HOST` goes to step 3A, `IN_DEVENV` to step 3B. If the request +included `--host`, go to step 3C instead. -If any prerequisite is missing, inform the user what needs to be set up. +## Step 3A — On the host (the usual case) -## Available Test Suites +```bash +python .agents/skills/run-e2e/scripts/devenv_e2e.py up --json +``` -Tests are organized under `test/new-e2e/tests/`, running `ls test/new-e2e/tests` should give the list of test packages +This starts a dev env at id `e2e-run` if one is not running, gives it the host's E2E config and +keypair, establishes Pulumi's backend, confirms the container can authenticate to AWS, and prints the +`run_prefix` to use in step 5. It is idempotent, so a reused env pays the setup cost once. +Its AWS check needs the user present: authenticating inside a new container means completing an SSO +flow there, which the env opens as a browser tab on their desktop. Tell them to expect it, and if the +check gives up, relay the `aws-vault login` command it prints and rerun `up` — that is a normal step on +a new env, not a misconfiguration. -# Issues when running tests -If the test was previously executed and that the infra is in a weird state, that Pulumi is not aware of, trying to rerun the test with the same stack can lead to strange error, like resource being replaced, while they should not exist at all yet. -To avoid that issue it is possible to execute the test with a stack with a different name, using --stack-name-suffix , please use short stack name suffix. -Stop the execution early when you detect that issue in the logs +Pass `--no-aws-check` when the target provisions locally rather than into a cloud, so a run that needs +no credentials is not gated behind that acceptance. -## Examples +Every failure path prints an actionable message, so relay that; the table is for picking the right +reference file and knowing whether the remedy belongs on the host. + +| Exit | Meaning | What to do | +|---|---|---| +| 0 | Ready | Continue to step 4, using the printed `run_prefix` | +| 2 | The host has no usable `~/.test_infra_config.yaml` | Read `references/setup.md`, then offer to run `dda inv -- e2e.setup` **on the host** and retry | +| 3 | The container would not hold this working tree — a git worktree, a checkout directory not named `datadog-agent`, `env.dev.clone-repos` enabled, or an env sitting at a different revision | Follow the remedy it prints; `references/devenv.md` explains each case. Offer `--host` if the checkout itself cannot be used | +| 4 | The container cannot authenticate to AWS | Run the `aws-vault login` it prints **inside the env** — its keyring cannot be primed from the host — then retry | +| 5 | Already inside a dev env | Step 2 misread the marker; go to step 3B | +| 6 | The env is in an error state, so its Pulumi stacks cannot be checked | Do not remove it on the user's behalf. Relay the message, which explains when recreating it is safe | +| — | The target provisions into Azure or GCP | Not handled yet: only the AWS credentials reach the container. Use `--host`; `references/devenv.md` has the detail | +| 1 or anything else | Something failed that has no dedicated remedy | Relay the message; `references/troubleshooting.md` for the symptom | + +## Step 3B — Already inside a dev env + +```bash +test -f ~/.test_infra_config.yaml && echo CONFIG_OK || echo CONFIG_MISSING +pulumi whoami >/dev/null 2>&1 && echo BACKEND_OK || echo BACKEND_MISSING +``` + +`BACKEND_MISSING` → `PULUMI_SKIP_UPDATE_CHECK=true dda inv -- e2e.setup --no-interactive`. Pulumi's +binary comes from the base image, but its plugins and backend selection live under `$HOME`, which is +not a persistent volume, so each container needs this once. Do not install Pulumi. + +`CONFIG_MISSING` → stop. This env was not prepared by this skill and has no E2E identity. Tell the +user to either recreate it with `devenv_e2e.py up` from the host, or copy their host config and +keypair in. Do not run interactive `dda inv -- e2e.setup` here — see the safeguards. + +This container also needs its own AWS authorization, which nothing on the host provides. Rather than +letting the test discover that ten minutes in, have the user establish it first if they have not +already — it opens a browser tab on their desktop and waits for them: ```bash -# Run all tests in a directory -dda inv new-e2e-tests.run --targets=./tests/windows/install-test +aws-vault login sso-agent-sandbox-account-admin-8h +``` + +With those in place, go to step 4 with: + +```bash +E2E_STACK_NAME_SUFFIX= dda inv -- new-e2e-tests.run --targets= [--run ] [flags] +``` + +The suffix matters here for the same reason the bootstrap injects one on the host path: stack names +take the container's user name, which is `dd` for everyone, so without something identifying you two +people running this suite collide in the shared cloud account. Derive it from an identity the container +already has — `git config user.email` is set from the host. -# Run a specific test -dda inv new-e2e-tests.run --targets=./tests/agent-platform/tests/upgrade --run TestUpgrade +## Step 3C — `--host` escape hatch -# Keep stack for debugging -dda inv new-e2e-tests.run --targets=./tests/containers --run TestContainerLinux --keep-stack +Verify `~/.test_infra_config.yaml` exists, `pulumi whoami` succeeds, and the host has a live AWS +session — here it is the host's own that counts, not a container's. Then run +`dda inv -- new-e2e-tests.run` directly. This is the documented path and it keeps Pulumi state on the +host where it survives, which makes it the better choice on an already-configured Linux or macOS +machine. It is not the default because an unconfigured or Windows host fails in ways the dev env +does not. -# Run with specific agent image -dda inv new-e2e-tests.run --targets=./tests/agent-platform/tests --agent-image "my-registry/agent:latest" +## Step 4 — Confirm before provisioning -# Run with stack name suffix -dda inv new-e2e-tests.run --targets=./tests/windows/install-test --stack-name-suffix 2 +Show the user, and get an explicit yes for: the exact command, the target and `--run` regex, where it +will run, what it will provision and in which account, roughly how long a target like this takes, and +whether the stack is destroyed afterwards. If you cannot tell what the target provisions, say so — that +is itself worth confirming before paying for it. +## Step 5 — Run it +Inside a dev env, prefix the command with the `run_prefix` from step 3A: + +```bash +dda env dev run -t linux-container --id e2e-run -- env \ + dda inv -- new-e2e-tests.run --targets= [--run ] [flags] +``` + +`dda env dev run` sends the arguments over SSH joined by spaces, quoting only those that contain a +space, so keep each argument space-free and pass variables through `env VAR=value` rather than a +shell prefix. A `--run` regex survives the remote shell as long as it has no `$name` in it; a +trailing `$` is safe. + +Everything else the run needs is already in the config the bootstrap installed, so do not add +absolute container paths to the command. On a Windows host, Git Bash rewrites those before they ever +reach the container — see `references/devenv.md`. + +Start it with `run_in_background: true` — these runs outlast a foreground Bash call. The first run in a +fresh env is the slowest by a wide margin, because the Go test binary compiles from a cold cache +before any infrastructure is touched; several minutes of apparent silence there is normal, and it is +another reason to keep the env around rather than recreate it per run. + +## Step 6 — Report + +``` +### E2E run — [--run ] +- Where: dev env `e2e-run` | host +- Command: +- Result: PASS | FAIL | SETUP FAILURE (failed before any test ran) +- Duration: +- Stack: — destroyed | kept (--keep-stack) +- Failures: + - — +- Diagnostics: (+ the `docker cp` command to retrieve it when it ran in a dev env) +- Next step: +``` + +A `SETUP FAILURE` is a different thing from a test failure and the distinction matters to the user, +so name it as such and take the symptom to +`references/troubleshooting.md` rather than reporting the raw +stderr. + +The report is complete when it names the exact command that ran, says whether the infrastructure was +destroyed, and for anything other than a pass either names at least one failing test with a reason or +states plainly that the run never reached a test. + +## Step 7 — Tear down + +The dev env is reusable and costs nothing while idle, so leave it running unless the user asks for +it to go. When they do: + +```bash +python .agents/skills/run-e2e/scripts/devenv_e2e.py down +``` -## Usage +It exits 6 and refuses while the container still knows about a Pulumi stack, or whenever it cannot +check — that state exists nowhere else. It prints the destroy command to run first. `--force` overrides +it, so only reach for that once you have confirmed nothing is running. -- `/run-e2e windows/install-test` — Run all Windows install tests -- `/run-e2e windows/install-test --run TestInstall` — Run only TestInstall -- `/run-e2e TestUpgrade` — Auto-find and run TestUpgrade -- `/run-e2e agent-platform --keep-stack` — Run with stack kept alive +## Safeguards + +The hard rules, in one place: + +- Get an explicit yes before provisioning (step 4). +- Never remove a dev env while a stack is live, and if a destroy fails, keep the env and say so — an + idle env costs nothing, an orphaned Kubernetes cluster does not (step 7). +- `--keep-stack` means keeping the dev env too, since the stack's state lives inside it. +- Do not run interactive `dda inv -- e2e.setup` inside a dev env. The container user is `dd`, so it + mints a second AWS keypair in the shared account and every later fresh container then fails; see + `references/setup.md`. +- Do not pass `--profile ci` or set `E2E_PROFILE=ci` on a developer machine. That skips the + local-config preflight entirely and fails further in, with a worse message. +- Run exactly the target that was asked for, one at a time (step 1). + +## Examples + +> "run TestVMSuite in ./examples" — the default path, from a host that may not be configured + +```bash +python .agents/skills/run-e2e/scripts/devenv_e2e.py up --json +dda env dev run -t linux-container --id e2e-run -- env E2E_STACK_NAME_SUFFIX=alice \ + dda inv -- new-e2e-tests.run --targets=./examples --run='^TestVMSuite$' +``` + +> "run TestDockerSuite and leave the VM up so I can look at it" — `--keep-stack` also means keeping the env + +```bash +dda env dev run -t linux-container --id e2e-run -- env \ + dda inv -- new-e2e-tests.run --targets=./tests/containers --run='^TestDockerSuite$' --keep-stack +``` + +> "TestDockerSuite failed with Pulumi wanting to replace resources, try again" — a clean stack name + +Ask the bootstrap for the suffix rather than passing `--stack-name-suffix` to the test: the flag sets +the same variable the `run_prefix` carries and would replace it, dropping the part that keeps your +stacks distinct from another developer's. + +```bash +python .agents/skills/run-e2e/scripts/devenv_e2e.py up --json --stack-name-suffix r2 +dda env dev run -t linux-container --id e2e-run -- env \ + dda inv -- new-e2e-tests.run --targets=./tests/containers --run='^TestDockerSuite$' +``` + +> "just run it here, my machine is already set up" — `--host`, skipping the container + +```bash +dda inv -- new-e2e-tests.run --targets=./examples --run='^TestVMSuite$' +``` -## Output +> "run the flare test" — the common shape: a test named, no package (step 1) -Show the user the full command before running, then report test results when done. +Offer the two or three candidate packages you already know of, or ask for the path. diff --git a/.agents/skills/run-e2e/references/devenv.md b/.agents/skills/run-e2e/references/devenv.md new file mode 100644 index 000000000000..476e128270d8 --- /dev/null +++ b/.agents/skills/run-e2e/references/devenv.md @@ -0,0 +1,209 @@ +# Running E2E tests inside a dev env + +Why the dev-env path needs a bootstrap script at all, and what each of its steps works around. +Read this when `devenv_e2e.py` exits non-zero, or when something behaves differently inside the +container than it does on the host. + +## What the container gets for free, and what it does not + +`dda env dev start` already shares the parts of the host that matter most for E2E: + +- The host's `~/.aws` is bind-mounted read-write at `/home/dd/.aws`, so the container resolves the same + profile definitions. Note that this shares configuration, not authorization — see below. +- `AWS_PROFILE`, `AWS_REGION` and `AWS_DEFAULT_REGION` are forwarded. A stray `AWS_PROFILE` on the + host therefore follows you into the container. +- `pulumi` is on `PATH` at `/usr/local/bin/pulumi`, inherited from the builder base image. + +What it does not give you: + +- **No E2E config.** `~/.test_infra_config.yaml` does not exist in a fresh container, and + `dda inv -- new-e2e-tests.run` refuses to start without it. +- **No Pulumi state directory.** Only `/var/lib/dd` and `/var/cache/dd` are persistent volumes. + `$HOME` is not, so `~/.pulumi` — the backend selection, the plugins, and the local stack state — + is recreated for each container and lost when it is removed. +- **No keypair.** The config's `privateKeyPath` and `publicKeyPath` are host-absolute paths that do + not resolve inside the container. +- **No AWS authorization, despite the shared `~/.aws`.** The framework authenticates as a profile + whose `credential_process` runs `aws-vault`, and aws-vault keeps its tokens in its own keyring, not + in `~/.aws/sso/cache`. That keyring is local to the container and there is no way to pre-authorize it + from the host, so authenticating in a new container means completing an SSO flow inside it, however + healthy the host's session is. The env's browser proxy opens that flow on your desktop, so it is + workable — but it needs a person, which is why the bootstrap warns before the step that triggers it + and gives up rather than waiting indefinitely. The bootstrap probes rather than predicts, because + the host's session state carries no information about the container's. + +## How `devenv_e2e.py up` closes the gaps + +**The keypair is mounted read-only at `/.e2e/`, not under `$HOME`.** On first start the image +entrypoint runs `chown -R dd: /home/dd` under `set -e`. A read-only mount anywhere inside `$HOME` +makes that `chown` fail, which aborts the entrypoint, so the container never reaches +`Server listening on :: port 22` and `dda env dev start` times out waiting for readiness. The error +surfaces as `Container ... is not ready`, with `chown: ... Read-only file system` further up the +container log. Mounting outside `$HOME` avoids it entirely. + +**A copy of the config is mounted read-only next to the keys, then copied again to `$HOME` inside the +container.** It has to live at `$HOME/.test_infra_config.yaml`: `getConfigFilePath` in +`/test/e2e-framework/testing/runner/local_profile.go` and `get_full_profile_path` in +`/tasks/e2e_framework/config.py` both hardcode that location and neither accepts an override. Since +`$HOME` cannot hold a read-only mount, a copy is the way in — and it has the nicer property that the +container works on its own copy, so nothing it does can reach the host's file. The script re-copies +on every `up`, so host edits propagate, and chmods it to `600` because it carries the Pulumi +passphrase and the Datadog API key. + +`dda env dev fs import` would be the obvious tool for this and is deliberately not used. It builds +its internal `mv` with paths wrapped in literal double quotes, which the nu shell passes through as +part of the filename, so it fails with `nu::shell::io::not_found` against a path like +`'"/home/dd/.test_infra_config.yaml"'` whenever the env's shell is nu — which it is by default under +`env.dev.universal-shell`. A plain `cp` of arguments the script controls sidesteps it. Expect +`fs export` to have the same problem, since it shares that quoting. + +**The key paths are rewritten in that copy, not overridden per run.** The copy the container gets is +generated on the host with `privateKeyPath` and `publicKeyPath` pointing at `/.e2e/`; everything else, +including the keypair name and both passphrases, is carried over untouched. That copy lives at +`/dda-e2e-/.test_infra_config.yaml` in a `0700` directory with mode `0600`, since it holds +the same secrets as the original, and `devenv_e2e.py down` deletes it along with the env. + +The alternative — exporting `E2E_AWS_PRIVATE_KEY_PATH` and `E2E_AWS_PUBLIC_KEY_PATH` on the test +command line, which would work because the runner builds +`parameters.NewCascadingStore(envValueStore, configFileValueStore)` and so prefers environment +variables — is avoided on purpose. **On a Windows host, MSYS path conversion rewrites absolute +POSIX-looking arguments when a command goes through Git Bash**, so `/.e2e/key.pem` arrives inside the +container as `C:/Users/.../git/.e2e/key.pem` and the run fails when it tries to reach the VM. Keeping +container paths out of the command line removes the exposure. `E2E_STACK_NAME_SUFFIX` is still passed +that way because it contains no slashes. + +If you do need to type a container path into a command yourself on Windows, prefix it with +`MSYS_NO_PATHCONV=1`. The full list of overridable values is in `references/flags.md`, sourced from +`/test/e2e-framework/testing/runner/parameters/store_env.go`. + +**`E2E_STACK_NAME_SUFFIX` is set to the host username.** `localProfile.NamePrefix()` derives the +stack prefix from the OS username, which is `dd` for every developer's container. Without a +per-developer suffix, two people running the same suite would collide on stack names and on the +resource names derived from them, in a shared cloud account. Pass `--stack-name-suffix S` to `up` and +it becomes `-S`, keeping that property. + +This is a workaround for a gap in the framework rather than the right fix. `localProfile.NamePrefix()` +already rejects usernames that do not identify a person — it maps `""` and `root` to `nouser` — and +`dd` is one it does not know about, while `ciProfile` takes its prefix as injected data rather than +guessing. The proper fix is a name-prefix runner parameter that `dda inv -- e2e.setup` writes once, after +which this injection and this paragraph both go away. Until then, two consequences are worth knowing: +the suffix has to be re-supplied on every command, and the `username` tag on provisioned resources is +derived separately inside the Pulumi program, so it reads `dd` regardless — which is why hunting for +orphaned resources by owner tag does not work from the dev-env path. + +**Pulumi's backend is established with `dda inv -- e2e.setup --no-interactive`.** The binary is already +present, but the plugins and the local-backend selection live in the ephemeral home. The +`--no-interactive` flag is what makes this safe to run inside a container: it does the Pulumi work +and skips `setup_aws_config` entirely. Never run the interactive form here — see +`references/setup.md` for what it would do to your AWS keypair. `PULUMI_SKIP_UPDATE_CHECK=true` is +set alongside it because `pulumi_version()` in `/tasks/e2e_framework/setup/pulumi.py` treats "a newer +Pulumi exists upstream" as not-installed and re-downloads it into the ephemeral home, where the +image's copy shadows it anyway. + +## The bootstrap only handles AWS targets + +It copies in and rewrites the paths for `configParams.aws` alone, and its access check probes the AWS +profile. A target provisioning into Azure or GCP would reach the container with `configParams.azure` or +`configParams.gcp` still holding host-absolute key paths, which do not resolve there, and its +credentials unverified. Run those with `--host` until this handles them; the shape of the fix is the +same three steps applied per configured provider. + +Targets that use the framework's local provisioners need none of this — no keypair, no cloud +credentials — but they still need the config file and the Pulumi backend, so the bootstrap is still the +way in. Give it `--no-aws-check` for those: the access check costs an interactive SSO acceptance, and +gating a local, cost-free run behind that is a toll for nothing. The bootstrap cannot work this out for +itself, because it never sees which target you are about to run. + +## Dev env states + +The states and their transitions are documented in +`/docs/public/tutorials/dev/env.md`. What matters here is +the preconditions: `dda env dev start` accepts only `nonexistent` and `stopped`, `remove` only `error` +and `stopped`, `stop` only `started`, and `start` refuses new mount options on a `stopped` env because +a stopped env keeps its saved configuration. The script handles this: + +| State | What it does | +|---|---| +| `started` | Nothing | +| `stopped` | Resumes with `dda env dev start` and no options. Never removes it — its Pulumi state may still be needed | +| `error` | Refuses. The state means the container exited non-zero, which covers both "never started" and "ran tests, then died" — and the second holds the only copy of those stacks' state, which cannot be inspected while it is down. Recreating is the user's call | +| `nonexistent` | Starts with the mounts | + +To change the mounts on an existing env, remove it first — but only once you are sure it has no live +stacks. + +The script starts envs with `--no-pull`, so a container is created from whatever image is already +local. That keeps a routine `up` from re-downloading a 12 GB image, at the cost of not picking up +image fixes on its own. To move to a current image, remove the env and pull explicitly: + +```bash +docker pull datadog/agent-dev-env-linux +``` + +## `env.dev.clone-repos` has to be off + +With that setting enabled, `dda env dev start` fetches datadog-agent from GitHub instead of +bind-mounting your checkout, so the container tests the default branch and not your changes. There is +no `--no-clone` flag to override it for a single command, so `devenv_e2e.py` detects it and asks you +to turn it off: + +```bash +dda config set env.dev.clone-repos false +``` + +It only affects envs created afterwards, so remove and recreate the env for it to take effect. + +Working around it by pre-mounting the checkout and letting the clone step skip a non-empty directory +does not work either: `git dd-clone` exits 1 in that case, `dda env dev start` reports failure, and +because `save_config()` runs only after a successful start, every later `dda env dev run` falls back +to the default shell and sends `&&` to nu, which fails. The revision check below is the backstop that +catches an env built this way regardless. + +## The checkout has to be a plain clone named `datadog-agent` + +`dda env dev start` derives the repo bind mount from the current directory's parent: it mounts +`/../datadog-agent` at `/repos/datadog-agent` and aborts with +`Local repository not found: datadog-agent` if that does not exist. + +Two consequences: + +- The checkout directory must be named `datadog-agent`. +- **A git worktree does not work.** Its `.git` is a file pointing into the main clone's + `.git/worktrees/`, which is outside every mount, so git inside the container fails. That is fatal + rather than cosmetic, because `new-e2e-tests.run` reads the commit SHA unconditionally to populate + `E2E_COMMIT_SHA`, so the run aborts before its own preflight. + +`/.agents/skills/follow-pr/create_devenv.sh` sidesteps the naming rule with +`--repo "$(git rev-parse --show-toplevel)"`, which makes the mount destination +`/repos/`. That is a POSIX-only trick: a Windows path contains a drive colon, which +makes the resulting `-v` spec unparseable. Do not copy it here. + +From a worktree, either run from the main clone or use `--host`. + +`devenv_e2e.py up` verifies this rather than trusting it: it compares `git rev-parse HEAD` on the host +with the same command in the container and refuses to continue if they differ. That catches every way +the container can end up holding the wrong code — a clone instead of a mount, or an env created before +you switched branches in a way that did not propagate — because testing the wrong revision silently is +worse than failing. + +## Windows hosts + +Pass `-t linux-container` to every `dda env dev` command, for the reason given in +`/docs/public/tutorials/dev/env.md`. The script always +passes it explicitly, since it is a no-op on other platforms. + +## Getting files out + +Test output is written to `$HOME/e2e-output//` with a `latest` symlink, resolved +from the home directory of whoever ran the test — so in the dev-env case, the container's. Use +`docker cp`, because `dda env dev fs export` shares the quoting bug described above: + +```bash +docker cp dda-linux-container-e2e-run:/home/dd/e2e-output ./e2e-output +``` + +For anything more interactive, get a shell in the env: + +```bash +dda env dev shell -t linux-container --id e2e-run +``` diff --git a/.agents/skills/run-e2e/references/flags.md b/.agents/skills/run-e2e/references/flags.md new file mode 100644 index 000000000000..8c7f275ce51f --- /dev/null +++ b/.agents/skills/run-e2e/references/flags.md @@ -0,0 +1,68 @@ +# Flags and runner variables + +Read this when the request needs more than a target and a test name — a locally built package or +image, dev mode, config-map overrides, retries — or when a single value from +`~/.test_infra_config.yaml` needs overriding for one run. + +## Test command + +`dda inv -- new-e2e-tests.run --help` describes every flag and is the reference. This file only groups +them by purpose and records the judgement calls the help text cannot make for you. + +`--tags`, `--targets`, `--configparams`, `--run` and `--skip` are repeatable; the rest take one value. + +- Choosing what runs: `--targets`, `--run`, `--skip`, `--tags`, `--osdescriptors`, `--no-recursive` +- Choosing what gets tested: `--agent-image`, `--cluster-agent-image`, `--local-package`, + `--pipeline-id`, `--flavor`, `--configparams` +- Infrastructure lifecycle: `--keep-stack`, `--stack-name-suffix`, `--max-retries`, `--timeout` +- Output: `--verbose`, `--cache`, `--logs-folder`, `--result-json`, `--junit-tar`, `--extra-flags` + +### Judgement calls + +- `--targets` resolves against `/test/new-e2e/`, so repeating that prefix inside the target is wrong. + It repeats as a flag, but prefer one target per run: each provisions its own stack. +- Anchor `--run` at both ends. `TestFlare` also selects `TestFlareOpts`. +- On the dev-env path, give `--stack-name-suffix` to `devenv_e2e.py up` rather than to the test. It sets + the same variable the bootstrap uses to keep your stacks distinct from other developers', so passing + it here replaces that instead of adding to it. +- `--keep-stack` in a dev env means keeping the env too, because the stack's state lives inside it. +- `dda build docker` is the supported way to produce and push an image for `--agent-image`; it prints + the matching command when it finishes. +- Never `--profile ci` on a developer machine. It skips the local-config preflight that exists to fail + early with a clear message, so the run fails later and less legibly instead. + +## Overriding config values for one run + +The runner resolves each parameter through +`parameters.NewCascadingStore(envValueStore, configFileValueStore)` +(`/test/e2e-framework/testing/runner/local_profile.go`), so an environment variable wins over +`~/.test_infra_config.yaml`. This is what lets the dev-env path keep the host's config file while +redirecting the key paths at the container's copies. The full mapping is +`/test/e2e-framework/testing/runner/parameters/store_env.go`; the ones that come up: + +| Variable | Overrides | +|---|---| +| `E2E_KEY_PAIR_NAME` | `configParams.aws.keyPairName` | +| `E2E_AWS_PRIVATE_KEY_PATH` | `configParams.aws.privateKeyPath` | +| `E2E_AWS_PUBLIC_KEY_PATH` | `configParams.aws.publicKeyPath` | +| `E2E_AWS_PRIVATE_KEY_PASSWORD` | `configParams.aws.privateKeyPassword` | +| `E2E_PULUMI_PASSWORD` | `configParams.pulumi.passphrase` | +| `E2E_API_KEY`, `E2E_APP_KEY` | `configParams.agent.apiKey` / `.appKey` | +| `E2E_STACK_NAME_SUFFIX` | Same as `--stack-name-suffix` | +| `E2E_DEV_MODE` | Same as `--keep-stack` | +| `E2E_EXTRA_RESOURCES_TAGS` | Extra tags on provisioned resources | +| `E2E_OUTPUT_DIR` | Where test output and diagnostics are written | +| `E2E_FAKEINTAKE_IMAGE_OVERRIDE` | The fakeintake image, instead of the pinned tag | + +Note `DD_API_KEY` is not part of this — the E2E path uses `E2E_API_KEY` and the `configParams.agent` +values, which are length-checked (32 and 40 characters). + +Pass these through `dda env dev run` with an `env VAR=value ...` prefix, for the reason given in the +skill's run step. + +## Passing raw flags to `go test` + +`--extra-flags` is appended verbatim after `-args`, which covers suite-specific flags the task does not +model. Reach for that rather than calling `go test` yourself: the invoke task is what computes the build +tags, runs the local-config preflight and exports `PULUMI_CONFIG_PASSPHRASE`, and a run that skips those +can fail for reasons that have nothing to do with the code under test. diff --git a/.agents/skills/run-e2e/references/setup.md b/.agents/skills/run-e2e/references/setup.md new file mode 100644 index 000000000000..c9436b686b03 --- /dev/null +++ b/.agents/skills/run-e2e/references/setup.md @@ -0,0 +1,45 @@ +# One-time E2E setup + +What `dda inv -- e2e.setup` does, and why it belongs on the host. Read this before offering to run it. + +Read `/docs/public/how-to/test/e2e.md` for what `dda inv -- e2e.setup` does, the AWS access it assumes, +and the Azure and GCP variants — it is the authoritative guide and this file does not restate it. What follows is only what that guide does not +cover: which machine to run setup on, and what breaks if you get that wrong. + +Two facts from it are worth having in mind here. Setup does not arrange AWS authentication itself, so +an expired session shows up as an aws-vault error partway through; `aws-vault login +sso-agent-sandbox-account-admin-8h` **on the host** fixes that, and does nothing for a container's own +authorization. And re-running is idempotent, so suggesting it is cheap even when you are not sure it is +needed. + +## Always run it on the host + +Two reasons, and they both bite hard. + +**The keypair name is derived from the OS username.** It is `e2e--`, and inside a +dev env the user is always `dd`. So an interactive setup in a container creates a *second* AWS +keypair, `e2e-agent-sandbox-dd`, in the shared sandbox account rather than reusing yours. + +**Worse, it only works once.** `_ensure_aws_keypair` in `/tasks/e2e_framework/setup/aws.py` has four +branches: keypair present both in AWS and on disk is a no-op; local files only get imported to AWS; +neither means create both; and **AWS-only is a hard failure**, deliberately, so it never silently +overwrites a keypair others might be using. A container's home directory is not persistent, so the +second fresh container hits exactly that branch — AWS has `e2e-agent-sandbox-dd`, the new container +has no key files — and fails with no way forward except deleting the keypair by hand. + +This is why the bootstrap copies the host's config and keypair into the container instead, and why the +only form of the task that is safe to run inside one is the `--no-interactive` one, which does the +Pulumi work and skips the AWS and config-file work entirely. `references/devenv.md` has the exact +command and the reason for the environment variable that accompanies it. + +## What the runner reads + +The config file is looked up at `$HOME/.test_infra_config.yaml` by both the Go runner and the invoke +preflight, and neither accepts an override — which is why the dev-env path has to place a copy there +rather than point at one. Individual values inside it can still be overridden per run through +environment variables; see `references/flags.md`. + +One side effect worth knowing: the preflight keeps `~/.aws/config` in sync by appending the SSO +profile when it is absent, for example after a role rename. `~/.aws` is bind-mounted read-write into a +dev env, so a run inside a container can write to the host's file. It is idempotent and skips when the +profile is already there. diff --git a/.agents/skills/run-e2e/references/troubleshooting.md b/.agents/skills/run-e2e/references/troubleshooting.md new file mode 100644 index 000000000000..256bb563baf3 --- /dev/null +++ b/.agents/skills/run-e2e/references/troubleshooting.md @@ -0,0 +1,61 @@ +# Troubleshooting E2E runs + +Read this on the first failure that is not a test assertion — in particular anything that fails +before the first `--- PASS` or `--- FAIL` line, which means the run never reached a test. + +The **Run it on** column is the important one. Some remedies only work on the host, because that is +where the canonical E2E config and keypair live. AWS authorization is the opposite: the host and the +container each hold their own, so authorizing one does nothing for the other. + +## Symptom to remedy + +| Symptom | Cause | Remedy | Run it on | +|---|---|---|---| +| `Local E2E config is missing or incomplete. Run \`dda inv -- e2e.setup\` once to configure` | No `~/.test_infra_config.yaml`, or it has no `configParams.aws.keyPairName` | `dda inv -- e2e.setup` — see `references/setup.md`, then re-run `devenv_e2e.py up` so the container picks up the new config | **Host.** Never the interactive form in a dev env | +| `pulumi: command not found` in a dev env | Something is wrong with the image — the base image puts Pulumi at `/usr/local/bin/pulumi` | Recreate the env; do not install Pulumi by hand | Env | +| `pulumi whoami` fails, or Pulumi cannot find a plugin, in a fresh env | Pulumi's backend selection and plugins live in `~/.pulumi`, which is not a persistent volume | `PULUMI_SKIP_UPDATE_CHECK=true dda inv -- e2e.setup --no-interactive` — this is what `devenv_e2e.py up` already does | Env | +| `No valid credentials sources found`, `ExpiredToken`, or an SSO prompt that never resolves, **in the dev env** | aws-vault's keyring is local to the container and holds no usable token. The host's session is irrelevant — a healthy one does not help, because aws-vault does not read the SSO cache in the shared `~/.aws` | `dda env dev run -t linux-container --id e2e-run -- aws-vault login sso-agent-sandbox-account-admin-8h`, which opens the flow on your desktop through the env's browser proxy, then rerun `up` | Env | +| The same symptoms **on the host**, with `--host` or during `dda inv -- e2e.setup` | The host's own SSO session has expired | `aws-vault login sso-agent-sandbox-account-admin-8h` | **Host** | +| `User: arn:aws:sts::... is not authorized to perform: ecr:BatchGetImage` | Wrong AWS account, almost always a stray `AWS_PROFILE`. It overrides the framework's own profile selection, and `dda env dev start` forwards it into the container | `unset AWS_PROFILE`, then recreate the env so it does not inherit it | **Host** | +| `error: the stack is currently locked by 1 lock(s)` | A lock left behind by an interrupted run | `dda inv -- new-e2e-tests.clean`. If that says `Cleanup supported for local state only`, run `pulumi login --local` first | Wherever the interrupted run happened | +| Pulumi plans to replace or delete resources that should not exist yet | Local Pulumi state has diverged from what is actually in the cloud | Stop the run as soon as you see it. Retry with `--stack-name-suffix ` for a clean stack, and reconcile the old one with `dda inv -- new-e2e-tests.clean -s` | Same place | +| `fatal: not a git repository`, or the run aborts before its preflight | The checkout in the container is a git worktree, whose `.git` points outside every mount. `new-e2e-tests.run` reads the commit SHA unconditionally | Run from the main clone, or use `--host`. See `references/devenv.md` | — | +| `fatal: detected dubious ownership` | The bind-mounted checkout has a different uid than the container user | `git config --global --add safe.directory /repos/datadog-agent` | Env | +| `Local repository not found: datadog-agent` from `dda env dev start` | Started from a directory whose parent has no `datadog-agent` directory | Start from a clone whose directory is named `datadog-agent`; `references/devenv.md` explains the mount rule | **Host** | +| The env's `/repos/datadog-agent` is at a different revision than your tree | Usually the env was created while `env.dev.clone-repos` was on, so it holds a shallow clone of the default branch instead of a mount | `dda config set env.dev.clone-repos false`, then recreate the env. `devenv_e2e.py up` refuses to continue and prints the commands | **Host** | +| A container path in a command arrives mangled, e.g. `C:/Users/.../git/.e2e/key.pem` instead of `/.e2e/key.pem` | MSYS path conversion, when a command with absolute POSIX-looking arguments is run through Git Bash on Windows | Prefix the command with `MSYS_NO_PATHCONV=1`. `devenv_e2e.py` avoids the issue by keeping container paths off the test command line | **Host**, Git Bash only | +| Diagnostics are missing after a failure that ran in a dev env | Output goes to `$HOME/e2e-output//` resolved from the *container's* home | `docker cp dda-linux-container-e2e-run:/home/dd/e2e-output ./e2e-output`. `dda env dev fs export` shares the quoting bug that breaks `fs import` under the nu shell | **Host** | +| The config or keypair looks wrong in a way not listed above — key missing in the region, ssh-agent not running, bad key format | Various | `dda inv -- e2e.setup.debug` diagnoses these and prints what it finds; `e2e.setup.debug-keys` covers the keypair alone | **Host** | +| Cloud resources outlive the env they were created from | The env was removed while a stack was live, taking `~/.pulumi` with it | Find them in whichever account the target provisions into, by the stack-name suffix the bootstrap injected or by team tag — not by the `username` tag, which reads `dd` for every dev-env run. Prevent it by never removing an env with live stacks | **Host**, cloud console or CLI | + +## Stack state lives in the container + +This is the failure mode worth internalising. In the dev-env path, Pulumi's local backend is inside +the container, under a `$HOME` that is not a persistent volume. Remove the container while a stack is +up and the state that could have destroyed that stack is gone, leaving whatever it provisioned with no +owner — and billing for it, if the target provisions into a cloud. + +So: destroy stacks before removing an env, and if a destroy fails, keep the env. `devenv_e2e.py down` +enforces this and only bypasses it with `--force`. + +```bash +# What the container still knows about +dda env dev run -t linux-container --id e2e-run -- pulumi stack ls --all --project e2elocal --json + +# Destroy and remove them +dda env dev run -t linux-container --id e2e-run -- dda inv -- new-e2e-tests.clean -s +``` + +## Cleanup commands + +```bash +dda inv -- new-e2e-tests.clean # remove local Pulumi locks +dda inv -- new-e2e-tests.clean -s # also destroy and remove local stacks +dda inv -- new-e2e-tests.clean --output # clear local test output +``` + +## Related + +`/.agents/skills/run-windows-e2e/references/troubleshooting.md` covers the same Pulumi-lock and +`AWS_PROFILE` ground from the Windows-suite angle, plus the crash dumps and event logs those suites +collect. diff --git a/.agents/skills/run-e2e/scripts/devenv_e2e.py b/.agents/skills/run-e2e/scripts/devenv_e2e.py new file mode 100644 index 000000000000..a03a954e029a --- /dev/null +++ b/.agents/skills/run-e2e/scripts/devenv_e2e.py @@ -0,0 +1,605 @@ +#!/usr/bin/env python3 +"""Prepare a `dda env dev` Linux container for running new-e2e tests, or tear one down. + +A fresh dev env cannot run `dda inv -- new-e2e-tests.run` as-is: the container has no +E2E config, and Pulumi's plugins and backend selection live under `$HOME`, which is +not a persistent volume. `up` closes both gaps and is idempotent, so a reused +environment pays the cost once. It prints the exact command prefix to run tests with, +because the container needs `E2E_*` overrides that only this script knows. + +Run `up` from a datadog-agent clone on the host. See /.agents/skills/run-e2e/references/devenv.md. +""" + +from __future__ import annotations + +import argparse +import copy +import functools +import getpass +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +try: + import yaml +except ImportError: + # Only `up` reads the config, so keep `down` — the path that stops stacks being orphaned — + # working without PyYAML, and report the missing dependency from `up` instead. + yaml = None + +# `dda env dev start` derives the bind mount from the current directory's parent, so the +# checkout has to be a directory of this name for the repo to appear in the container. +REPO_NAME = "datadog-agent" +CONFIG_NAME = ".test_infra_config.yaml" +CONTAINER_HOME = "/home/dd" +# Deliberately outside $HOME: the image entrypoint runs `chown -R dd: /home/dd` on first +# start, which fails on a read-only mount and aborts the whole container startup. +CONTAINER_KEY_DIR = "/.e2e" +CONTAINER_REPO = f"/repos/{REPO_NAME}" +CONTAINER_CONFIG = f"{CONTAINER_HOME}/{CONFIG_NAME}" +ENV_TYPE = "linux-container" +SSO_PROFILE = "sso-agent-sandbox-account-admin-8h" +# `dda inv -- e2e.setup` writes `[profile exec-]` with a credential_process, and that +# derived profile is the one the framework authenticates as. Deriving it here keeps a role rename +# to a single edit. Not to be confused with the AWS_PROFILE environment variable, whose presence +# is itself a common cause of failures — see /.agents/skills/run-e2e/references/troubleshooting.md. +EXEC_PROFILE = f"exec-{SSO_PROFILE}" +PULUMI_PROJECT = "e2elocal" +SCRIPT = Path(__file__).name + +# Exit codes. The skill branches on these, so each one maps to a distinct remedy. +ERROR = 1 +NO_HOST_CONFIG = 2 +UNUSABLE_CHECKOUT = 3 +NO_AWS_SESSION = 4 +ALREADY_IN_DEVENV = 5 +STACK_STILL_LIVE = 6 + + +def fail(code: int, message: str) -> None: + print(message, file=sys.stderr) + sys.exit(code) + + +def child_env() -> dict[str, str]: + """Environment for `dda` subprocesses, scrubbed of things that break it.""" + env = dict(os.environ) + # `dda` renders emoji; on a Windows console that raises UnicodeEncodeError unless + # Python is forced to UTF-8. + env["PYTHONUTF8"] = "1" + # A parent `uv run` exports these, and they make the nested `uv` that bootstraps + # `dda` resolve against the wrong environment. + for var in [k for k in env if k.startswith("UV_")] + ["VIRTUAL_ENV"]: + env.pop(var, None) + return env + + +def run( + command: list[str], *, cwd: Path | None = None, capture: bool = True, timeout: int | None = None +) -> subprocess.CompletedProcess: + try: + return subprocess.run( + command, + cwd=cwd, + env=child_env(), + capture_output=capture, + text=True, + encoding="utf-8", + errors="replace", + check=False, + timeout=timeout, + ) + except subprocess.TimeoutExpired: + return subprocess.CompletedProcess(command, returncode=1, stdout="", stderr=f"timed out after {timeout}s") + + +@functools.cache +def dda() -> str: + found = shutil.which("dda") + if found is None: + fail(ERROR, "`dda` is not on PATH. See /docs/public/setup/required.md.") + return found + + +def env_command(instance: str, subcommand: str, *args: str, executable: str | None = None) -> list[str]: + # `-t` is required on Windows hosts, where the default type is the unimplemented + # `windows-container`, and is a no-op elsewhere. + return [executable or dda(), "env", "dev", subcommand, "-t", ENV_TYPE, "--id", instance, *args] + + +def in_devenv() -> bool: + """Whether this process is already inside a Linux dev env. + + The dev env entrypoint creates /.started once it has finished setting the container up. + """ + return Path("/.started").is_file() + + +def resolve_repo_root() -> Path: + result = run(["git", "rev-parse", "--show-toplevel"]) + if result.returncode != 0: + fail(UNUSABLE_CHECKOUT, "Not inside a git repository. Run this from a datadog-agent clone.") + root = Path(result.stdout.strip()).resolve() + + # The mount is `/datadog-agent`, and git just resolved this directory, so matching + # the name is the whole check — the parent necessarily contains it. + if root.name != REPO_NAME: + fail( + UNUSABLE_CHECKOUT, + f"This checkout is at {root}, but the dev env mounts /{REPO_NAME}, so the\n" + f"checkout directory must be named {REPO_NAME}. Either run from a clone that is, or\n" + "run the tests on the host with --host.", + ) + + # In a worktree, .git is a file pointing into the main clone, which is outside every + # mount. new-e2e-tests.run reads the commit SHA unconditionally, so it would abort. + if not (root / ".git").is_dir(): + fail( + UNUSABLE_CHECKOUT, + f"{root} is a git worktree. Its .git points outside the container's mounts, so git\n" + "commands fail there and the test run aborts before its preflight. Either run from\n" + "the main clone, or run the tests on the host with --host.", + ) + return root + + +def load_host_config() -> tuple[dict, Path, Path]: + """Return the host's E2E config and the AWS keypair paths it points at. + + This is a pre-check, not the authority: `_check_e2e_local_config_or_exit` in + /tasks/new_e2e_tests.py is what actually gates the run. Checking here only buys a failure + before a container is started rather than 10 minutes into one, so if that function grows a + new requirement, expect the run to fail on it despite `up` succeeding. + """ + config_path = Path.home() / CONFIG_NAME + setup_hint = "Run `dda inv -- e2e.setup` on the host (~30s, one question, opens an SSO browser flow)." + if not config_path.is_file(): + fail(NO_HOST_CONFIG, f"{config_path} does not exist. {setup_hint}") + + config = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {} + aws = config.get("configParams", {}).get("aws") or {} + if not aws.get("keyPairName"): + fail(NO_HOST_CONFIG, f"{config_path} has no configParams.aws.keyPairName. {setup_hint}") + + keys = [] + for field in ("privateKeyPath", "publicKeyPath"): + value = aws.get(field) + if not value: + fail(NO_HOST_CONFIG, f"{config_path} has no configParams.aws.{field}. {setup_hint}") + key_path = Path(value).expanduser() + if not key_path.is_file(): + fail(NO_HOST_CONFIG, f"configParams.aws.{field} points at {key_path}, which does not exist. {setup_hint}") + keys.append(key_path) + + return config, keys[0], keys[1] + + +def container_config_dir(instance: str) -> Path: + return Path(tempfile.gettempdir()) / f"dda-e2e-{instance}" + + +def restrict_to_owner(path: Path) -> None: + """Restrict a file holding secrets so that only its owner can read it. + + Condensed from `restrict_file_to_owner` in /tasks/e2e_framework/tool.py, which cannot be + imported here because that module pulls in invoke. Windows is the common host for this + path and there `os.chmod` only toggles the read-only attribute, so the ACL has to be + replaced outright or the mode below does nothing at all. + """ + if os.name != "nt": + path.chmod(0o600) + return + + user = os.environ.get("USERNAME") or getpass.getuser() + # SYSTEM and Administrators are named by SID because their names are localized. + run([ + "icacls", + str(path), + "/inheritance:r", + "/grant:r", + f"{user}:(F)", + "*S-1-5-18:(F)", + "*S-1-5-32-544:(F)", + ]) + + +def container_key(path: Path) -> str: + """Where `up` mounts a host keyfile inside the container.""" + return f"{CONTAINER_KEY_DIR}/{path.name}" + + +def write_container_config(config: dict, instance: str, private_key: Path, public_key: Path) -> Path: + """Write a copy of the E2E config whose key paths are the container's, and return its path. + + Rewriting the paths here rather than exporting `E2E_AWS_*_KEY_PATH` at run time keeps absolute + container paths off the test command line. On a Windows host those get rewritten by MSYS path + conversion when an agent runs the command through Git Bash — `/.e2e/key.pem` silently becomes + `C:/Users/.../git/.e2e/key.pem` — and the run then fails when it tries to reach the VM. + """ + config = copy.deepcopy(config) + config["configParams"]["aws"]["privateKeyPath"] = container_key(private_key) + config["configParams"]["aws"]["publicKeyPath"] = container_key(public_key) + + # A second copy of the Pulumi passphrase and API keys, so keep the directory and the file to + # this user. `/tmp` is world-readable on Linux. + directory = container_config_dir(instance) + directory.mkdir(mode=0o700, parents=True, exist_ok=True) + path = directory / CONFIG_NAME + # Lock the file down while it is still empty, so the secrets are never briefly readable. + path.touch() + restrict_to_owner(path) + path.write_text(yaml.safe_dump(config, sort_keys=False), encoding="utf-8") + return path + + +def stack_suffix(extra: str | None) -> str: + """A per-developer stack suffix. + + Stack names are prefixed with the OS username, which is `dd` for every developer's + container, so without this two people running the same suite collide on stack and + resource names in the shared cloud account. Sanitized the same way as + `get_stack_name_prefix` in /tasks/e2e_framework/tool.py, so the suffix matches the naming + the rest of the framework produces: EKS rejects `.`, and spaces cause trouble on Windows. + """ + try: + host_user = getpass.getuser() + except Exception: # noqa: BLE001 - no password entry and no matching env var + host_user = "nouser" + host_user = host_user.replace(".", "-").replace(" ", "-") + return f"{host_user}-{extra}" if extra else host_user + + +def check_clone_repos_disabled() -> None: + """Refuse to create an env that would clone the repo instead of mounting this tree. + + Cloning makes `dda env dev start` fetch datadog-agent from GitHub rather than bind-mounting + the local checkout, so the tests would not exercise the working tree at all. There is no CLI + flag to turn it off for a single command, hence the instruction rather than an override. + + Two settings turn it on — `env.dev.clone-repos`, and `clone` under an `envs.` table, + which takes precedence over it — so both spellings are matched. `check_repo_revision` is the + real backstop; this only turns a silent wrong-code run into an early, explicable failure. + """ + if not re.search(r"^\s*clone(-repos)?\s*=\s*true", run([dda(), "config", "show"]).stdout, re.MULTILINE): + return + + config_path = run([dda(), "config", "find"]).stdout.strip() + fail( + UNUSABLE_CHECKOUT, + f"Repository cloning is enabled in {config_path or 'your dda config'}, so a new dev env\n" + "would clone datadog-agent from GitHub instead of mounting this checkout, and the tests\n" + "would not exercise your changes. Turn it off — this only affects envs created afterwards:\n" + " dda config set env.dev.clone-repos false\n" + "If that is already false, an `envs` table in the same file sets `clone` directly; remove\n" + "it. Then run this script again.", + ) + + +def env_state(instance: str) -> str: + """The dev env's state, as `dda env dev status` reports it (`State: `).""" + result = run(env_command(instance, "status")) + match = re.search(r"State:\s*(\S+)", result.stdout) + return match.group(1).lower() if match else "unknown" + + +def ensure_started(instance: str, root: Path, mounts: list[tuple[Path, str]]) -> None: + state = env_state(instance) + if state == "started": + print(f"Dev env `{instance}` is already started.", file=sys.stderr) + return + + if state == "stopped": + # A stopped env keeps its saved configuration and refuses new mount options, so + # resume it as-is. Its Pulumi state may still matter, so never remove it here. + print(f"Resuming dev env `{instance}`...", file=sys.stderr) + if run(env_command(instance, "start"), cwd=root, capture=False).returncode != 0: + fail(ERROR, f"Could not resume dev env `{instance}`. See /.agents/skills/run-e2e/references/devenv.md.") + return + + if state == "error": + # `error` is any container that exited non-zero, which covers both "never finished + # starting" and "ran tests, then died" — and the second holds the only copy of those + # stacks' state. Nothing can distinguish them while the container is down, so removing + # it automatically would risk orphaning live resources, the very thing `down` refuses + # to do. Leave the decision with the user. + fail( + STACK_STILL_LIVE, + f"Dev env `{instance}` is in an error state, so it cannot be prepared or inspected.\n" + "Its container exited: either it never finished starting, or it ran tests and died\n" + "afterwards — in which case it holds the only copy of their Pulumi state, and removing\n" + "it would orphan whatever those stacks still have running. That cannot be checked from\n" + "outside. If you know it never got as far as provisioning, recreate it:\n" + f" python {SCRIPT} down --id {instance} --force\n" + f" python {SCRIPT} up --id {instance}", + ) + + if state != "nonexistent": + # `starting`, `stopping` or a state `dda env dev status` did not report in a form we parse. + fail(ERROR, f"Dev env `{instance}` is in state `{state}`; wait for it to settle and retry.") + + check_clone_repos_disabled() + + command = env_command(instance, "start", "--no-pull") + for host_path, container_path in mounts: + # Read-only so a test run inside the container cannot corrupt the host's keypair. + # as_posix() keeps a Windows drive path parseable as a docker -v spec. + command += ["-v", f"{host_path.as_posix()}:{container_path}:ro"] + + print(f"Starting dev env `{instance}` (this boots a container)...", file=sys.stderr) + # cwd matters: the repo bind mount is derived from the current directory's parent. + if run(command, cwd=root, capture=False).returncode != 0: + fail(ERROR, f"`dda env dev start` failed for `{instance}`. See /.agents/skills/run-e2e/references/devenv.md.") + + +def check_repo_revision(instance: str, root: Path) -> None: + """Confirm the container's checkout is this working tree and not something else. + + An env created before the explicit repo mount existed, or one created while + `env.dev.clone-repos` was set, holds a shallow clone of the default branch instead. That + would silently test the wrong code, which is worse than failing. + """ + host = run(["git", "rev-parse", "HEAD"], cwd=root).stdout.strip() + result = run(env_command(instance, "run", "--", "git", "-C", CONTAINER_REPO, "rev-parse", "HEAD")) + container = result.stdout.strip().splitlines()[-1].strip() if result.returncode == 0 else "" + + if container == host: + return + + # git itself may have refused rather than the revision being wrong — a uid mismatch on the + # bind mount produces `detected dubious ownership`, which needs a different fix. + git_error = result.stderr.strip() or result.stdout.strip() + detail = "" if result.returncode == 0 else f"\ngit in the container said:\n{git_error}" + fail( + UNUSABLE_CHECKOUT, + f"The dev env's {CONTAINER_REPO} is not this working tree: it is at\n" + f" {container or ''}\n" + f"while {root} is at\n" + f" {host}\n" + "Tests there would not exercise your changes. Unless the output below points elsewhere,\n" + "recreate the env so it mounts this tree:\n" + f" python {SCRIPT} down --id {instance} --force\n" + f" python {SCRIPT} up --id {instance}\n" + f"Do that only once you are sure the env holds no live Pulumi stacks.{detail}", + ) + + +def install_config(instance: str) -> None: + """Copy the mounted E2E config into the container's home directory. + + Both the Go runner and the invoke preflight look for it at `$HOME/.test_infra_config.yaml` + and neither accepts an override, but it cannot be mounted there: the entrypoint chowns all + of `$HOME`, which fails on a read-only mount. Copying from the read-only mount keeps the + host's file untouchable while giving the container a writable copy of its own, and re-running + it on every `up` keeps that copy in step with the host. + + `dda env dev fs import` would be the obvious tool, but it quotes its paths in a way that the + nu shell passes through as part of the filename, so it fails outright when the env's shell is + nu. A plain `install` of arguments we control avoids the problem. + """ + print("Installing the E2E config in the dev env...", file=sys.stderr) + # `install -m` rather than a copy followed by a chmod: it is one round trip instead of two, + # and it never leaves the file — which carries the Pulumi passphrase and the Datadog API key — + # world-readable in between, which a copy off a Windows filesystem otherwise does. + command = env_command( + instance, "run", "--", "install", "-m", "600", f"{CONTAINER_KEY_DIR}/{CONFIG_NAME}", CONTAINER_CONFIG + ) + if run(command).returncode != 0: + fail(ERROR, f"Could not install the E2E config in `{instance}`.") + + +def ensure_pulumi_backend(instance: str) -> None: + """Select Pulumi's local backend and install its plugins inside the container. + + The binary already comes from the base image, but $HOME is not a persistent volume, + so ~/.pulumi has to be re-established for each container. + """ + if run(env_command(instance, "run", "--", "pulumi", "whoami")).returncode == 0: + print("Pulumi backend already configured.", file=sys.stderr) + return + + print("Configuring Pulumi backend and plugins in the dev env...", file=sys.stderr) + # --no-interactive keeps this to Pulumi only. The interactive path would run as the + # container user `dd` and mint a second AWS keypair, which then makes every later + # fresh container fail; see /.agents/skills/run-e2e/references/setup.md. + command = env_command( + instance, + "run", + "--", + "env", + # Otherwise a newer upstream Pulumi is treated as "not installed" and re-downloaded + # into the ephemeral home, where the image's copy shadows it anyway. + "PULUMI_SKIP_UPDATE_CHECK=true", + "dda", + "inv", + "--", + "e2e.setup", + "--no-interactive", + ) + if run(command, capture=False).returncode != 0: + fail(ERROR, "Failed to configure Pulumi in the dev env. See /.agents/skills/run-e2e/references/troubleshooting.md.") + + +def check_aws(instance: str) -> None: + """Confirm the container can assume the sandbox role before a test run depends on it. + + Asking the container is the only way to know. The host's `~/.aws` is shared with it, but the + framework authenticates through a profile whose credential_process runs aws-vault, and + aws-vault keeps its tokens in its own keyring rather than in that shared directory — so the + host's session state says nothing about whether this will work, and a healthy host session + cannot prime the container. + + Consequently this call is not passive: with no usable token in that keyring, aws-vault starts + an SSO flow and the env's browser proxy opens it on the user's desktop. Hence the warning + before it and the timeout after it — unattended, there is nobody to accept the prompt. The + timeout reaps the local process only, so an `aws` left behind inside the container may briefly + contend with the next attempt. + """ + print( + "Checking AWS access from the dev env. If its credential helper has no token yet, this\n" + "opens an AWS SSO tab on your desktop that you need to accept before it can continue.", + file=sys.stderr, + ) + result = run( + env_command(instance, "run", "--", "aws", "sts", "get-caller-identity", "--profile", EXEC_PROFILE), + timeout=180, + ) + if result.returncode == 0: + return + fail( + NO_AWS_SESSION, + "The dev env cannot authenticate to AWS. Its credential helper keeps tokens in a keyring\n" + "local to the container, which cannot be primed from the host, so authorize it there —\n" + "this opens a browser tab through the env's proxy and waits for you to accept it:\n" + f" dda env dev run -t {ENV_TYPE} --id {instance} -- aws-vault login {SSO_PROFILE}\n" + "Then run this script again.\n" + f"\naws sts said:\n{result.stderr.strip() or result.stdout.strip()}", + ) + + +def live_stacks(instance: str) -> list[str] | None: + """Stacks the container still knows about, or None if they could not be listed.""" + result = run( + env_command(instance, "run", "--", "pulumi", "stack", "ls", "--all", "--project", PULUMI_PROJECT, "--json") + ) + if result.returncode != 0: + return None + try: + return [stack["name"] for stack in json.loads(result.stdout) if "name" in stack] + except (json.JSONDecodeError, TypeError): + return None + + +def command_up(args: argparse.Namespace) -> None: + if in_devenv(): + fail( + ALREADY_IN_DEVENV, + "Already inside a dev env — run `dda inv -- new-e2e-tests.run` directly instead of\n" + "nesting another environment.", + ) + + if yaml is None: + fail(ERROR, "PyYAML is required. Install it, or run this script with the interpreter `dda` uses.") + + root = resolve_repo_root() + config, private_key, public_key = load_host_config() + container_config = write_container_config(config, args.id, private_key, public_key) + + mounts = [ + (container_config, f"{CONTAINER_KEY_DIR}/{CONFIG_NAME}"), + (private_key, container_key(private_key)), + (public_key, container_key(public_key)), + ] + ensure_started(args.id, root, mounts) + check_repo_revision(args.id, root) + install_config(args.id) + ensure_pulumi_backend(args.id) + if args.no_aws_check: + # A target on the framework's local provisioners needs no cloud credentials, and the check + # below costs an interactive SSO acceptance, so gating a local run behind it would be a + # toll for nothing. The caller has to opt out because `up` never sees the target. + print("Skipping the AWS access check; the run will fail late if the target needs it.", file=sys.stderr) + else: + check_aws(args.id) + + # Everything else the run needs is in the config copy. This one has no equivalent there, and + # being slash-free it survives Git Bash's path conversion on a Windows host. + env_args = [f"E2E_STACK_NAME_SUFFIX={stack_suffix(args.stack_name_suffix)}"] + # Spelled `dda` rather than the resolved executable path, since this gets pasted into a shell. + run_prefix = env_command(args.id, "run", "--", "env", *env_args, executable="dda") + + if args.json: + print(json.dumps({"run_prefix": run_prefix}, indent=2)) + else: + print("Dev env ready. Run tests with:") + print(" ".join(run_prefix) + " dda inv -- new-e2e-tests.run --targets=") + + +def removal_blocker(instance: str, state: str) -> str | None: + """Why this env should not be removed yet, or None if it is safe to.""" + if state == "error": + # `up` cannot resume this state either, so do not send the reader there — that would be a + # loop. Their only options are to decide it holds nothing, or to salvage it by hand. + return ( + f"Dev env `{instance}` is `error`, so its Pulumi stacks cannot be checked and it cannot\n" + "be started to check them. Pass --force once you are satisfied it never provisioned\n" + "anything, or recover its state by hand first." + ) + + if state != "started": + # Nothing is listening, so the stacks cannot be checked, but a stopped env can be resumed. + return ( + f"Dev env `{instance}` is `{state}`, so its Pulumi stacks cannot be checked. Start it\n" + f"with `python {SCRIPT} up --id {instance}` to check them, or pass --force." + ) + + stacks = live_stacks(instance) + if stacks is None: + return ( + f"Could not list Pulumi stacks in `{instance}`, so it is not safe to remove: any stack\n" + "state it holds exists nowhere else. Inspect it with\n" + f" dda env dev shell -t {ENV_TYPE} --id {instance}\n" + "or force removal with --force once you are sure nothing is running." + ) + if stacks: + return ( + f"Dev env `{instance}` still has {len(stacks)} stack(s): {', '.join(stacks)}.\n" + "Its ~/.pulumi is not a persistent volume, so removing it now would orphan whatever\n" + "those stacks still hold in the cloud. Destroy them first:\n" + f" dda env dev run -t {ENV_TYPE} --id {instance} -- dda inv -- new-e2e-tests.clean -s" + ) + return None + + +def command_down(args: argparse.Namespace) -> None: + state = env_state(args.id) + if state == "nonexistent": + print(f"Dev env `{args.id}` does not exist; nothing to remove.", file=sys.stderr) + return + + blocker = removal_blocker(args.id, state) + if blocker: + if not args.force: + fail(STACK_STILL_LIVE, blocker) + print(blocker, file=sys.stderr) + + # `stop --remove` is the only removal `dda` accepts while the container is running, and + # `remove` the only one it accepts once it is not. + removal = ("stop", "--remove") if state == "started" else ("remove",) + print(f"Removing dev env `{args.id}` (state `{state}`)...", file=sys.stderr) + if run(env_command(args.id, *removal), capture=False).returncode != 0: + fail(ERROR, f"Failed to remove dev env `{args.id}`.") + # The host-side config copy only exists to be mounted into the env that is now gone. + shutil.rmtree(container_config_dir(args.id), ignore_errors=True) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + subparsers = parser.add_subparsers(dest="command", required=True) + + up = subparsers.add_parser("up", help="start and prepare a dev env for E2E runs (idempotent)") + up.add_argument("--id", default="e2e-run", help="dev env identifier (default: e2e-run)") + up.add_argument("--json", action="store_true", help="emit machine-readable output") + up.add_argument("--stack-name-suffix", help="combined with the host username into E2E_STACK_NAME_SUFFIX") + up.add_argument( + "--no-aws-check", + action="store_true", + help="skip the AWS access check, for a target that provisions locally and needs no cloud credentials", + ) + up.set_defaults(func=command_up) + + down = subparsers.add_parser("down", help="remove a dev env, refusing while stacks are live") + down.add_argument("--id", default="e2e-run", help="dev env identifier (default: e2e-run)") + down.add_argument("--force", action="store_true", help="remove even if stacks are live or unknown") + down.set_defaults(func=command_down) + + args = parser.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/AGENTS.md b/AGENTS.md index c3144f8fde72..3a567547ee2a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -102,7 +102,8 @@ Go tests run via `dda inv test --targets=` (see the `dda inv` table abo - Key docs: `test/e2e-framework/AGENTS.md` (framework), `test/fakeintake/AGENTS.md` (intake mock), `docs/public/how-to/test/e2e.md` (setup & running) - Use `/write-e2e` skill or read those docs directly to write new E2E tests -- Run locally: `dda inv new-e2e-tests.run --targets=./tests//...` +- Run locally: `dda inv new-e2e-tests.run --targets=./tests//...`, or use the `/run-e2e` + skill, which runs the test in a `dda env dev` sandbox and triages setup failures ### Manual QA - When the agent needs to be inspected in a given environment (e.g. EKS, ECS, a cloud VM) that is not easily reproducible locally, use the manual QA infrastructure. diff --git a/tasks/new_e2e_tests.py b/tasks/new_e2e_tests.py index 699330eefebc..0f4dfcf6bb43 100644 --- a/tasks/new_e2e_tests.py +++ b/tasks/new_e2e_tests.py @@ -531,13 +531,17 @@ def _compute_go_test_timeout(explicit: str | None, now: datetime.datetime | None help={ "profile": "Override auto-detected runner profile (local or CI)", "tags": "Build tags to use", - "targets": "Target packages (same as dda inv test)", + "targets": "Target packages, relative to the module (same as dda inv test). Repeatable", "configparams": "Set overrides for ConfigMap parameters (same as -c option in test-infra-definitions)", "verbose": "Verbose output: log all tests as they are run (same as gotest -v) [default: True]", - "run": "Only run tests matching the regular expression", + "run": "Only run tests matching the regular expression. Anchor it to avoid matching tests that share a prefix", "skip": "Only run tests not matching the regular expression", + "recursive": "Include subpackages of each target [default: True]", + "osdescriptors": "Restrict the run to these OS descriptors, comma-separated (e.g. 'ubuntu:22.04')", "agent_image": 'Full image path for the agent image (e.g. "repository:tag") to run the e2e tests with', "cluster_agent_image": 'Full image path for the cluster agent image (e.g. "repository:tag") to run the e2e tests with', + "local_package": "Directory holding a locally built Agent package to install instead of a published one; build one with `dda inv omnibus.build-repackaged-agent`", + "flavor": 'Agent package flavor to install (e.g. "datadog-agent")', "stack_name_suffix": "Suffix to add to the stack name, it can be useful when your stack is stuck in a weird state and you need to run the tests again", "use_prebuilt_binaries": "Use pre-built test binaries instead of building on the fly", "max_retries": "Maximum number of retries for failed tests, default 3", @@ -545,6 +549,11 @@ def _compute_go_test_timeout(explicit: str | None, now: datetime.datetime | None "keep_stack": "Keep the stack after running the test, you are responsible for destroying the stack later.", "timeout": "Go test timeout (Go duration string, e.g. '1h55m'). Defaults to CI_JOB_TIMEOUT minus a teardown buffer when running in GitLab CI, otherwise to 4h.", "pipeline_id": "GitLab pipeline ID to use; the commit SHA is automatically fetched from this pipeline for container-based tests", + "cache": "Allow the Go test cache. Disabled by default so that re-running a passing test really re-runs it", + "extra_flags": "Flags appended verbatim to the go test command after -args, for suite-specific flags this task does not model", + "logs_folder": "Directory the Agent logs collected from the environment are written to", + "result_json": "Path to write the machine-readable test results to", + "junit_tar": "Path to write a tarball of JUnit XML reports to", }, ) def run(