|
| 1 | +# CI wiring |
| 2 | + |
| 3 | +Load when adding a new test package or changing which build artifacts a test consumes. A test in a package no job references never runs. |
| 4 | + |
| 5 | +Where the wiring lives: |
| 6 | + |
| 7 | +| File | Holds | |
| 8 | +|---|---| |
| 9 | +| `.gitlab-ci.yml` | The rule template that decides when the job runs | |
| 10 | +| `.gitlab/test/e2e/e2e.yml` | The job itself (Linux and cross-platform) | |
| 11 | +| `.gitlab/windows/test/e2e/windows.yml` and `.gitlab/windows/test/e2e_install_packages/windows.yml` | Windows jobs | |
| 12 | +| `.gitlab/JOBOWNERS` | Who is notified when the job fails | |
| 13 | + |
| 14 | +## The rule template |
| 15 | + |
| 16 | +Rule template names are abbreviated and do not track directory names — `tests/agent-runtimes` is gated by `.on_arun_or_e2e_changes`. Find the existing one rather than guessing: |
| 17 | + |
| 18 | +```bash |
| 19 | +grep -n '_or_e2e_changes:' .gitlab-ci.yml |
| 20 | +``` |
| 21 | + |
| 22 | +A new one references the shared branch rule and adds the paths that should trigger it: |
| 23 | + |
| 24 | +```yaml |
| 25 | +.on_myarea_or_e2e_changes: |
| 26 | + - !reference [.on_e2e_main_release_or_rc] |
| 27 | + - changes: |
| 28 | + paths: |
| 29 | + - comp/myarea/**/* |
| 30 | + - pkg/myarea/**/* |
| 31 | + - test/new-e2e/tests/myarea/**/* |
| 32 | + compare_to: $COMPARE_TO_BRANCH |
| 33 | +``` |
| 34 | +
|
| 35 | +`.on_e2e_main_release_or_rc` is what makes the job run on `main`, release branches, and release candidates regardless of the diff. The `changes` block is what additionally runs it on a pull request that touches those paths. List the implementation paths, not only the test path — otherwise a change to the feature will not exercise its own test. |
| 36 | + |
| 37 | +## The job |
| 38 | + |
| 39 | +Extend a template that already declares the right artifact dependencies rather than hand-writing `needs`: |
| 40 | + |
| 41 | +| Template | Brings in | Use for | |
| 42 | +|---|---|---| |
| 43 | +| `.new_e2e_template` | test binaries, tooling, fakeintake | Tests needing no agent package | |
| 44 | +| `.new_e2e_template_needs_deb_x64` | `agent_deb-x64-a7`, `agent_deb-x64-a7-fips` | Host tests on Ubuntu or Debian | |
| 45 | +| `.new_e2e_template_needs_container_deploy_linux` | `qa_agent_linux`, `qa_agent_linux_jmx`, `qa_dca`, `qa_dogstatsd` | Docker and Kubernetes on Linux | |
| 46 | +| `.new_e2e_template_needs_container_deploy` | the above plus the Windows agent images | Container tests covering Windows | |
| 47 | +| `.new_e2e_template_needs_windows_x64` | `windows_msi_and_bosh_zip_x64-a7` and its FIPS variant | Windows host tests (defined in `.gitlab/windows/test/e2e/windows.yml`) | |
| 48 | + |
| 49 | +```yaml |
| 50 | +new-e2e-myarea: |
| 51 | + extends: .new_e2e_template_needs_deb_x64 |
| 52 | + rules: |
| 53 | + - !reference [.on_myarea_or_e2e_changes] |
| 54 | + - !reference [.manual] |
| 55 | + variables: |
| 56 | + TARGETS: ./tests/myarea |
| 57 | + TEAM: myteam |
| 58 | + EXTRA_PARAMS: --skip "Windows" |
| 59 | + ON_NIGHTLY_FIPS: "true" |
| 60 | +``` |
| 61 | + |
| 62 | +`TARGETS` is relative to `test/new-e2e/`. `TEAM` routes test results. `EXTRA_PARAMS` passes `--run` and `--skip` regexes, which is how a suite split across `_nix_test.go` and `_win_test.go` gets divided between jobs. `ON_NIGHTLY_FIPS` also runs the job in the nightly FIPS pipeline. |
| 63 | + |
| 64 | +When none of the templates fits, compose from the shared reference so you inherit the base dependencies: |
| 65 | + |
| 66 | +```yaml |
| 67 | + needs: |
| 68 | + - !reference [.needs_new_e2e_template] |
| 69 | + - agent_rpm-x64-a7 |
| 70 | +``` |
| 71 | + |
| 72 | +Other artifact jobs: `agent_rpm-x64-a7` (RPM distributions), `deploy_windows_testing-a7` and `deploy_windows_testing-a7-fips` (Windows MSI), `deploy_installer_oci` (Fleet Automation packages). |
| 73 | + |
| 74 | +Ask for only the artifacts the test consumes. Without `needs`, GitLab waits for every earlier stage; with too many, the job blocks on builds it never uses and lengthens the pipeline for everyone. A test that hangs waiting for an image usually has a missing `needs`, not a broken test. |
| 75 | + |
| 76 | +## Windows jobs |
| 77 | + |
| 78 | +Each Windows installer test function provisions its own VM, so those jobs fan out with `parallel: matrix`, one entry per test function, and select with an anchored regex: |
| 79 | + |
| 80 | +```yaml |
| 81 | + parallel: |
| 82 | + matrix: |
| 83 | + - E2E_MSI_TEST: TestInstall |
| 84 | + - E2E_MSI_TEST: TestUpgrade |
| 85 | + variables: |
| 86 | + TARGETS: ./tests/windows/install-test |
| 87 | + EXTRA_PARAMS: --run "$E2E_MSI_TEST$" |
| 88 | +``` |
| 89 | + |
| 90 | +Adding a test function to one of those packages means adding a matrix entry, otherwise it never runs. |
| 91 | + |
| 92 | +## Pre-initialising expensive infrastructure |
| 93 | + |
| 94 | +A cluster that takes five to ten minutes to create can be built once by a separate job: |
| 95 | + |
| 96 | +```yaml |
| 97 | +new-e2e-myarea-init: |
| 98 | + extends: .new_e2e_template |
| 99 | + stage: e2e_init |
| 100 | + variables: |
| 101 | + TARGETS: ./tests/myarea |
| 102 | + E2E_INIT_ONLY: "true" |
| 103 | +
|
| 104 | +new-e2e-myarea: |
| 105 | + extends: .new_e2e_template |
| 106 | + needs: |
| 107 | + # extends replaces needs rather than merging, and the inherited before_script |
| 108 | + # unpacks these artifacts — dropping them fails the job on a missing tarball. |
| 109 | + - !reference [.needs_new_e2e_template] |
| 110 | + - new-e2e-myarea-init |
| 111 | + variables: |
| 112 | + TARGETS: ./tests/myarea |
| 113 | + E2E_PRE_INITIALIZED: "true" |
| 114 | +``` |
| 115 | + |
| 116 | +`new-e2e-containers-eks` in `.gitlab/test/e2e/e2e.yml` is the in-tree version; it re-references `.new_e2e_template_needs_container_deploy` alongside its init job for the same reason. |
| 117 | + |
| 118 | +Worth it for EKS and similar; unnecessary for a single VM. |
| 119 | + |
| 120 | +## Ownership |
| 121 | + |
| 122 | +Add the job to `.gitlab/JOBOWNERS`, which is what dispatches a failure notification: |
| 123 | + |
| 124 | +``` |
| 125 | +new-e2e-myarea* @DataDog/myteam |
| 126 | +``` |
| 127 | + |
| 128 | +`new-e2e*` defaults to `@DataDog/agent-devx`, so a job without its own entry pages the framework team instead of the team that owns the behavior. Add the test directory to `.github/CODEOWNERS` as well — that governs review, not notifications, and the two are separate on purpose. |
| 129 | + |
| 130 | +## Budgets and branch coverage |
| 131 | + |
| 132 | +`test/new-e2e/codereview_guideline.md` § "Keeping tests fast" sets the wall-time budgets. A job gated on every pull request regardless of paths is rare by design and needs justifying. |
| 133 | + |
| 134 | +Most E2E jobs run only on `main`, release branches, and release candidates. A change whose only coverage is such a job gets no pull-request signal — say so in the report, since a reviewer cannot tell from a green pipeline. That is a disclosure, not a label: a test-only pull request still takes `qa/no-code-change`, and `qa/rc-required` is reserved for changes that genuinely can only be validated on a release candidate. |
| 135 | + |
| 136 | +## Dynamic test skipping |
| 137 | + |
| 138 | +Some e2e jobs prune themselves from the inside. A job whose `rules` reference `.dynamic_tests` (`.gitlab-ci.yml`) is created on any change under `pkg/`, `cmd/`, or `comp/` — deliberately broad — and then the `--impacted` flag on `new-e2e-tests.run` consults a coverage index and skips the tests in that job the diff does not touch. It selects tests within a job, never between jobs, so it neither replaces nor relaxes the `changes` rule above. |
| 139 | + |
| 140 | +Three consequences for a test author: |
| 141 | + |
| 142 | +- A test the index does not know about is never skipped. The skip list is `indexed tests − impacted tests` (`tasks/libs/dynamic_test/index.py`), so a newly added test always runs. |
| 143 | +- Pruning happens on dev branches only. `main`, release branches, tagged commits, and triggered pipelines run everything, as does setting `RUN_E2E_TESTS=on` or the breakglass secret. |
| 144 | +- A failure to load the index is logged and the run falls back to the full suite, so a missing index costs time rather than coverage. |
| 145 | + |
| 146 | +Ask in `#agent-devx-help` when a job needs an artifact or a cloud capability that does not exist yet. |
0 commit comments