Add E2E CI artifact dependency generation and validation tooling#47814
Add E2E CI artifact dependency generation and validation tooling#47814
Conversation
This PR introduces tooling to generate and validate E2E test artifact dependencies from GitLab CI configuration. The CI configuration (.gitlab/test/e2e/e2e.yml) is the source of truth, and manifests (e2e-dependencies.yaml) document what artifacts each test area needs. Changes: - Add `dda e2e generate-manifests` command to generate manifests from CI - Add `dda e2e generate-ci-deps` command to validate CI against manifests - Generate 12 e2e-dependencies.yaml manifests for test areas - Add pre-commit hook to validate E2E CI dependencies - Update E2E documentation to reflect CI-as-source-of-truth workflow Key features: - Handles GitLab !reference tags and template inheritance - Resolves variables from extended templates - Detects init jobs (no artifacts needed) - Supports test-specific artifact overrides via pattern matching - Handles jobs without test patterns via MANUAL: prefix - Comprehensive validation workflow Validation: - Generated manifests for all 12 test areas - Ran `dda e2e generate-ci-deps --validate` - all pass - Verified template variable resolution works correctly - Tested MANUAL: pattern handling for jobs without --run flags - Confirmed init job detection (suffix, variable, stage)
pducolin
added
changelog/no-changelog
dd-octo-sts
bot
added
internal
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 302ebf2263
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # Skip MANUAL: patterns - they can't be automatically matched | ||
| if pattern.startswith("MANUAL:"): | ||
| continue |
There was a problem hiding this comment.
This logic unconditionally skips MANUAL: overrides, and because it only evaluates test_specific entries when test_pattern is present, jobs without a --run flag always fall back to default_artifacts. In the added manifests this already affects test/new-e2e/tests/agent-platform/e2e-dependencies.yaml (the MANUAL:new-e2e-package-signing-suse-a7-x86_64 entry), so --validate reports a mismatch for the SUSE package-signing job and the suggested non-validate "fix" path would push the wrong artifact set.
Useful? React with 👍 / 👎.
| for template_name in extends: | ||
| if template_name in ci_config: | ||
| template = ci_config[template_name] | ||
| if isinstance(template, dict) and "variables" in template: | ||
| merged_vars.update(template["variables"]) |
There was a problem hiding this comment.
Variable resolution only reads templates listed directly in extends, so it misses values inherited through a parent template chain. In .gitlab/test/e2e/e2e.yml, new-e2e-ssi-eks extends .new-e2e_ssi_extra_distribution, which gets TARGETS from .new-e2e_ssi; with the current implementation TARGETS is nil, so the job is skipped during manifest generation/validation and that area's dependencies are silently incomplete.
Useful? React with 👍 / 👎.
| # Add new artifacts | ||
| for artifact in sorted(artifacts): | ||
| new_needs.append(artifact) |
There was a problem hiding this comment.
When applying updates, artifact dependencies are re-added as bare strings, which strips optional: true from any artifact originally expressed as {"job": ..., "optional": true}. If a mismatch is fixed in write mode, jobs with optional artifact needs (for example Windows artifacts in E2E jobs) become hard requirements, which can break pipelines that intentionally allow those producers to be absent.
Useful? React with 👍 / 👎.
Files inventory check summaryFile checks results against ancestor 88d4c44f: Results for datadog-agent_7.78.0~devel.git.530.302ebf2.pipeline.102361774-1_amd64.deb:No change detected |
1 participant
What does this PR do?
Introduces tooling to generate and validate E2E test artifact dependencies from GitLab CI configuration. This establishes the GitLab CI configuration (
.gitlab/test/e2e/e2e.yml) as the source of truth, with manifests (e2e-dependencies.yaml) serving as documentation of what artifacts each test area needs. Next we'll generate the e2e.yml from these manifest, migrating the source of truth closer to tests.New Commands:
dda e2e generate-manifests- Generate manifests from current CI configurationdda e2e generate-ci-deps- Validate CI configuration against manifests #TODO remove this, have a --validate in previous commandKey Features:
!referencetags and template inheritanceTARGETSfrom.new_e2e_template_needs_*)MANUAL:prefix for manual reviewGenerated Artifacts:
e2e-dependencies.yamlmanifests across test areasMotivation
E2E tests depend on specific GitLab CI build artifacts (e.g.,
qa_agent_linux,agent_deb,deploy_*). Previously, these dependencies were implicit in CI jobneeds:sections, making it difficult to:This tooling makes dependencies explicit and provides automated validation.
Describe how you validated your changes
1. Manifest Generation:
2. Validation:
3. Template Variable Resolution:
agent-platformarea where jobs inheritTARGETSfrom templates--runpatterns4. Edge Cases:
new-e2e-containers-init) correctly detected via:-initsuffixE2E_INIT_ONLY=truevariablee2e_initstage!referencetags correctly parsed and resolvedMANUAL:prefix for review5. Pre-commit Hook:
6. Documentation:
MANIFESTS-STATUS.mdaccurately reflects current stateAdditional Notes
Non-Functional Change: This PR only adds tooling and documentation. It does not modify any test code or CI job behavior. Labels:
qa/no-code-change- No runtime code changeschangelog/no-changelog- Tooling/documentation onlyFuture Work:
dda e2e update-ci-depsto automatically update CI from manifests (if workflow reverses)Files Changed:
.dda/extend/commands/e2e/generate_manifests/__init__.py- Manifest generation command.dda/extend/commands/e2e/generate_ci_deps/__init__.py- Validation commandtest/new-e2e/tests/*/e2e-dependencies.yaml- 12 manifest filestest/new-e2e/E2E-CI-DEPENDENCIES.md- Updated workflow documentation.pre-commit-config.yaml- Added validation hook