Skip to content

Commit e3f4fd6

Browse files
authored
[AI-6483] [AI-6481] Build deterministic Dispatcher test batching plans (DataDog#24687)
* Build deterministic Dispatcher test batching plans - Add changed-file target discovery, concrete job expansion, configurable batching strategies, and partition validation. - Introduce stable batch identities and use them for runner/gatherer correlation. - Preserve standalone CI matrix behavior while adding focused batching, configuration, message, and execution tests. * Carry Python version and Agent image through Dispatcher test plans Each BatchJob now records the major.minor Python its runner must set up and, when it runs E2E tests, the Agent image to run them against. The version comes from Hatch's own `python` value rather than the environment name, so repos that do not follow the `pyX.Y-` naming convention resolve correctly. - Add `ddev.e2e.agent_images`, mapping a Python version and platform to an Agent image. Superseded Agent lines are pinned to their final release; the current line tracks the development build. Base tags only, since ddev already appends `-jmx` per environment at E2E runtime. - Add `[dispatcher] default_python_version`, used where a Hatch environment declares no Python and for targets that define no environments. Also applies review findings on the planning layer: - Collapse `TestUnit` to a single environment and drop `split_environments`. Both paths produced the same job set and differed only in where the job name was built, so job naming now has one implementation. - Freeze `BatchJob` and validate batch coverage by value instead of `id()`, so a strategy may rebuild equal jobs rather than pass the originals through. - Derive capacity from the config inside the strategy and validator instead of passing it alongside the config it already lives on. - Rename the `units.Platform` namedtuple to `PlatformSpec`, which no longer collides with `messages.Platform`. - Drop `RepositoryWideRule.exempt_files`. It cancelled the whole repository-wide expansion whenever `agent_requirements.in` appeared in a change set, which never fires for dependency bumps (they do not touch the base package sources) and only fires when a change genuinely does warrant the full set. - Keep git diagnostics off stdout so they cannot reach the strict diff parser. - Move duplicated test builders into `tests.helpers.batching`. * Give platform identity a single home in ddev.utils.platform Adds `PlatformName` to the module that already owns platform-name vocabulary (`get_platform_name`, `normalize_platform_name`) and uses it for every platform value across messages, planning, and Agent image selection, replacing the `messages.Platform` enum and the string constants in `agent_images`. `normalize_platform_name` keeps returning a plain string: `platform.system()` can report systems ddev does not target, and `Platform.linux` is defined as "neither Windows nor macOS" on purpose. Raw platform strings from `.ddev/config.toml` and `manifest.json` are parsed into `PlatformName` once, in `resolve_platforms`, which names the offending target when a value is not recognized. Everything downstream holds the enum, so the unsupported-platform check no longer sits in the middle of unit expansion. Also adds `find_unpublished_images`, built on `ddev.utils.docker_registry`, so a mistyped or withdrawn tag can be caught before any job runs instead of failing every E2E job. It queries each distinct image once. `get_agent_image` stays pure and offline so plans remain deterministic and reproducible; the registry check is explicitly called rather than implied by a lookup. A `requires_ci` test uses it to assert every image in the table is served by registry.datadoghq.com. * Trim planning comments and warn on targets with no testable environment - Cut module and class docstrings to what is not already in the code, drop Sphinx roles and double backticks - Add batching/AGENTS.md describing the package for future readers - Drop a redundant str() around the platform in artifact_name - Log a warning when a target resolves no testable environment at all * Use ddev's git module for change detection instead of a batching-local one - Delete batching/git.py; ChangedFile and the diff parser now come from ddev.utils.git - Move comparison-base selection to cli/ci/tests/changes.py, outside the batching package - changes.get_changed_files takes a GitRepository rather than a bespoke GitProvider * Fold message construction into the build module assembly.py held one function with one call site and no test file of its own; building the TestBatch messages is the last step of build_test_batches. The shared jobs() test builder moves to tests.helpers.batching so both test modules can use it. * Rebalance the planning test suite toward its lower layers - Drop build-level tests that re-assert edge cases owned by targets, units, jobs or strategy - Split validate_batches tests out of test_strategy.py into test_validation.py - Share make_job, FakeIntegration, FakeRegistry, drain_queue and the changed-file builders through tests.helpers.batching instead of four near-identical copies - Collapse repeated single-case tests into parametrized tables - Cover three untested paths: an oversized integration spilling from an open batch, platforms read from manifest classifier tags, and the repository-wide rule gated on the repo name * Select rename sources through the shared affected_paths * Reconcile the gatherer tests with the batch progress aggregate Master's progress aggregate replaced the expected-batches counter and kept Platform in messages. The correlation tests now register the batch they report, the local job and queue builders defer to the shared helpers, and the new progress tests use PlatformName. * Fix repository-wide target selection and validate planned Python versions - eligible_targets asked the registry with an empty selection, which ddev resolves to changed, so a shared-framework change planned only the already changed targets - the repository-wide rule matched only a change's destination, missing a shared file renamed away - a unit-only environment's Python version reached the plan unchecked, since only E2E resolves an Agent image - FakeRegistry now honours the selection, so the first bug fails a test - the batching test helpers move to the subtree that uses them * Run unconstrained environments on every platform a target declares An environment that names no platform was routed only to the first one the target listed, and the remaining platforms fell through to a synthesised environment claiming the default Python and no E2E. That silently dropped Windows E2E for disk, dns_check, ibm_mq, network and tcp_check, and made the order of the configured platform list load-bearing. Unconstrained environments now run on every platform the target declares, so each job carries a real environment with its own Python version and facets. A target that resolves no environment at all is dropped with a warning when definitions are built rather than planned with an invented one, and a platform no environment covers is warned about and planned empty. * Stop the planner failing on platforms it has no runner for Review follow-ups on the planning package: - `resolve_platforms` parsed every `Supported OS` classifier tag before collapsing to Windows-exclusivity, so the eight manifests advertising AIX raised and aborted the whole run. `ci_matrix.py` collapses first and never validates those values, so this was a regression against existing data. The manifest list now only decides Windows-exclusivity and strict parsing is kept for the hand-written CI override, which also now rejects a repeated platform instead of planning two identically named jobs. - Fix a test importing `_normalize_agent_image_name`, renamed in master by DataDog#24639. The restack did not conflict, and the resulting collection error aborted the whole suite. - Planning failures all raise `PlanningError`. Agent-image errors are wrapped at the boundary in `jobs.py`, naming the job, rather than made to subclass it. - Drop the `default_python_version` argument nothing read; `EnvironmentProvider` owns it. - Record the `ci_matrix.py` relationship and the per-target hatch subprocess cost in the package `AGENTS.md`, and correct its docstring and protocol rules. - Drop a test of pydantic's own frozen behaviour, and fold the dependency-bump case into the parametrized set it duplicated. * Record why the unit and E2E flags are intent rather than a decision * Mark the unit and E2E facet flags for per-integration configuration
1 parent d041c3e commit e3f4fd6

35 files changed

Lines changed: 3012 additions & 242 deletions

ddev/changelog.d/24687.added

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Add deterministic Dispatcher test batch planning, with each job carrying the Python version it runs under and the Agent image its E2E tests run against. The internal ``max_jobs_per_batch`` option moves from ``[dispatcher]`` to ``[dispatcher.batching]``.
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Dispatcher Test Batching
2+
3+
This package turns a set of changed files into the batches of test jobs the Dispatcher dispatches
4+
to GitHub Actions. It is pure planning: nothing here runs tests, calls GitHub, or touches the
5+
network. See the repository-wide [AGENTS.md](../../../../../../../AGENTS.md) for general
6+
conventions.
7+
8+
## Pipeline
9+
10+
```
11+
changed files -> affected targets -> test units -> batch jobs -> job groups -> TestBatch messages
12+
(see below) targets.py units.py jobs.py strategy/ build.py
13+
```
14+
15+
Everything is composed by `build.py`, the package's public entry point, which also turns the
16+
final groups into messages. Callers use `build_test_units` or `build_test_batches` and never
17+
assemble the stages themselves.
18+
19+
Changed files arrive as `ChangedFile` records and are not produced here. `ddev.utils.git` reads
20+
them from git, and `../changes.py` decides which two commits a CI run compares.
21+
22+
| Module | Role |
23+
| --- | --- |
24+
| `build.py` | Composes the stages and adapts concrete `Repository`/`Integration` objects to them. The package's public entry point. |
25+
| `targets.py` | Maps changed files to affected target names through ordered, independent rules. |
26+
| `units.py` | Expands targets into `TestUnit` values: one target, one platform, one environment. |
27+
| `jobs.py` | Turns each unit into the concrete `BatchJob` the workflow runs. |
28+
| `strategy/` | Packs jobs into capacity-bounded groups. `types.py` is the contract, `default.py` the implementation. |
29+
| `validation.py` | Checks any strategy's partition against the execution contract. |
30+
| `exceptions.py` | `PlanningError` and `BatchValidationError`. |
31+
32+
The `BatchJob` type itself lives in `../messages.py`, alongside the other Dispatcher messages.
33+
34+
## Relationship to `ci_matrix.py`
35+
36+
The implementation this package shadows is `ddev/src/ddev/utils/scripts/ci_matrix.py`, and that is
37+
still the one CI uses. The two will run side by side until the Dispatcher takes over, so a
38+
behavioural change here that CI does not make is a divergence, not an improvement.
39+
40+
Some values are duplicated between them on purpose: `ci_matrix.py` must run standalone with no
41+
dependencies, so it cannot import from this package. `PLATFORMS` and the path patterns are the
42+
copies that matter. Change one and change the other.
43+
44+
Environment discovery is the one place they deliberately differ. This package asks Hatch through an
45+
injected `EnvironmentProvider`, where `ci_matrix.py` reads the `hatch.toml` matrix directly. Asking
46+
Hatch is accurate but costs one subprocess per target, and the repository-wide rule selects every testable
47+
target, so a `datadog_checks_base` change means hundreds of serial subprocesses. That needs
48+
concurrency or a `hatch.toml`-reading provider before this runs on real pull requests.
49+
50+
## Rules
51+
52+
**Planning is deterministic and offline.** The same changed files must always produce the same
53+
plan, byte for byte. Never introduce ordering that depends on a set, a dict built from an unordered
54+
source, wall-clock time, or randomness, and never make a network call while planning. Registry
55+
lookups belong in an explicit preflight such as `find_unpublished_images`, not in the plan itself.
56+
57+
**External systems come in through injected protocols.** There are five of them: `RepositoryFacts`,
58+
`TargetRule`, `EnvironmentProvider`, `BatchStrategy` and `AgentImageResolver`. They exist so that
59+
tests never need a real repository or Hatch, and so the pieces can be recomposed later. A planning
60+
function depends on the protocol and never constructs the adapter itself. A concrete adapter may
61+
live beside its protocol, the way `RegistryRepositoryFacts` does, as long as it imports its
62+
dependency lazily or behind a type-checking guard.
63+
64+
**Validation is independent of the strategy.** A strategy is untrusted input: `validate_batches`
65+
must catch a partition that drops, duplicates, overfills, or illegally splits, no matter which
66+
callable produced it. Do not move a check into a strategy.
67+
68+
**Parse strictly, but only what a human wrote.** Reject configuration you do not understand instead
69+
of guessing at what it probably meant: a Python version that is not `major.minor`, an unknown or
70+
repeated platform name in a CI override, or a target reaching expansion with no environments all
71+
raise `PlanningError`. The alternative is a plan that looks fine and tests the wrong thing.
72+
73+
Generated or advertised data is different. `manifest.json` lists platforms ddev has no runner for,
74+
such as AIX, so the supported OS list is filtered rather than parsed. Failing on it would turn
75+
someone else's metadata into an outage for every target in the run.
76+
77+
**Failures surface as `PlanningError`.** Anything that stops a plan being produced raises it, so a
78+
future command has one thing to catch. Errors from outside the package, such as the Agent-image
79+
exceptions, are wrapped at the boundary that calls them rather than made to subclass it, which
80+
would point the dependency the wrong way.
81+
82+
**Comments explain intent, not mechanics.** State the contract and the reasoning a caller cannot
83+
infer from the signature, and use inline field comments for per-field notes. Do not restate what the
84+
code says, do not narrate what other modules do, and do not use Sphinx roles (`:class:`, `:func:`)
85+
or double backticks.
86+
87+
## Keeping this file current
88+
89+
Update it in the same change that makes it wrong. Adding, removing, or renaming a module means
90+
updating the pipeline diagram and the module table. Changing a boundary, a protocol, or one of the
91+
rules above means updating that section.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
@AGENTS.md
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# (C) Datadog, Inc. 2026-present
2+
# All rights reserved
3+
# Licensed under a 3-clause BSD style license (see LICENSE)
Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
# (C) Datadog, Inc. 2026-present
2+
# All rights reserved
3+
# Licensed under a 3-clause BSD style license (see LICENSE)
4+
"""Public entry points that turn changed files into test units and batches."""
5+
6+
from __future__ import annotations
7+
8+
import logging
9+
from dataclasses import dataclass
10+
from typing import TYPE_CHECKING
11+
12+
from ddev.cli.ci.tests.batching.exceptions import PlanningError
13+
from ddev.cli.ci.tests.batching.jobs import expand_batch_jobs
14+
from ddev.cli.ci.tests.batching.strategy import BatchStrategy, default_strategy
15+
from ddev.cli.ci.tests.batching.targets import (
16+
RegistryRepositoryFacts,
17+
default_target_rules,
18+
find_affected_targets,
19+
)
20+
from ddev.cli.ci.tests.batching.units import (
21+
ResolvedEnvironment,
22+
TargetDefinition,
23+
TestUnit,
24+
expand_test_units,
25+
resolve_platforms,
26+
)
27+
from ddev.cli.ci.tests.batching.validation import validate_batches
28+
from ddev.cli.ci.tests.messages import TestBatch
29+
from ddev.e2e.agent_images import PYTHON_VERSION_PATTERN
30+
31+
if TYPE_CHECKING:
32+
from collections.abc import Sequence
33+
34+
from ddev.cli.ci.tests.batching.targets import TargetRule
35+
from ddev.cli.ci.tests.batching.units import EnvironmentProvider
36+
from ddev.cli.ci.tests.dispatcher_config import BatchingConfig
37+
from ddev.cli.ci.tests.messages import BatchJob
38+
from ddev.integration.core import Integration
39+
from ddev.repo.core import Repository
40+
from ddev.utils.git import ChangedFile
41+
from ddev.utils.hatch import Environment
42+
from ddev.utils.platform import Platform, PlatformName
43+
44+
logger = logging.getLogger(__name__)
45+
46+
47+
def build_test_units(
48+
repo: Repository,
49+
changed_files: Sequence[ChangedFile],
50+
*,
51+
environment_provider: EnvironmentProvider,
52+
rules: Sequence[TargetRule] | None = None,
53+
) -> list[TestUnit]:
54+
"""Turn changed files into the complete, deterministic list of test units.
55+
56+
Without explicit `rules`, the default set is used, with the repository-wide rule enabled only
57+
for the core repository.
58+
"""
59+
if rules is None:
60+
rules = default_target_rules(is_core=repo.name == "core")
61+
62+
facts = RegistryRepositoryFacts(repo.integrations)
63+
target_names = find_affected_targets(changed_files, facts, rules=rules)
64+
65+
definitions: list[TargetDefinition] = []
66+
for name in target_names:
67+
ci_override = repo.config.get(f"/overrides/ci/{name}", {}) or {}
68+
if ci_override.get("exclude", False):
69+
continue
70+
71+
integration = repo.integrations.get(name)
72+
platforms = resolve_platforms(ci_override.get("platforms", []), _supported_os(integration), target=name)
73+
environments = tuple(environment_provider(integration, platforms))
74+
if not environments:
75+
# A `hatch.toml` makes a target testable, so one that enables no test or E2E
76+
# environment contradicts itself. Deliberate opt-out is `overrides.ci.<target>.exclude`.
77+
logger.warning("%s has a hatch.toml but no testable environment", name)
78+
continue
79+
80+
definitions.append(
81+
TargetDefinition(
82+
name=name,
83+
display_name=integration.display_name,
84+
platforms=tuple(platforms),
85+
runners=ci_override.get("runners", {}),
86+
environments=environments,
87+
)
88+
)
89+
90+
return expand_test_units(definitions)
91+
92+
93+
def build_test_batches(
94+
repo: Repository,
95+
changed_files: Sequence[ChangedFile],
96+
*,
97+
environment_provider: EnvironmentProvider,
98+
config: BatchingConfig,
99+
strategy: BatchStrategy = default_strategy,
100+
rules: Sequence[TargetRule] | None = None,
101+
) -> list[TestBatch]:
102+
"""Turn changed files into the complete, ordered list of `TestBatch` messages.
103+
104+
The partition is validated independently of the strategy that produced it. Empty input yields
105+
no batches.
106+
"""
107+
units = build_test_units(
108+
repo,
109+
changed_files,
110+
environment_provider=environment_provider,
111+
rules=rules,
112+
)
113+
jobs = expand_batch_jobs(units)
114+
job_groups = strategy(jobs, config=config)
115+
validate_batches(job_groups, jobs, config=config)
116+
return create_test_batches(job_groups)
117+
118+
119+
def create_test_batches(job_groups: Sequence[Sequence[BatchJob]]) -> list[TestBatch]:
120+
"""Build ordered `TestBatch` messages, numbering from `batch-01` on every call.
121+
122+
The message `id` is set to the same value as `batch_id` for now; processors correlate on
123+
`batch_id`, so the two are free to diverge later.
124+
"""
125+
batches: list[TestBatch] = []
126+
for index, group in enumerate(job_groups, start=1):
127+
batch_id = f"batch-{index:02d}"
128+
integrations = list(dict.fromkeys(job.target for job in group))
129+
batches.append(
130+
TestBatch(
131+
id=batch_id,
132+
batch_id=batch_id,
133+
job_list=list(group),
134+
jobs_count=len(group),
135+
integrations=integrations,
136+
)
137+
)
138+
return batches
139+
140+
141+
def _supported_os(integration: Integration) -> list[str]:
142+
# TODO(manifest): platform detection reads `manifest.json` classifier tags. A planned change
143+
# will remove ddev tooling's dependency on the manifest; revisit this once that lands.
144+
supported_os: list[str] = []
145+
for classifier_tag in integration.manifest.get("/tile/classifier_tags", []) or []:
146+
key, _, value = classifier_tag.partition("::")
147+
if key == "Supported OS":
148+
supported_os.append(value)
149+
return supported_os
150+
151+
152+
@dataclass(frozen=True, eq=False)
153+
class HatchEnvironmentProvider:
154+
"""An `EnvironmentProvider` backed by ddev's Hatch integration."""
155+
156+
platform: Platform
157+
default_python_version: str
158+
159+
def __call__(self, integration: Integration, platforms: Sequence[PlatformName]) -> list[ResolvedEnvironment]:
160+
from ddev.utils.hatch import list_environments
161+
162+
return resolve_hatch_environments(
163+
list_environments(self.platform, integration),
164+
platforms,
165+
default_python_version=self.default_python_version,
166+
)
167+
168+
169+
def resolve_hatch_environments(
170+
environments: Sequence[Environment],
171+
platforms: Sequence[PlatformName],
172+
*,
173+
default_python_version: str,
174+
) -> list[ResolvedEnvironment]:
175+
"""Map ddev `Environment` values onto target platforms, keeping environments that test anything.
176+
177+
An environment constrained to specific platforms is routed only to those the target also runs
178+
on; an unconstrained one runs on every platform the target runs on.
179+
180+
The Python version comes from Hatch's own `python` value, never from the environment name,
181+
which only encodes it by convention.
182+
"""
183+
if not platforms:
184+
return []
185+
186+
by_name = {str(platform): platform for platform in platforms}
187+
resolved: list[ResolvedEnvironment] = []
188+
for environment in environments:
189+
if not (environment.test_env or environment.e2e_env):
190+
continue
191+
192+
if environment.platforms:
193+
# Raw configuration, so a platform ddev does not target drops out of the intersection
194+
# instead of failing the plan.
195+
candidate_platforms = [by_name[name] for name in environment.platforms if name in by_name]
196+
else:
197+
candidate_platforms = list(platforms)
198+
199+
python_version = environment.python or default_python_version
200+
if not PYTHON_VERSION_PATTERN.match(python_version):
201+
raise PlanningError(
202+
f'Environment {environment.name!r} reports Python {python_version!r}; '
203+
f'expected a `major.minor` version such as `3.13`'
204+
)
205+
206+
for platform in candidate_platforms:
207+
resolved.append(
208+
ResolvedEnvironment(
209+
name=environment.name,
210+
platform=platform,
211+
python_version=python_version,
212+
test_available=environment.test_env,
213+
e2e_available=environment.e2e_env,
214+
)
215+
)
216+
return resolved
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# (C) Datadog, Inc. 2026-present
2+
# All rights reserved
3+
# Licensed under a 3-clause BSD style license (see LICENSE)
4+
"""Exceptions raised while constructing and validating test batch plans."""
5+
6+
from __future__ import annotations
7+
8+
9+
class PlanningError(Exception):
10+
"""Raised when a valid batch plan cannot be produced under the configured policy."""
11+
12+
13+
class BatchValidationError(PlanningError):
14+
"""Raised when a batch partition violates the execution contract."""

0 commit comments

Comments
 (0)