[pull] master from DataDog:master - #717

#24687)

* Build deterministic Dispatcher test batching plans

- Add changed-file target discovery, concrete job expansion, configurable batching strategies, and partition validation.
- Introduce stable batch identities and use them for runner/gatherer correlation.
- Preserve standalone CI matrix behavior while adding focused batching, configuration, message, and execution tests.

* Carry Python version and Agent image through Dispatcher test plans

Each BatchJob now records the major.minor Python its runner must set up and,
when it runs E2E tests, the Agent image to run them against. The version comes
from Hatch's own `python` value rather than the environment name, so repos that
do not follow the `pyX.Y-` naming convention resolve correctly.

- Add `ddev.e2e.agent_images`, mapping a Python version and platform to an
  Agent image. Superseded Agent lines are pinned to their final release; the
  current line tracks the development build. Base tags only, since ddev already
  appends `-jmx` per environment at E2E runtime.
- Add `[dispatcher] default_python_version`, used where a Hatch environment
  declares no Python and for targets that define no environments.

Also applies review findings on the planning layer:

- Collapse `TestUnit` to a single environment and drop `split_environments`.
  Both paths produced the same job set and differed only in where the job name
  was built, so job naming now has one implementation.
- Freeze `BatchJob` and validate batch coverage by value instead of `id()`, so
  a strategy may rebuild equal jobs rather than pass the originals through.
- Derive capacity from the config inside the strategy and validator instead of
  passing it alongside the config it already lives on.
- Rename the `units.Platform` namedtuple to `PlatformSpec`, which no longer
  collides with `messages.Platform`.
- Drop `RepositoryWideRule.exempt_files`. It cancelled the whole repository-wide
  expansion whenever `agent_requirements.in` appeared in a change set, which
  never fires for dependency bumps (they do not touch the base package sources)
  and only fires when a change genuinely does warrant the full set.
- Keep git diagnostics off stdout so they cannot reach the strict diff parser.
- Move duplicated test builders into `tests.helpers.batching`.

* Give platform identity a single home in ddev.utils.platform

Adds `PlatformName` to the module that already owns platform-name vocabulary
(`get_platform_name`, `normalize_platform_name`) and uses it for every platform
value across messages, planning, and Agent image selection, replacing the
`messages.Platform` enum and the string constants in `agent_images`.

`normalize_platform_name` keeps returning a plain string: `platform.system()`
can report systems ddev does not target, and `Platform.linux` is defined as
"neither Windows nor macOS" on purpose. Raw platform strings from
`.ddev/config.toml` and `manifest.json` are parsed into `PlatformName` once, in
`resolve_platforms`, which names the offending target when a value is not
recognized. Everything downstream holds the enum, so the unsupported-platform
check no longer sits in the middle of unit expansion.

Also adds `find_unpublished_images`, built on `ddev.utils.docker_registry`, so a
mistyped or withdrawn tag can be caught before any job runs instead of failing
every E2E job. It queries each distinct image once. `get_agent_image` stays pure
and offline so plans remain deterministic and reproducible; the registry check is
explicitly called rather than implied by a lookup. A `requires_ci` test uses it
to assert every image in the table is served by registry.datadoghq.com.

* Trim planning comments and warn on targets with no testable environment

- Cut module and class docstrings to what is not already in the code, drop Sphinx roles and double backticks
- Add batching/AGENTS.md describing the package for future readers
- Drop a redundant str() around the platform in artifact_name
- Log a warning when a target resolves no testable environment at all

* Use ddev's git module for change detection instead of a batching-local one

- Delete batching/git.py; ChangedFile and the diff parser now come from ddev.utils.git
- Move comparison-base selection to cli/ci/tests/changes.py, outside the batching package
- changes.get_changed_files takes a GitRepository rather than a bespoke GitProvider

* Fold message construction into the build module

assembly.py held one function with one call site and no test file of its own; building the
TestBatch messages is the last step of build_test_batches. The shared jobs() test builder moves
to tests.helpers.batching so both test modules can use it.

* Rebalance the planning test suite toward its lower layers

- Drop build-level tests that re-assert edge cases owned by targets, units, jobs or strategy
- Split validate_batches tests out of test_strategy.py into test_validation.py
- Share make_job, FakeIntegration, FakeRegistry, drain_queue and the changed-file builders
  through tests.helpers.batching instead of four near-identical copies
- Collapse repeated single-case tests into parametrized tables
- Cover three untested paths: an oversized integration spilling from an open batch, platforms
  read from manifest classifier tags, and the repository-wide rule gated on the repo name

* Select rename sources through the shared affected_paths

* Reconcile the gatherer tests with the batch progress aggregate

Master's progress aggregate replaced the expected-batches counter and kept Platform in messages. The correlation tests now register the batch they report, the local job and queue builders defer to the shared helpers, and the new progress tests use PlatformName.

* Fix repository-wide target selection and validate planned Python versions

- eligible_targets asked the registry with an empty selection, which ddev resolves to changed, so a shared-framework change planned only the already changed targets
- the repository-wide rule matched only a change's destination, missing a shared file renamed away
- a unit-only environment's Python version reached the plan unchecked, since only E2E resolves an Agent image
- FakeRegistry now honours the selection, so the first bug fails a test
- the batching test helpers move to the subtree that uses them

* Run unconstrained environments on every platform a target declares

An environment that names no platform was routed only to the first one the target listed, and the remaining platforms fell through to a synthesised environment claiming the default Python and no E2E. That silently dropped Windows E2E for disk, dns_check, ibm_mq, network and tcp_check, and made the order of the configured platform list load-bearing.

Unconstrained environments now run on every platform the target declares, so each job carries a real environment with its own Python version and facets. A target that resolves no environment at all is dropped with a warning when definitions are built rather than planned with an invented one, and a platform no environment covers is warned about and planned empty.

* Stop the planner failing on platforms it has no runner for

Review follow-ups on the planning package:

- `resolve_platforms` parsed every `Supported OS` classifier tag before collapsing to
  Windows-exclusivity, so the eight manifests advertising AIX raised and aborted the whole
  run. `ci_matrix.py` collapses first and never validates those values, so this was a
  regression against existing data. The manifest list now only decides Windows-exclusivity
  and strict parsing is kept for the hand-written CI override, which also now rejects a
  repeated platform instead of planning two identically named jobs.
- Fix a test importing `_normalize_agent_image_name`, renamed in master by #24639. The
  restack did not conflict, and the resulting collection error aborted the whole suite.
- Planning failures all raise `PlanningError`. Agent-image errors are wrapped at the
  boundary in `jobs.py`, naming the job, rather than made to subclass it.
- Drop the `default_python_version` argument nothing read; `EnvironmentProvider` owns it.
- Record the `ci_matrix.py` relationship and the per-target hatch subprocess cost in the
  package `AGENTS.md`, and correct its docstring and protocol rules.
- Drop a test of pydantic's own frozen behaviour, and fold the dependency-bump case into
  the parametrized set it duplicated.

* Record why the unit and E2E flags are intent rather than a decision

* Mark the unit and E2E facet flags for per-integration configuration