feat(fakeintake): pin fakeintake image version for e2e tests (#53728)

## What & why

Today e2e tests consume the fakeintake image via a hardcoded `:latest` tag. `:latest` is published only on merge to main, which has two drawbacks:

1. **No PR feedback** — a fakeintake code change is not exercised by e2e on the PR that makes it. CI already builds an immutable `v<sha>` image and sets `FAKEINTAKE_IMAGE_OVERRIDE`, but only 2 suites honored that var; everything else used `:latest`.
2. **Uncontrolled propagation** — merging moves `:latest`, so every branch instantly picks up the new fakeintake; a breaking change hits everywhere at once with no reproducibility.

This PR **pins the fakeintake version**: fakeintake changes are exercised on the PR, and branches use a fixed tag — picking up a new fakeintake only when they rebase onto a main that bumped the pin.

## Changes

- **`test/fakeintake/version/`** — new package: `VERSION` file (source of truth, starts at `v1`) + `ImageURL()` helper that returns `FAKEINTAKE_IMAGE_OVERRIDE` when set, else `<registry>:<pinned-tag>`.
- **Consumers** — all 4 fakeintake image defaults (aws/azure/gcp scenarios + local docker component) now resolve through `version.ImageURL(...)`, so the override is honored **globally**. The two ad-hoc env reads were simplified away.
- **Publish** — `publish_fakeintake_pinned` publishes `fakeintake:<VERSION>` on main. `publish_fakeintake` (`v<sha>`) and `publish_fakeintake_latest` are unchanged (`:latest` still published for external consumers; no test references it).
- **Enforcement** — `dda inv fakeintake.check-version-bump` requires a **strictly-increasing** VERSION when `test/fakeintake/**` changes, run in the merge queue so two concurrent same-value bumps force a re-bump (no clobbered/mutable tags).
- Docs updated in both AGENTS.md files.

## Developer workflow

Change fakeintake code → bump `test/fakeintake/version/VERSION` in the same PR. Your PR's e2e runs your built image (via the override); on merge the pinned tag is published; other branches adopt it on rebase (visible as a VERSION bump in their diff).

## Verification

`dda inv fakeintake.build`, version unit tests (4/4), `dda inv linter.go` on all changed Go packages, `linter.python`, and `linter.gitlab-ci` all pass locally.

## Open items to confirm on CI

- **GCP registry coverage** — the pinned job uses `IMG_REGISTRIES: public` (same as the `:latest` job that already reaches `registry.datadoghq.com`, gcp's registry), so the pinned tag should land there too. The `dd-pkg` registry-group mapping lives outside this repo — worth confirming against a real CI run.
- **Immutable publish** — `dd-pkg publish-image` has no fail-on-overwrite flag; immutability rests on the strictly-increasing merge-queue check instead (noted in a job comment).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: kevin.fairise <kevin.fairise@datadoghq.com>

Author: KevinFairise2

.gitlab-ci.yml

@@ -475,6 +475,22 @@ variables:
     - .gitlab/deploy/container_build/fakeintake.yml
     - .gitlab/deploy/dev_container_deploy/fakeintake.yml
 
+# fakeintake changes that rebuild the published image (server binary only).
+# Client/CLI/docs changes don't. Keep in sync with _is_server_file() in tasks/fakeintake.py.
+.fakeintake_server_paths: &fakeintake_server_paths
+  paths:
+    - test/fakeintake/cmd/server/**/*
+    - test/fakeintake/server/**/*
+    - test/fakeintake/aggregator/**/*
+    - test/fakeintake/api/**/*
+    - test/fakeintake/go.mod
+    - test/fakeintake/go.sum
+    - test/fakeintake/Dockerfile
+    - test/fakeintake/version/VERSION
+    - .gitlab/build/binary_build/fakeintake.yml
+    - .gitlab/deploy/container_build/fakeintake.yml
+    - .gitlab/deploy/dev_container_deploy/fakeintake.yml
+
 # Paths that impact the final build artifacts (rpm, deb, docker images, etc.)
 # Used to conditionally skip expensive tests like SMP when only non-artifact files change
 # like documentation, CI config, e2e tests.
@@ -924,6 +940,15 @@ workflow:
   - <<: *if_disable_e2e_tests
     when: never
   - !reference [.except_mergequeue]
+  # A fakeintake server change rebuilds the image: run e2e against the built
+  # v<sha> image via the override. Must precede the run-all/main/release/rc cases
+  # (first-match-wins) so the override is never dropped. Needs publish_fakeintake.
+  - changes:
+      <<: *fakeintake_server_paths
+      compare_to: $COMPARE_TO_BRANCH
+    variables:
+      E2E_FAKEINTAKE_IMAGE_OVERRIDE: "public.ecr.aws/datadog/fakeintake:v$CI_COMMIT_SHORT_SHA"
+    when: on_success
   - <<: *if_run_all_e2e_tests
     when: on_success
   - <<: *if_main_branch
@@ -932,6 +957,12 @@ workflow:
     when: on_success
   - if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
     when: on_success
+  # Other fakeintake changes (client/CLI/docs) don't rebuild the image but do
+  # compile into the test binaries — run e2e against the pinned image (no override).
+  - changes:
+      <<: *fakeintake_paths
+      compare_to: $COMPARE_TO_BRANCH
+    when: on_success
   - changes:
       paths:
         - .gitlab/test/e2e/e2e.yml
@@ -977,14 +1008,9 @@ workflow:
   - <<: *if_windows_installer_changes
     when: on_success
 
+# fakeintake changes (and the override) are handled by .on_e2e_main_release_or_rc.
 .on_e2e_or_fakeintake_changes_or_manual:
   - !reference [.on_e2e_main_release_or_rc]
-  - changes:
-      <<: *fakeintake_paths
-      compare_to: $COMPARE_TO_BRANCH
-    variables:
-      FAKEINTAKE_IMAGE_OVERRIDE: "public.ecr.aws/datadog/fakeintake:v$CI_COMMIT_SHORT_SHA"
-    when: on_success
   - changes:
       paths:
         - test/new-e2e/test-infra-definition/*
@@ -1476,22 +1502,16 @@ workflow:
         - .go-version
       compare_to: $COMPARE_TO_BRANCH
 
+# Gates build/publish/version-check jobs on image-rebuilding (server) changes.
+# The e2e override is set in .on_e2e_main_release_or_rc, not here.
 .on_fakeintake_changes:
   - changes:
-      <<: *fakeintake_paths
+      <<: *fakeintake_server_paths
       compare_to: $COMPARE_TO_BRANCH
 
-.on_fakeintake_changes_with_image_override:
-  - changes:
-      <<: *fakeintake_paths
-      compare_to: $COMPARE_TO_BRANCH
-    variables:
-      FAKEINTAKE_IMAGE_OVERRIDE: "public.ecr.aws/datadog/fakeintake:v$CI_COMMIT_SHORT_SHA"
-    when: on_success
-
 .on_fakeintake_changes_on_main:
   - changes:
-      <<: *fakeintake_paths
+      <<: *fakeintake_server_paths
     <<: *if_main_branch
 
 .fast_tests_variables_named_branches:

.gitlab/JOBOWNERS

@@ -9,6 +9,8 @@ slack_teams_channels_check           @DataDog/agent-devx
 lint*                                @DataDog/agent-devx
 notify*                              @DataDog/agent-devx
 check_modules_replace                @DataDog/agent-devx
+fakeintake_check_version_bump        @DataDog/agent-devx
+publish_fakeintake_pinned            @DataDog/agent-devx
 
 # Deps build
 build_clang_*                        @DataDog/ebpf-platform

.gitlab/build/binary_build/fakeintake.yml

@@ -10,3 +10,18 @@ build_fakeintake:
   tags: ["arch:amd64", "specific:true"]
   script:
     - dda inv -- fakeintake.build
+
+# Runs in the merge queue (no .except_mergequeue) on purpose: the queue rebases each PR
+# onto up-to-date main before running this, so two PRs that bumped VERSION to the same
+# value will have the second one fail here and be forced to re-bump. See
+# tasks/fakeintake.py:check_version_bump and test/fakeintake/AGENTS.md.
+fakeintake_check_version_bump:
+  stage: binary_build
+  timeout: 15m
+  rules:
+    - !reference [.on_fakeintake_changes]
+  needs: []
+  image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
+  tags: ["arch:amd64", "specific:true"]
+  script:
+    - dda inv -- fakeintake.check-version-bump

.gitlab/deploy/dev_container_deploy/fakeintake.yml

@@ -28,3 +28,29 @@ publish_fakeintake_latest:
     IMG_DESTINATIONS: fakeintake:latest
     IMG_REGISTRIES: public
     IMG_SIGNING: "false"
+
+# Publishes the pinned tag (fakeintake:<VERSION>) from main only, when the VERSION
+# file changes. Gating on VERSION means a bump always targets a brand-new tag;
+# strict monotonicity (fakeintake_check_version_bump) prevents clobbering a released
+# tag, since dd-pkg publish-image has no fail-on-overwrite flag.
+publish_fakeintake_pinned:
+  extends: .docker_publish_job_definition
+  stage: dev_container_deploy
+  rules:
+    - !reference [.except_mergequeue]
+    # if_main_branch inlined: YAML anchors are file-local (only !reference is cross-file).
+    # No compare_to: it defaults to main, so on main it would be a main-vs-main diff that
+    # never matches; the default diff vs the previous commit detects the merge's VERSION bump.
+    - if: $CI_COMMIT_BRANCH == "main"
+      changes:
+        paths:
+          - test/fakeintake/version/VERSION
+  needs:
+    - job: docker_build_fakeintake
+      optional: false
+  before_script:
+    - export IMG_DESTINATIONS="fakeintake:$(cat test/fakeintake/version/VERSION)"
+  variables:
+    IMG_SOURCES: registry.ddbuild.io/ci/datadog-agent/fakeintake:v${CI_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}
+    IMG_REGISTRIES: public
+    IMG_SIGNING: "false"

.gitlab/test/e2e/e2e.yml

@@ -138,13 +138,23 @@
       annotations:
         - $EXTERNAL_LINKS_PATH
 
+# Wait for the fakeintake image to publish (v<sha> on PRs, pinned tag on main).
+# Both optional: absent on non-fakeintake pipelines. Reference from every e2e
+# needs block (directly or via .needs_new_e2e_template) so no job races the image.
+.needs_fakeintake_publish:
+  - job: publish_fakeintake
+    optional: true
+  - job: publish_fakeintake_pinned
+    optional: true
+
 .needs_new_e2e_template:
   - go_e2e_deps
   - job: go_e2e_test_binaries
     artifacts: false
   - go_tools_deps
   - job: new-e2e-base-coverage
     optional: true
+  - !reference [.needs_fakeintake_publish]
 
 new-e2e-base-coverage:
   extends: .new_e2e_template
@@ -289,6 +299,7 @@ new-e2e-containers-eks-init:
   needs:
     - go_e2e_deps
     - go_tools_deps
+    - !reference [.needs_fakeintake_publish]
   rules:
     - !reference [.on_container_or_e2e_changes]
     - !reference [.manual]
@@ -326,6 +337,7 @@ new-e2e-containers-openshift-init:
   needs:
     - go_e2e_deps
     - go_tools_deps
+    - !reference [.needs_fakeintake_publish]
   rules:
     - !reference [.on_e2e_main_release_or_rc]
     - changes:
@@ -445,14 +457,11 @@ new-e2e-privateactionrunner:
   needs:
     - !reference [.needs_new_e2e_template]
     - agent_deb-x64-a7
-    - job: publish_fakeintake
-      optional: true
     - qa_agent_linux
     - qa_dca
   rules:
     - !reference [.except_disable_e2e_tests]
     - !reference [.except_mergequeue]
-    - !reference [.on_fakeintake_changes_with_image_override]
     - !reference [.on_privateactionrunner_or_e2e_changes]
     - !reference [.manual]
   variables:
@@ -525,6 +534,7 @@ new-e2e-npm-eks-init:
   needs:
     - go_e2e_deps
     - go_tools_deps
+    - !reference [.needs_fakeintake_publish]
   rules:
     - !reference [.on_npm_or_e2e_changes]
     - !reference [.manual]
@@ -1023,6 +1033,7 @@ new-e2e-otel-eks-init:
   needs:
     - go_e2e_deps
     - go_tools_deps
+    - !reference [.needs_fakeintake_publish]
   variables:
     TARGETS: ./tests/otel
     TEAM: otel
@@ -1089,6 +1100,7 @@ new-e2e-otel:
   needs:
     - go_e2e_deps
     - go_tools_deps
+    - !reference [.needs_fakeintake_publish]
   rules:
     - !reference [.on_e2e_main_release_or_rc]
     - changes:

.gitlab/test/e2e_pre_test/e2e_pre_test.yml

@@ -10,8 +10,12 @@ e2e_pre_test:
     - job: go_e2e_test_binaries
       artifacts: false
     - go_tools_deps
+    # PR pipelines run against v<sha> (publish_fakeintake); main uses the pinned
+    # tag (publish_fakeintake_pinned). Both optional: absent otherwise.
     - job: publish_fakeintake
       optional: true
+    - job: publish_fakeintake_pinned
+      optional: true
   script:
     - dda inv -- -e new-e2e-tests.run --targets ./test-infra-definition --result-json $E2E_RESULT_JSON --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
   after_script:

tasks/fakeintake.py

@@ -2,10 +2,36 @@
 Build or use the fake intake client CLI
 """
 
+import os
+
 from invoke import task
+from invoke.exceptions import Exit
 
+from tasks.libs.common.color import color_message
+from tasks.libs.common.git import get_ancestor_base_branch, get_changed_files, get_common_ancestor
 from tasks.libs.common.go import go_build
 
+VERSION_FILE = "test/fakeintake/version/VERSION"
+
+# Paths that rebuild the published image (client/CLI/docs don't).
+# Keep in sync with .fakeintake_server_paths in .gitlab-ci.yml.
+SERVER_PATH_PREFIXES = (
+    "test/fakeintake/cmd/server/",
+    "test/fakeintake/server/",
+    "test/fakeintake/aggregator/",
+    "test/fakeintake/api/",
+)
+SERVER_FILES = (
+    "test/fakeintake/go.mod",
+    "test/fakeintake/go.sum",
+    "test/fakeintake/Dockerfile",
+)
+
+
+def _is_server_file(path: str) -> bool:
+    """True if changing `path` rebuilds the fakeintake image (needs a VERSION bump)."""
+    return path in SERVER_FILES or path.startswith(SERVER_PATH_PREFIXES)
+
 
 @task
 def build(ctx):
@@ -24,3 +50,69 @@ def test(ctx):
     """
     with ctx.cd("test/fakeintake"):
         ctx.run("go test ./...")
+
+
+def _parse_version(raw: str) -> int:
+    version = raw.strip()
+    if not version.startswith("v") or not version[1:].isdigit():
+        raise Exit(
+            code=1,
+            message=color_message(
+                f"Invalid {VERSION_FILE} content {raw!r}: expected a 'v<int>' tag (e.g. 'v1')", "red"
+            ),
+        )
+    return int(version[1:])
+
+
+@task
+def check_version_bump(ctx):
+    """
+    Ensure test/fakeintake/version/VERSION is bumped whenever the fakeintake image changes.
+
+    The pinned tag in VERSION is what e2e-framework's fakeintake defaults resolve to
+    (see test/fakeintake/version). Only server-side changes rebuild the published image
+    (see _is_server_file); such a merge must ship a strictly greater VERSION than its base
+    branch so the newly published image gets a unique, immutable tag (see
+    test/fakeintake/AGENTS.md). Client/CLI/docs changes don't touch the image, so they
+    don't require a bump.
+    """
+    base_branch = os.environ.get("COMPARE_TO_BRANCH") or get_ancestor_base_branch()
+
+    # Resolve the merge-base as a concrete commit. get_common_ancestor fetches the
+    # base ref when it is missing (CI does shallow clones with S3 caching), which a
+    # raw `git diff <base>...HEAD` cannot do — that fails with "unknown revision".
+    merge_base = get_common_ancestor(ctx, "HEAD", base_branch)
+
+    changed_files = [f.strip() for f in get_changed_files(ctx, base=merge_base) if f.strip()]
+    server_changes = [f for f in changed_files if _is_server_file(f)]
+
+    if not server_changes:
+        print(color_message("No fakeintake image (server) changes detected, VERSION bump not required", "green"))
+        return
+
+    with open(VERSION_FILE) as f:
+        new_version_raw = f.read()
+    new_version = _parse_version(new_version_raw)
+
+    # VERSION may not exist at the merge-base (bootstrap PR / baseline reset), so
+    # warn=True and treat a missing base file as version 0 instead of crashing.
+    base_version_result = ctx.run(f"git show {merge_base}:{VERSION_FILE}", hide=True, warn=True)
+    base_version = _parse_version(base_version_result.stdout) if base_version_result.ok else 0
+
+    if new_version <= base_version:
+        raise Exit(
+            code=1,
+            message=color_message(
+                f"fakeintake image changed ({len(server_changes)} server file(s), e.g. {server_changes[0]}) but "
+                f"{VERSION_FILE} was not bumped: it is 'v{new_version}', which must be strictly greater than "
+                f"{base_branch}'s 'v{base_version}'. Bump {VERSION_FILE} to at least 'v{base_version + 1}' in this PR.",
+                "red",
+            ),
+        )
+
+    print(
+        color_message(
+            f"{VERSION_FILE} bumped from 'v{base_version}' to 'v{new_version}', OK",
+            "green",
+        )
+    )

test/e2e-framework/AGENTS.md

@@ -182,6 +182,28 @@ dda inv new-e2e-tests.run --targets=./tests/<area>/...
 Use `e2e.WithDevMode()` to keep infrastructure alive after a failure so you can
 SSH in and inspect the agent directly.
 
+## Fakeintake image version
+
+Every fakeintake default (`scenarios/{aws,azure,gcp}/fakeintake/params.go`,
+`components/datadog/fakeintake/docker.go`) resolves through
+`components/datadog/fakeintake.ImageURL(...)`: it uses the
+`FakeintakeImageOverride` runner parameter (`E2E_FAKEINTAKE_IMAGE_OVERRIDE`) when
+set — read through the runner parameter store like any other `E2E_*` value, not
+`os.Getenv` — otherwise the pinned tag from `test/fakeintake/version.Tag`.
+`WithImageURL(...)` on any fakeintake provisioner still wins over both.
+
+CI wiring (`.gitlab-ci.yml`): the `.on_e2e_main_release_or_rc` rule — inherited
+by every e2e job through its team rule (`.on_<team>_or_e2e_changes`) — sets
+`E2E_FAKEINTAKE_IMAGE_OVERRIDE` to the PR-built `v<sha>` image on a fakeintake
+*server* change (`.fakeintake_server_paths`). So such a PR runs the **whole**
+e2e suite against the PR's image (including mixed PRs), and no e2e job can miss
+the override. `.needs_new_e2e_template` gains optional needs on `publish_fakeintake`
+(PR `v<sha>`) and `publish_fakeintake_pinned` (main pinned tag) so e2e waits for
+the image to exist. Plain `.on_fakeintake_changes` is for non-consuming
+build/publish/version-check jobs only. See `test/fakeintake/AGENTS.md`
+§ "Image version pinning" for the full workflow (bumping VERSION, the
+strictly-increasing CI check, publish jobs).
+
 ## Key files
 
 - `testing/e2e/suite.go` — `BaseSuite` and `Run()` (test entry point)

test/e2e-framework/components/datadog/fakeintake/BUILD.bazel

@@ -5,13 +5,17 @@ go_library(
     srcs = [
         "component.go",
         "docker.go",
+        "imageurl.go",
     ],
     importpath = "github.com/DataDog/datadog-agent/test/e2e-framework/components/datadog/fakeintake",
     visibility = ["//visibility:public"],
     deps = [
         "//test/e2e-framework/common/config",
         "//test/e2e-framework/components",
+        "//test/e2e-framework/testing/runner",
+        "//test/e2e-framework/testing/runner/parameters",
         "//test/fakeintake/server/rcstore",
+        "//test/fakeintake/version",
         "@com_github_pulumi_pulumi_docker_sdk_v4//go/docker",
         "@com_github_pulumi_pulumi_sdk_v3//go/pulumi",
     ],