Merge branch 'main' into andrew.glaude/PrimTagsDDOT

Author: ichinaski

.agents/skills/update-otel-deps/SKILL.md

@@ -79,12 +79,26 @@ Use this only when: the automated workflow has a persistent infrastructure failu
 dda inv collector.update   # bumps go.mod + OCB YAML files to latest OTel version
 dda inv collector.generate # regenerates OTel Agent code from the new manifests
 dda inv components.lint-components --fix
-dda inv modules.add-all-replace
 dda inv tidy               # reconcile transitive dependencies
+dda inv modules.add-all-replace  # must run AFTER tidy (see note below)
 bazel run //:go_mod_tidy_all
 dda inv generate-licenses  # update license inventory
 ```
 
+> **Run `modules.add-all-replace` after `tidy`, not before.** `collector.generate` rewrites
+> `comp/otelcol/collector-contrib/impl/go.mod` with short-form replace paths
+> (`=> ../../../logs-library`) where the repo standard is the canonical form
+> (`=> ../../../../comp/logs-library`). Running `add-all-replace` before `tidy` leaves the short
+> form in place and `check_modules_replace` fails in CI. Verify it converged by re-running it and
+> confirming no `go.mod` changes.
+
+> **`read_old_version` trusts the OCB manifest, which can be stale.** `collector.update`
+> search-replaces the version recorded in `comp/otelcol/collector-contrib/impl/manifest.yaml`.
+> If someone bumped the Go deps directly without running `collector.update` (as #53984 did), the
+> manifest lags and every file referencing the *actual* previous version is silently skipped.
+> Always compare the manifest version against a real dependency before trusting the bump:
+> `grep 'otelcol v' go.mod`.
+
 If a **specific version** was requested, skip `inv collector.update` and do a repo-wide search-and-replace of the old version string (find it in `tasks/collector.py` — the `OCB_VERSION` / `OTEL_CONTRIB_VERSION` constants). Then run the remaining commands above.
 
 After the commands finish, scan for any files the task missed:
@@ -106,9 +120,11 @@ This is the **most common failure**. The DD flare extension tests compare OTel r
 Run the test locally to get the actual diff:
 
 ```bash
-dda inv test --targets=./comp/otelcol/ddflareextension/impl/... --build-include=otlp -- -run TestGetConfDump
+dda inv test --targets=./comp/otelcol/ddflareextension/impl/... --build-include=test,otlp -- -run TestGetConfDump
 ```
 
+> **The `test` tag is required.** `--build-include` *replaces* the default build-tag set rather than adding to it. With only `otlp`, the files gated behind `//go:build test` (`configstore_test.go`, `extension_test.go`, `factory_test.go`) never compile, so `-run TestGetConfDump` matches nothing — the run still reports "ALL TESTS PASSED" from the handful of untagged tests. Confirm the test really ran by checking `test_output.json` for `"Test":"TestGetConfDump"`.
+
 The failure output shows exactly which lines differ. Apply the diffs to the golden files in:
 - `comp/otelcol/ddflareextension/impl/testdata/` (unit test golden files)
 - `test/new-e2e/tests/otel/otel-agent/testdata/` (E2E test golden files — apply the **same changes**)
@@ -169,9 +185,40 @@ Then run `dda inv tidy` again. Use this only when upgrading the Agent code is no
 
 ```bash
 dda inv linter.go --targets=./comp/otelcol/...                                        # lint the changed OTel components
-dda inv test --targets=./comp/otelcol/ddflareextension/impl/... --build-include=otlp  # confirm tests pass
+dda inv test --targets=./comp/otelcol/ddflareextension/impl/... --build-include=test,otlp  # confirm tests pass
+```
+
+> **`--targets=./comp/otelcol/...` only covers the root module.** Several OTel packages are
+> *separate Go modules* and are invisible to that invocation — it will report "0 issues" while
+> they are silently unlinted. An upstream API change typically breaks exactly these. Lint each one
+> in its own module context:
+>
+> ```bash
+> for m in comp/otelcol/otlp/components/datadogconfig \
+>          comp/otelcol/otlp/components/exporter/serializerexporter \
+>          comp/otelcol/otlp/components/processor/infraattributesprocessor; do
+>   dda inv linter.go --module="$m"
+> done
+> ```
+>
+> Find the full list with `find comp/otelcol -name go.mod`. Note that `comp/host-profiler` also
+> consumes OTel APIs and is gated behind `//go:build linux`, so it cannot be linted from macOS
+> without a `x86_64-linux-gnu-gcc` cross toolchain — rely on CI's `lint_linux-x64` for it.
+
+### Upstream API moves
+
+When a symbol goes `undefined` after the bump, it has usually been *promoted* from an experimental
+`x`-prefixed package into its stable counterpart rather than deleted. Confirm before rewriting call
+sites, then rename:
+
+```bash
+rg -n "^func Validate|^type Validator" "$(go env GOMODCACHE)"/go.opentelemetry.io/collector/confmap@v1.64.0/*.go
 ```
 
+Seen in v0.158.0: `xconfmap.Validate` / `xconfmap.Validator` → `confmap.Validate` / `confmap.Validator`
+(identical signatures). This breaks `lint_*`, `bazel:test:*`, and `build_host_profiler_binary_*`
+together, so a large fan-out of failures often has a single upstream cause.
+
 Open a draft PR to let CI catch any remaining failures. The PR title convention is:
 `Update OTel Collector dependencies to v<VERSION>`
 
@@ -183,8 +230,12 @@ Once the PR is open, run `/dd:ci:fix` to automatically fetch, analyze, and fix a
 
 - [ ] Automated workflow checked — succeeded (auto-PR reviewed) or failure diagnosed and fixed
 - [ ] `dda inv collector.update` + `collector.generate` + `tidy` + `generate-licenses` all completed without error (automated or manual)
-- [ ] No remaining references to the old OTel version in tracked files
-- [ ] ddflareextension unit + E2E golden files updated and tests pass
+- [ ] `modules.add-all-replace` run **after** `tidy` and confirmed idempotent (re-run produces no `go.mod` diff)
+- [ ] No remaining references to the old OTel version in tracked files — check the *actual* dep version, not just the manifest's
+- [ ] ddflareextension unit + E2E golden files updated and tests pass (with the `test` build tag)
+- [ ] Separate OTel Go modules linted individually with `--module=` (not just `--targets=./comp/otelcol/...`)
+- [ ] `test/fakeintake/version/VERSION` bumped if `tidy` touched `test/fakeintake/go.mod`
+- [ ] Release note added under `releasenotes/notes/` (or `changelog/no-changelog` applied)
 - [ ] OCB build script succeeds (or known issue tracked)
 - [ ] Static quality gate limits raised if breached (exemption approved)
 - [ ] Draft PR open and CI green (use `/dd:ci:fix` for any remaining failures)

.bazelrc

@@ -98,14 +98,19 @@ common:cache:frontend --remote_cache=grpcs://buildbarn-frontend-datadog-agent.us
 common:cache --config=cache:frontend
 common:cache --noremote_upload_local_results
 
+# Explicit local-only config: disable the remote cache while keeping the
+# wrapper-injected --disk_cache. Use `--config=no-remote-cache` (or
+# DD_BAZEL_REMOTE_CACHE=off) to opt out of tools/bazel auto-selection.
+common:no-remote-cache --remote_cache=
+
 # CI config ------------------------------------------------------------------------------------------------------------
 common:ci --config=adms
 common:ci --config=lint
 common:ci --noexperimental_convenience_symlinks # not CI-suitable: "These symlinks are only for the user's convenience"
 common:ci --remote_download_regex=.*/test\.xml$ # force-download test.xml even under --remote_download_outputs=toplevel, so junit collection can read it locally
 
 # Project/Language configs --------------------------------------------------------------------------------------
-import %workspace%/bazel/configs/flavors.bazelrc
+import %workspace%/bazel/configs/go_tests.bazelrc
 import %workspace%/bazel/configs/rust.bazelrc
 
 # Global release config ------------------------------------------------------------------------------------------------

.github/CODEOWNERS

@@ -272,6 +272,7 @@
 /cmd/agent/dist/conf.d/oracle.d/    @DataDog/database-monitoring
 /cmd/agent/dist/conf.d/oracle-dbm.d/    @DataDog/database-monitoring
 /cmd/agent/dist/conf.d/network_path.d/  @DataDog/network-path
+/cmd/agent/dist/conf.d/network_config_management.d/  @DataDog/ndm-integrations
 /cmd/agent/dist/conf.d/orchestrator_ecs.d/  @DataDog/ecs-experiences
 /cmd/agent/dist/conf.d/sbom.d/          @DataDog/agent-security @DataDog/container-integrations
 /cmd/agent/dist/conf.d/snmp.d/          @DataDog/network-device-monitoring-core
@@ -383,6 +384,7 @@
 /comp/forwarder @DataDog/agent-metric-pipelines
 /comp/healthplatform @DataDog/fleet-remediation
 /comp/host-profiler @DataDog/opentelemetry-agent @DataDog/profiling-full-host
+/comp/kubeactions @DataDog/container-integrations
 /comp/logs @DataDog/agent-log-pipelines
 /comp/logs-library @DataDog/agent-log-pipelines
 /comp/metadata @DataDog/fleet-automation
@@ -407,6 +409,7 @@
 /comp/core/configstream @DataDog/fleet-automation
 /comp/core/configstreamconsumer @DataDog/fleet-automation
 /comp/core/configsync @DataDog/fleet-automation
+/comp/core/delegatedauth @DataDog/credential-management
 /comp/core/diagnose @DataDog/fleet-remediation
 /comp/core/flare @DataDog/fleet-remediation
 /comp/core/gui @DataDog/fleet-remediation
@@ -478,6 +481,7 @@
 /comp/forwarder/eventplatform/impl/pipelines_eventmanagement.go        @DataDog/windows-products
 /comp/forwarder/eventplatform/impl/pipelines_datastreams.go            @DataDog/data-streams-monitoring
 /comp/forwarder/eventplatform/impl/pipelines_dataobservability.go      @DataDog/data-observability
+/comp/forwarder/eventplatform/impl/pipelines_datasecurity.go           @DataDog/sensitive-data-scanner
 /comp/forwarder/eventplatform/impl/pipelines_softwareinventory.go      @DataDog/windows-products
 
 /comp/core/agenttelemetry               @DataDog/fleet-remediation
@@ -520,6 +524,7 @@
 /pkg/jmxfetch/                          @DataDog/agent-metric-pipelines
 /pkg/metrics/                           @DataDog/agent-metric-pipelines
 /pkg/metrics/metricsource.go            @DataDog/agent-metric-pipelines @DataDog/agent-integrations
+/pkg/notableevents/                     @DataDog/windows-products
 /pkg/serializer/                        @DataDog/agent-metric-pipelines
 /pkg/serializer/internal/metrics/origin_mapping.go  @DataDog/agent-metric-pipelines @DataDog/agent-integrations
 /pkg/serverless/                        @DataDog/serverless-azure-gcp
@@ -658,7 +663,7 @@
 /pkg/procmgr/rust/src/platform/windows.rs          @DataDog/agent-runtimes @DataDog/windows-products
 /pkg/procmgr/rust/src/transport/named_pipe.rs       @DataDog/agent-runtimes @DataDog/windows-products
 /pkg/privateactionrunner/               @DataDog/action-platform
-/pkg/privateactionrunner/bundles/remoteaction/rshell  @DataDog/action-platform @DataDog/dd-agent-mcp
+/pkg/privateactionrunner/bundles/remoteaction/rshell  @DataDog/action-platform @DataDog/dd-agent-mcp @DataDog/fleet-remediation
 /pkg/privateactionrunner/bundles/remoteaction/networkconfigmanagement  @DataDog/action-platform @DataDog/ndm-integrations
 /pkg/privateactionrunner/bundles/remoteaction/networks  @DataDog/action-platform @Datadog/network-path
 /pkg/proto/                             @DataDog/agent-runtimes
@@ -683,6 +688,7 @@
 /pkg/tagset/                            @DataDog/agent-runtimes
 /pkg/util/                              @DataDog/agent-runtimes
 /pkg/util/aggregatingqueue              @DataDog/container-integrations @DataDog/container-platform
+/pkg/util/aws/creds/                    @DataDog/credential-management
 /pkg/util/cloudproviders/cloudfoundry/  @DataDog/agent-integrations
 /pkg/util/clusteragent/                 @DataDog/container-platform
 /pkg/util/confmaputils                       @DataDog/opentelemetry-agent @DataDog/profiling-full-host

.gitlab-ci.yml

@@ -474,12 +474,15 @@ variables:
 
 # fakeintake changes that rebuild the published image (server binary only).
 # Client/CLI/docs changes don't. Keep in sync with _is_server_file() in tasks/fakeintake.py.
+# Only `.go` files under the server paths matter: the image is built from the Go sources
+# via test/fakeintake/Dockerfile, so BUILD.bazel and test fixtures don't affect it
+# (rules:changes has no exclusion syntax, hence the extension in the glob).
 .fakeintake_server_paths: &fakeintake_server_paths
   paths:
-    - test/fakeintake/cmd/server/**/*
-    - test/fakeintake/server/**/*
-    - test/fakeintake/aggregator/**/*
-    - test/fakeintake/api/**/*
+    - test/fakeintake/cmd/server/**/*.go
+    - test/fakeintake/server/**/*.go
+    - test/fakeintake/aggregator/**/*.go
+    - test/fakeintake/api/**/*.go
     - test/fakeintake/go.mod
     - test/fakeintake/go.sum
     - test/fakeintake/Dockerfile

.gitlab/build/bazel/test.yml

@@ -33,33 +33,33 @@
   - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
 
 bazel:test:linux-amd64:
-  extends: [ .bazel:test:reporting, .bazel:runner:linux-amd64, .bazel:test ]
+  extends: [.bazel:test:reporting, .bazel:runner:linux-amd64, .bazel:test]
   script:
     - !reference [.bazel:test:reporting:ci_links]
-    - bazel test --jobs=$KUBERNETES_CPU_REQUEST --keep_going --remote_download_outputs=toplevel --build_event_json_file=$BEP_FILE --config=no-dd-agent-go-tests //...
+    - bazel test --config=gorace --jobs=$KUBERNETES_CPU_REQUEST --keep_going --remote_download_outputs=toplevel --build_event_json_file=$BEP_FILE //...
 
 bazel:test:linux-arm64:
-  extends: [ .bazel:test:reporting, .bazel:runner:linux-arm64, .bazel:test ]
+  extends: [.bazel:test:reporting, .bazel:runner:linux-arm64, .bazel:test]
   script:
     - !reference [.bazel:test:reporting:ci_links]
-    - bazel test --jobs=$KUBERNETES_CPU_REQUEST --keep_going --remote_download_outputs=toplevel --build_event_json_file=$BEP_FILE --config=no-dd-agent-go-tests //...
+    - bazel test --config=gorace --jobs=$KUBERNETES_CPU_REQUEST --keep_going --remote_download_outputs=toplevel --build_event_json_file=$BEP_FILE //...
 
 bazel:test:macos-amd64:
-  extends: [ .bazel:test:reporting, .bazel:runner:macos-amd64, .bazel:test ]
+  extends: [.bazel:test:reporting, .bazel:runner:macos-amd64, .bazel:test]
   script:
     - !reference [.install_python_dependencies]
     - !reference [.bazel:test:reporting:ci_links]
-    - bazel test --keep_going --remote_download_outputs=toplevel --build_event_json_file=$BEP_FILE --config=no-dd-agent-go-tests //...
+    - bazel test --config=gorace --keep_going --remote_download_outputs=toplevel --build_event_json_file=$BEP_FILE //...
   after_script:
     - !reference [.install_python_dependencies]
     - !reference [.bazel:test:reporting, after_script]
 
 bazel:test:macos-arm64:
-  extends: [ .bazel:test:reporting, .bazel:runner:macos-arm64, .bazel:test ]
+  extends: [.bazel:test:reporting, .bazel:runner:macos-arm64, .bazel:test]
   script:
     - !reference [.install_python_dependencies]
     - !reference [.bazel:test:reporting:ci_links]
-    - bazel test --keep_going --remote_download_outputs=toplevel --build_event_json_file=$BEP_FILE --execution_log_compact_file=$CI_PROJECT_DIR/bazel-exec.log --config=no-dd-agent-go-tests //...
+    - bazel test --config=gorace --keep_going --remote_download_outputs=toplevel --build_event_json_file=$BEP_FILE --execution_log_compact_file=$CI_PROJECT_DIR/bazel-exec.log //...
   after_script:
     - !reference [.install_python_dependencies]
     - !reference [.bazel:test:reporting, after_script]
@@ -69,112 +69,22 @@ bazel:test:macos-arm64:
       - bazel-bep-*.json
       - bazel-exec.log
 
+# Windows splits the build from the Go tests, unlike the other platforms: the
+# runner has 16 CPUs, and compiling the rest of the repo alongside ~1k
+# race-instrumented tests starves them enough that tests waiting on wall-clock
+# time fail. This job covers everything but the Go tests.
 bazel:test:windows-amd64:
-  extends: [ .bazel:runner:windows-amd64, .bazel:test ]
+  extends: [.bazel:runner:windows-amd64, .bazel:test]
   variables:
     POWERSHELL_SCRIPT: >-
-      bazel test --keep_going --config=no-dd-agent-go-tests
-      //... --
+      bazel test --config=gorace --config=no-dd-agent-go-tests --keep_going //... --
       -//bazel/rules/dd_packaging/...
       -//packages/agent/linux/...
 
-# Per-flavor test jobs. Each runs the tests for one agent flavor, mirroring the
-# dda inv test --flavor=<f> topology. --config=<flavor> sets
-# --test_tag_filters=flavor_<flavor>, selecting only the matching go_test variants.
-.bazel:test:flavor:
-  extends: [ .bazel:test, .bazel:test:reporting ]
-  script:
-    - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || echo 'Failed to fetch an API key, no metrics will be emitted'; export DD_API_KEY
-    - !reference [.bazel:test:reporting:ci_links]
-    - bazel test --keep_going --remote_download_outputs=toplevel --test_arg=-test.v --build_event_json_file=$BEP_FILE --build_tests_only --config=$FLAVOR //pkg/... //comp/... //cmd/...
-    - dda inv -- -e bazel.ensure-test-parity --bep $BEP_FILE --flavor-name $FLAVOR --emit-metrics
-
-# macOS flavor variant: re-installs dda in both script and after_script since
-# macOS shell executors run each phase in a fresh shell.
-.bazel:test:flavor:macos:
-  extends: .bazel:test:flavor
-  script:
-    - !reference [.install_python_dependencies]
-    - !reference [.bazel:test:flavor, script]
-  after_script:
-    - !reference [.install_python_dependencies]
-    - !reference [.bazel:test:reporting, after_script]
-
-bazel:test:linux-amd64:base:
-  extends: [ .bazel:runner:linux-amd64, .bazel:test:flavor ]
-  variables:
-    FLAVOR: base
-
-bazel:test:linux-amd64:dogstatsd:
-  extends: [ .bazel:runner:linux-amd64, .bazel:test:flavor ]
-  variables:
-    FLAVOR: dogstatsd
-
-bazel:test:linux-amd64:heroku:
-  extends: [ .bazel:runner:linux-amd64, .bazel:test:flavor ]
-  variables:
-    FLAVOR: heroku
-
-bazel:test:linux-amd64:iot:
-  extends: [ .bazel:runner:linux-amd64, .bazel:test:flavor ]
-  variables:
-    FLAVOR: iot
-
-bazel:test:linux-arm64:base:
-  extends: [ .bazel:runner:linux-arm64, .bazel:test:flavor ]
-  variables:
-    FLAVOR: base
-
-bazel:test:linux-arm64:dogstatsd:
-  extends: [ .bazel:runner:linux-arm64, .bazel:test:flavor ]
-  variables:
-    FLAVOR: dogstatsd
-
-bazel:test:linux-arm64:heroku:
-  extends: [ .bazel:runner:linux-arm64, .bazel:test:flavor ]
-  variables:
-    FLAVOR: heroku
-
-bazel:test:linux-arm64:iot:
-  extends: [ .bazel:runner:linux-arm64, .bazel:test:flavor ]
-  variables:
-    FLAVOR: iot
-
-bazel:test:macos-amd64:base:
-  extends: [ .bazel:runner:macos-amd64, .bazel:test:flavor:macos ]
-  variables:
-    FLAVOR: base
-
-bazel:test:macos-amd64:dogstatsd:
-  extends: [ .bazel:runner:macos-amd64, .bazel:test:flavor:macos ]
-  variables:
-    FLAVOR: dogstatsd
-
-bazel:test:macos-arm64:base:
-  extends: [ .bazel:runner:macos-arm64, .bazel:test:flavor:macos ]
-  variables:
-    FLAVOR: base
-
-bazel:test:macos-arm64:dogstatsd:
-  extends: [ .bazel:runner:macos-arm64, .bazel:test:flavor:macos ]
-  variables:
-    FLAVOR: dogstatsd
-
-# Windows can't extend .bazel:test:flavor: the Windows runner template consumes
-# POWERSHELL_SCRIPT instead of `script:`, and the test pattern needs to exclude
-# Linux-only packaging targets.
-bazel:test:windows-amd64:base:
-  extends: [ .bazel:runner:windows-amd64, .bazel:test ]
+bazel:test:windows-amd64:go:
+  extends: [.bazel:runner:windows-amd64, .bazel:test]
   variables:
     POWERSHELL_SCRIPT: >-
-      bazel test --keep_going --build_tests_only --config=base --test_arg=-test.v
-      --build_event_json_file=$CI_PROJECT_DIR/bazel-bep-base.json
-      //pkg/... //comp/... //cmd/...;
-      dda inv -- -e bazel.ensure-test-parity
-      --bep $CI_PROJECT_DIR/bazel-bep-base.json
-      --flavor-name base
-  artifacts:
-    when: always
-    paths:
-      - bazel-bep-*.json
-    expire_in: 1 week
+      bazel test --config=gorace --config=dd-agent-go-tests-only --build_tests_only --keep_going //... --
+      -//bazel/rules/dd_packaging/...
+      -//packages/agent/linux/...

.gitlab/test/e2e/e2e.yml

@@ -464,6 +464,7 @@ new-e2e-agent-telemetry:
     - agent_deb-x64-a7
     - agent_deb-x64-a7-fips
     - qa_agent_linux
+    - qa_agent_full
     - qa_dca
   rules:
     - !reference [.on_agenttelemetry_or_e2e_changes]

.gitlab/test/functional_test/files_inventory_check.yml

@@ -13,6 +13,7 @@ files_inventory_check:
     GIT_STRATEGY: clone
   needs:
     - agent_deb-x64-a7
+    - iot_agent_deb-x64
   before_script:
     - !reference [.setup_github_token_comment_pr]
   script: