refactor(ebpf): add Bazel convenience targets, remove Python verification glue (#48667)

### What does this PR do?

Migrates runtime compilation bundles from Ninja to Bazel, adds central convenience targets, and removes Python verification glue code.

**Runtime compilation bundle migration (Ninja → Bazel):**

- Introduces `runtime_compilation_bundle` macro in `bazel/rules/ebpf/runtime_compilation.bzl` that chains `include_headers` → `integrity` to produce flattened `.c` files and Go integrity hash `.go` files.
- Removes `ninja_runtime_compilation_files()` and its Ninja rules (`headerincl`, `integrity`, `rctool`) from `tasks/system_probe.py`. The Linux branch of `ninja_generate()` is now a no-op — runtime compilation is fully handled by Bazel.
- Expands `bazel_build_ebpf()` to build `_BAZEL_RUNTIME_FLAT_TARGETS` and `_BAZEL_RUNTIME_GEN_TARGETS`, then copies flattened `.c` files to `pkg/ebpf/bytecode/build/runtime/` and `.go` hash files to `pkg/ebpf/bytecode/runtime/`.
- Updates `save_build_outputs()` to include both flattened `.c` files and `.go` hash files in the CI tarball artifact, so omnibus `go build` and packaging can find them.
- Keeps `go:generate` directives in the 9 source files as a fallback for `go build` users.

**Convenience targets:**

- **`//pkg/ebpf:all_ebpf_programs`** — a `filegroup` that collects every eBPF `.o` (prebuilt, CO-RE, inplace) and runtime flattened `.c` file, enabling a single `bazel build` invocation for all eBPF artifacts.
- **`//pkg/ebpf:verify_generated_files`** — a `test_suite` aggregating all `write_source_file` diff tests (Linux/Windows cgo godefs), runnable with a single `bazel test` command.

**Python glue removal:**

- Removed `_BAZEL_CGO_GODEFS_TARGETS`, `_BAZEL_CGO_GODEFS_WIN_TARGETS`, and `_bazel_verify_cgo_godefs()` / `_godefs_test_targets()` helper functions from `tasks/system_probe.py` (~60 lines).
- Replaced inline verification in `build_object_files()` with a single `bazel test //pkg/ebpf:verify_generated_files` call that works on both Linux and Windows.
- Added the same verification call in `build_cws_object_files()` and `kmt.build_object_files()`.

**Macro visibility cleanup:**

- All three eBPF macros (`ebpf_prog`, `ebpf_program_suite`, `cgo_godefs`, `runtime_compilation_bundle`) now pass `visibility` through from callers instead of hardcoding `["//visibility:public"]`. All ~60 callsites explicitly set `visibility = ["//visibility:public"]`.
- Set `--skip_incompatible_explicit_targets` globally in `.bazelrc` so `target_compatible_with` silently skips platform-incompatible targets.

### Motivation

The Python orchestration in `tasks/system_probe.py` maintained ~100 lines of target lists and helper functions for both cgo godefs verification and runtime compilation. This was fragile (target lists had to stay in sync with BUILD files) and was the last remaining Ninja workload on Linux. Moving verification into a Bazel `test_suite` and runtime compilation into a Bazel macro:

- Eliminates the Ninja dependency for Linux runtime compilation builds
- Eliminates a class of sync bugs between Python and Bazel
- Reduces Python glue code as part of the broader effort to minimize invoke task complexity
- Gives developers simple one-liner commands for common workflows

### Describe how you validated your changes

- Verified `bazel query` resolves both new targets and all their transitive dependencies
- Confirmed `--skip_incompatible_explicit_targets` correctly skips Linux-only targets on macOS without errors
- CI: buildifier, eBPF object builds, cgo godefs verification, omnibus packaging, and KMT functional tests all pass
- KMT `TestGRPCScenarios/runtime_compiled` exercises runtime compilation end-to-end (flattened `.c` → clang compile → eBPF load) — confirms Bazel-produced bundles work correctly

### Additional Notes

The `_BAZEL_EBPF_INPLACE_TARGETS`, `_BAZEL_EBPF_CORE_TARGETS`, and `_BAZEL_EBPF_PREBUILT_TARGETS` Python lists in `tasks/system_probe.py` remain — they drive the `_copy_output` logic that maps Bazel outputs to specific staging directories with `.stripped` variant handling, which a simple `filegroup` cannot express. Removing those requires a different approach (e.g., `pkg_install` rules or a Bazel run target that does the copying).

Runtime compilation hash `.go` files remain `.gitignore`d — they are generated by Bazel during the build and copied to the source tree, but not committed. To update them locally: `bazel run //pkg/ebpf/bytecode:<name>_verify`.

Co-authored-by: joseph.gette <joseph.gette@datadoghq.com>

Author: JSGette

.bazelrc

@@ -20,6 +20,7 @@ common --experimental_disk_cache_gc_max_size=5G # Cap applied whenever --disk_ca
 common --experimental_ui_max_stdouterr_bytes=1073741819 # why?
 common --http_timeout_scaling=3.0 # At least one attempt reaches 30s (3,6,12,24,30,30,30,30) instead of only 10s (1,2,4,8,10,10,10,10)
 common --incompatible_strict_action_env # Do not leak local environment variables into the build context
+common --skip_incompatible_explicit_targets # Let target_compatible_with skip rather than fail
 common --test_output=errors # Print test errors to console output instead of only capturing them in buried test.log
 common --verbose_failures
 

AGENTS.md

@@ -108,6 +108,39 @@ The development configuration file should be placed at `dev/dist/datadog.yaml`.
 - See `pkg/collector/corechecks/ebpf/AGENTS.md` for detailed structure
 - Quick reference: `.cursor/rules/system_probe_modules.mdc` for common patterns and pitfalls
 
+### eBPF Bazel Build
+
+eBPF programs, runtime compilation bundles, and cgo godefs type definitions
+are built with Bazel. Two convenience targets in `pkg/ebpf/BUILD.bazel`
+cover the most common workflows:
+
+```bash
+# Build every eBPF .o program and runtime flattened .c file at once
+bazel build //pkg/ebpf:all_ebpf_programs
+
+# Verify all committed cgo godefs files are up to date.
+# Covers both Linux and Windows targets; incompatible tests are
+# skipped automatically via target_compatible_with.
+bazel test //pkg/ebpf:verify_generated_files
+```
+
+When a `verify_generated_files` test fails, run the corresponding
+`write_source_file` target to update the committed file:
+
+```bash
+# Update a single cgo godefs output
+bazel run //pkg/ebpf:types_godefs
+```
+
+Runtime compilation integrity hash files (`pkg/ebpf/bytecode/runtime/*.go`) are
+`.gitignored` and generated during the build by `bazel_build_ebpf()`.  To update
+one locally: `bazel run //pkg/ebpf/bytecode:<name>_verify`.
+
+Key Bazel macros:
+- `ebpf_prog` / `ebpf_program_suite` (`bazel/rules/ebpf/ebpf.bzl`) — compile `.c` → `.o`
+- `cgo_godefs` (`bazel/rules/ebpf/cgo_godefs.bzl`) — `go tool cgo -godefs` + `write_source_file` verification
+- `runtime_compilation_bundle` (`bazel/rules/ebpf/runtime_compilation.bzl`) — flatten headers + generate integrity hash `.go` file
+
 ## Testing Strategy
 
 ### Unit Tests

bazel/rules/ebpf/cgo_godefs.bzl

@@ -188,6 +188,7 @@ def _cgo_godefs_macro_impl(name, visibility, src, deps, hdrs, platform):
         )
         write_source_file(
             name = name + "_test_file",
+            visibility = visibility,
             in_file = ":" + name + "_test_out",
             out_file = test_file,
             check_that_out_file_exists = False,

bazel/rules/ebpf/runtime_compilation.bzl

@@ -1,11 +1,19 @@
-"""Macro for generating runtime compilation bundles (flattened .c + integrity hash .go)."""
+"""Macro for generating runtime compilation bundles (flattened .c + integrity hash .go).
+
+Each bundle produces a flattened .c file (build artifact, not committed) and an
+integrity hash .go file.  When out_go_file is set, a write_source_file target
+is emitted so ``bazel test //<pkg>:<name>_verify_test`` checks the committed
+hash file and ``bazel run //<pkg>:<name>_verify`` updates it.
+"""
 
 load("@bazel_lib//lib:run_binary.bzl", "run_binary")
+load("@bazel_lib//lib:write_source_files.bzl", "write_source_file")
 
-def _runtime_compilation_bundle_impl(name, visibility, src_c, out_name, include_dirs, header_deps):
+def _runtime_compilation_bundle_impl(name, visibility, src_c, out_name, include_dirs, header_deps, out_go_file):
     flat_name = "{}_flat".format(name)
     run_binary(
         name = flat_name,
+        visibility = visibility,
         tool = "//pkg/ebpf:include_headers",
         srcs = [src_c] + header_deps,
         outs = ["{}/{}.c".format(name, out_name)],
@@ -33,21 +41,34 @@ def _runtime_compilation_bundle_impl(name, visibility, src_c, out_name, include_
     # redundant `import "…/runtime"` and qualifies newAsset as runtime.newAsset.
     # Inside a Bazel sandbox the paths never match, so strip the bogus import
     # and prefix to keep the output identical to the ninja/go-generate build.
+    gen_name = "{}_gen".format(name)
     native.genrule(
-        name = name,
+        name = gen_name,
         srcs = [":{}".format(raw_name)],
         outs = ["{}/{}.go".format(name, out_name)],
         cmd = "sed -e '/^import \"github.com\\/DataDog\\/datadog-agent\\/pkg\\/ebpf\\/bytecode\\/runtime\"/d' -e 's/runtime\\.newAsset/newAsset/g' $< > $@",
         visibility = visibility,
     )
 
+    if out_go_file:
+        write_source_file(
+            name = "{}_verify".format(name),
+            visibility = visibility,
+            in_file = ":{}".format(gen_name),
+            out_file = out_go_file,
+            check_that_out_file_exists = False,
+            # Out files are .gitignored; tag as manual so bazel test //... skips them.
+            tags = ["manual"],
+        )
+
 runtime_compilation_bundle = macro(
     doc = "Chains include_headers and integrity to produce a runtime compilation bundle.",
     attrs = {
         "src_c": attr.label(mandatory = True, allow_single_file = [".c"], configurable = False),
         "out_name": attr.string(mandatory = True, configurable = False),
         "include_dirs": attr.string_list(mandatory = True, configurable = False),
         "header_deps": attr.label_list(mandatory = True, configurable = False),
+        "out_go_file": attr.label(mandatory = False, allow_single_file = [".go"], configurable = False),
     },
     implementation = _runtime_compilation_bundle_impl,
 )

cmd/system-probe/subcommands/ebpf/testdata/BUILD.bazel

@@ -8,5 +8,6 @@ ebpf_prog(
         "-g",
         "-DDEBUG=1",
     ],
+    visibility = ["//visibility:public"],
     deps = ["//pkg/ebpf/c:ebpf_c_headers"],
 )

pkg/collector/corechecks/ebpf/AGENTS.md

@@ -84,25 +84,32 @@ if enabled := pkgconfigsetup.SystemProbe().GetBool("<check_name>.enabled"); enab
 
 ### 9. Register eBPF Programs in Build System
 
-eBPF programs and CGO type generation are managed by **Bazel**. Runtime
-compilation bundles are still generated by **ninja** (via `ninja_runtime_compilation_files`).
-
-**Add the eBPF CO-RE program** in the check's `BUILD.bazel` using the
-`ebpf_co_re_program` rule (see existing targets in `pkg/collector/corechecks/ebpf/c/runtime/`).
-Then add the Bazel target to `_BAZEL_EBPF_CORE_TARGETS` (or the appropriate
-list) in `tasks/system_probe.py`.
-
-**Add runtime compilation support** in `ninja_runtime_compilation_files()` in
-`tasks/system_probe.py`:
+eBPF programs, CGO type generation, and runtime compilation bundles are all
+managed by **Bazel**.
+
+**Add the eBPF CO-RE program** in the check's `c/runtime/BUILD.bazel` using
+`ebpf_program_suite` (see existing targets in
+`pkg/collector/corechecks/ebpf/c/runtime/BUILD.bazel`). Then:
+1. Add the target to `_BAZEL_EBPF_CORE_TARGETS` in `tasks/system_probe.py`
+   (needed for the copy step that stages `.o` files).
+2. Add it to the `all_ebpf_programs` filegroup in `pkg/ebpf/BUILD.bazel`.
+
+**Add runtime compilation support** by creating a `runtime_compilation_bundle`
+target in `pkg/ebpf/bytecode/BUILD.bazel`:
 ```python
-runtime_compiler_files = {
-    # ... existing entries
-    "pkg/collector/corechecks/ebpf/probe/<check>/<check>.go": "<check-name>",
-}
+runtime_compilation_bundle(
+    name = "<check-name>",
+    header_deps = _CORECHECK_HEADERS,
+    include_dirs = ["pkg/ebpf/c"],
+    out_go_file = "//pkg/ebpf/bytecode/runtime:<check-name>.go",
+    out_name = "<check-name>",
+    src_c = "//pkg/collector/corechecks/ebpf/c/runtime:<check-name>-kern.c",
+)
 ```
-- This enables runtime compilation fallback when CO-RE isn't available
-- The key is the Go file with `//go:generate` directives
-- The value is the base name for generated C and Go files
+Then add the `_flat` target to `_BAZEL_RUNTIME_FLAT_TARGETS` in
+`tasks/system_probe.py` and both the `_flat` and `_verify_test` targets to the
+convenience targets in `pkg/ebpf/BUILD.bazel` (`all_ebpf_programs` and
+`verify_generated_files` respectively).
 
 **Add CGO type generation** by creating a `cgo_godefs` target in the check's
 `BUILD.bazel`:
@@ -116,16 +123,13 @@ cgo_godefs(
     src = "<check>_kern_types.go",
 )
 ```
-Then add the `_gen` target to `_BAZEL_CGO_GODEFS_TARGETS` in `tasks/system_probe.py`:
-```python
-_BAZEL_CGO_GODEFS_TARGETS = [
-    # ... existing entries
-    "//pkg/collector/corechecks/ebpf/probe/<check>:<check>_kern_types_godefs_gen",
-]
-```
+Then add the `_test` and `_test_file_test` targets to the
+`verify_generated_files` test suite in `pkg/ebpf/BUILD.bazel`.
+
 - Generates Go types from C structs for BPF map keys/values
 - Header file must use `__u32`, `__u64` etc. types and include `ktypes.h`
-- Use `bazel test` to verify committed files match, `bazel run` to regenerate
+- `bazel test //pkg/ebpf:verify_generated_files` checks all committed files
+- `bazel run //<pkg>:<name>_godefs` regenerates a single output
 
 ## Building
 

pkg/collector/corechecks/ebpf/c/runtime/BUILD.bazel

@@ -25,6 +25,7 @@ ebpf_program_suite(
     name = "oom-kill",
     src = "oom-kill-kern.c",
     core = True,
+    visibility = ["//visibility:public"],
     deps = [
         ":corecheck_ebpf_headers",
         "//pkg/ebpf/c:ebpf_c_headers",
@@ -35,6 +36,7 @@ ebpf_program_suite(
     name = "tcp-queue-length",
     src = "tcp-queue-length-kern.c",
     core = True,
+    visibility = ["//visibility:public"],
     deps = [
         ":corecheck_ebpf_headers",
         "//pkg/ebpf/c:ebpf_c_headers",
@@ -45,6 +47,7 @@ ebpf_program_suite(
     name = "ebpf",
     src = "ebpf-kern.c",
     core = True,
+    visibility = ["//visibility:public"],
     deps = [
         ":corecheck_ebpf_headers",
         "//pkg/ebpf/c:ebpf_c_headers",
@@ -55,6 +58,7 @@ ebpf_program_suite(
     name = "noisy-neighbor",
     src = "noisy-neighbor-kern.c",
     core = True,
+    visibility = ["//visibility:public"],
     deps = [
         ":corecheck_ebpf_headers",
         "//pkg/ebpf/c:ebpf_c_headers",

pkg/collector/corechecks/ebpf/probe/ebpfcheck/BUILD.bazel

@@ -11,4 +11,5 @@ cgo_godefs(
     name = "c_types_godefs",
     src = "c_types.go",
     hdrs = ["//pkg/collector/corechecks/ebpf/c/runtime:corecheck_ebpf_headers"],
+    visibility = ["//visibility:public"],
 )

pkg/collector/corechecks/ebpf/probe/noisyneighbor/BUILD.bazel

@@ -11,4 +11,5 @@ cgo_godefs(
     name = "ebpf_types_godefs",
     src = "ebpf_types.go",
     hdrs = ["//pkg/collector/corechecks/ebpf/c/runtime:corecheck_ebpf_headers"],
+    visibility = ["//visibility:public"],
 )

pkg/collector/corechecks/ebpf/probe/oomkill/BUILD.bazel

@@ -11,4 +11,5 @@ cgo_godefs(
     name = "c_types_godefs",
     src = "c_types.go",
     hdrs = ["//pkg/collector/corechecks/ebpf/c/runtime:corecheck_ebpf_headers"],
+    visibility = ["//visibility:public"],
 )