Support testing Beats receivers from Beats repo

[belimawr]

Currently Beats receivers are implemented in the Beats repo, however there is no entrypoint/binary that can be compiled and executed within the Beats repo for testing or debugging.

To circumvent this limitation some tests have been implemented in the elastic-agent repository, ex: TestFilebeatReceiverLogAsFilestream.

It would be nice to be able to test and debug Beats receivers directly from the Beats repo.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

We had the famous otelbeat for a while, but it was eventually removed. The way I do it today is by writing a test that uses the oteltestcol package, that is the closest we can get in the beats repo to a real collector today.

[github-actions]

tl;dr: Beats already has native receiver-testing primitives in-repo, but there is no first-class, documented entrypoint from repo root, so support is present but hard to discover and run consistently.

Recommendation

Create a dedicated root-level receiver test target (for example mage ReceiverTest or make receiver-test) that runs receiver-focused packages, and document it in x-pack/otel/README.md.

Findings

Evidence from code

• Native receiver test harness exists and is shared:
  • x-pack/otel/oteltest/oteltest.go:69-205 (CheckReceivers) creates/starts receivers via Factory.CreateLogs and validates emitted logs/status.
• Native local collector test path exists (no Elastic-Agent translation required):
  • x-pack/otel/oteltestcol/collector.go:103-107 registers filebeatreceiver + metricbeatreceiver factories directly.
  • x-pack/otel/oteltestcol/collector_test.go:15-42 uses raw collector YAML (receivers:) in tests.
• Receiver tests are present but fragmented:
  • x-pack/filebeat/fbreceiver/receiver_test.go:623-695 includes TestConsumeContract.
  • x-pack/metricbeat/mbreceiver/receiver_test.go:40-462 has multiple tests using CheckReceivers, but no equivalent consume-contract test.
• Root aggregation does not obviously surface receiver tests as a dedicated workflow:
  • magefile.go:63-73 (XPack) omits x-pack/otel.
  • Makefile:3-4 project lists omit x-pack/otel.
  • .github/workflows/sweep-otel-beatreceiver-isolation.yml:73 shows explicit receiver-focused go test -race command in a sweeper context.

Verification

Commands run

go test ./x-pack/filebeat/fbreceiver -run TestConsumeContract -count=1

Output:

github.com/aws/aws-sdk-go-v2/service/ec2: .../compile: signal: killed

I could not complete end-to-end runtime verification in this runner due resource limits, but the code-path evidence above is direct and consistent.

Detailed Action Plan

Implementation plan

1. Add a root-level task that runs receiver-focused tests explicitly:
   • go test -race ./x-pack/filebeat/fbreceiver/... ./x-pack/metricbeat/mbreceiver/...
   • Optionally include ./x-pack/otel/... if desired for harness coverage.
2. Document this command/target in x-pack/otel/README.md under a new "Testing Beats receivers" section.
3. Align parity across receivers by adding a consume-contract test for metricbeat receiver (pattern from x-pack/filebeat/fbreceiver/receiver_test.go:623-695).
4. Optionally centralize duplicated test helpers (socket path / monitoring readiness) into x-pack/otel/oteltest for consistency.
5. Add CI wiring (or reuse an existing pipeline step) so this target is exercised outside ad-hoc/sweeper workflows.

Related Items

Links

Type	Link	Relevance
File	x-pack/otel/oteltest/oteltest.go	Shared receiver test harness
File	x-pack/otel/oteltestcol/collector.go	Registers Beats receivers in test collector
File	x-pack/filebeat/fbreceiver/receiver_test.go	Contains consume-contract test
File	x-pack/metricbeat/mbreceiver/receiver_test.go	Receiver tests lacking consume-contract parity
Workflow	.github/workflows/sweep-otel-beatreceiver-isolation.yml	Existing explicit receiver race-test command

Note

🔒 Integrity filtering filtered 16 items

Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:Support testing Beats receivers from Beats repo #49658 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49658 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #38120 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49566 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49354 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49353 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49238 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49327 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #48846 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49405 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #48311 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #48834 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• issue:[Flaky Test] x-pack/metricbeat/mbreceiver TestReceiverStatus #49618 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• pr:x-pack/oteltest: use CollectT for receiver status assertions #49621 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• issue:Remove beatconverter and the oteltranslate package #47904 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• issue:x-pack/oteltest: use CollectT for receiver status assertions #49621 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[cmacknz]

We should have a collector distribution specifically for testing the components we define in beats in a real collector, following the example of https://github.com/elastic/opentelemetry-collector-components/tree/main/distributions/elastic-components or the similar approach in colletor contrib upstream.

Elastic Agent will test against the official EDOT collector distro we send to customers, other repositories defining collector components can have their own internal distro for testing the components they define without issue.

[strawgate]

/ai can you tell me how much memory you have access to in your sandbox

[github-actions]

In this sandbox run, I can see about 15 GiB RAM available to the container (MemTotal: 15Gi, MemAvailable: ~14Gi, from /proc/meminfo and free -h).

I don’t see an explicit cgroup memory cap exposed (/sys/fs/cgroup/... memory limit files are empty here), so the practical limit appears to be the host/container memory currently assigned (~15 GiB).

If helpful, I can also add a tiny script/Make target in the repo to print this consistently during CI runs.

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:Support testing Beats receivers from Beats repo #49658 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Mention in Issue

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[strawgate]

/ai can you try to reproduce the OOM issue? What is taking so much memory? the sdk build should be using ~3gb so how are you running out of memory if you have 15 available?

[github-actions]

I reproduced the OOM behavior locally, and it’s not the OTel SDK builder itself.

The kill happens during go test compile of the receiver packages, specifically while compiling heavy transitive deps:

• github.com/aws/aws-sdk-go-v2/service/ec2: ... compile: signal: killed
• github.com/vmware/govmomi/vim25/types: ... compile: signal: killed

Repro commands I ran:

• /usr/bin/time -v go test -c ./x-pack/metricbeat/mbreceiver
• /usr/bin/time -v go test -run TestMultipleReceivers -count=1 ./x-pack/filebeat/fbreceiver ./x-pack/metricbeat/mbreceiver

Why it’s heavy:

• x-pack/metricbeat/mbreceiver/factory.go imports full module registries (metricbeat/include + x-pack/metricbeat/include).
• metricbeat/include/list_common.go registers all OSS modules, including vsphere (govmomi).
• x-pack/metricbeat/include/list.go registers x-pack modules including AWS.
• x-pack/filebeat/fbreceiver/factory.go imports x-pack/filebeat/include, which registers AWS inputs/modules.

So receiver tests compile “all modules” for both beats, not just receiver-needed bits.

Also CI for x-pack/otel enables both RACE_DETECTOR and TEST_COVERAGE (.buildkite/x-pack/pipeline.xpack.otel.yml), which increases compile/test memory pressure.

Important detail: /usr/bin/time reports the parent go test RSS (around 0.8–1.6 GiB in my runs), but the OOM victim is a child compiler process; the true peak is higher than the parent RSS and can exceed the effective runner/container limit even when host free -h looks like 15 GiB.

Most likely fix direction:

1. Introduce receiver-specific minimal include lists (only modules needed by fbreceiver/mbreceiver tests/runtime).
2. Keep full includes for regular beats binaries, but stop using them in receiver factories.
3. As immediate mitigation, avoid race+coverage together on this job, or split into separate jobs.

If you want, I can put up a PR that adds slim include registries for the two receiver factories and updates tests accordingly.

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:Support testing Beats receivers from Beats repo #49658 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Mention in Issue

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[belimawr]

We had the famous otelbeat for a while, but it was eventually removed. The way I do it today is by writing a test that uses the oteltestcol package, that is the closest we can get in the beats repo to a real collector today.

That works for testing, but it is not perfect. I had it causing problems because the seccomp got installed, then the test used a syscall that is blocked by default. It also does not work for debugging a specific behaviour/config.

Craig's suggestion seems to be the best, it will also provide better isolation for tests.

[strawgate]

/ai Can you write a script that dumps process memory usage at some interval and watch the build? We want to figure out what is taking so much memory, we do not want to guess.

Please work through this until you find a solution that successfully reduces peak memory usage and doesn't crash with OOM

[mauri870]

I gave this a shot. It seemed like a straightforward task, but I got stuck several times and only managed to push through the errors by adding a go.mod to each component (a requirement?) and going a bit crazy with replace directives.

I reached a point where ocb runs without errors, but it gets stuck in a seemingly infinite loop in go mod tidy, likely due to the replace hell.

If anyone wants to take a look: https://github.com/mauri870/beats/pull/new/beats-otel-distribution

EDIT: I managed to fix most issues, now it compiles and starts, produces a 433MB binary and seems to run correctly with the example configuration. Still have to deal with the replace hell.