Refactor redis and mqtt tests

[khushijain21]

Proposed commit message

This PR moves some test functions into helper packages required for e2e testing for redis and mqtt inputs in otel mode

Related issues

• Part of [beatreceiver] Add e2e test to compare docs from redis and mqtt input #51565

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: cf2b4ec8-c72f-4b1b-a7b0-b4223feac69c

📥 Commits

Reviewing files that changed from the base of the PR and between 8f14f7b and c92f474.

📒 Files selected for processing (2)

• filebeat/input/mqtt/testutil/testutil.go
• filebeat/input/redis/testutil/testutil.go

🚧 Files skipped from review as they are similar to previous changes (2)

• filebeat/input/redis/testutil/testutil.go
• filebeat/input/mqtt/testutil/testutil.go

---

📝 Walkthrough

Walkthrough

Changes

This change introduces shared testutil packages for MQTT and Redis integration tests, consolidating broker/host configuration, client/publisher creation, and message emission logic previously duplicated in test files. mqtt_integration_test.go and redis_integration_test.go are refactored to use these helpers, replacing local hostPort derivation, publisher/client creation, and data-emission code. The Redis input _meta Dockerfile is emptied, and docker-compose.yml now builds Redis services from the shared testing/environments/docker/redis context, whose Dockerfile is upgraded to redis:7.2.4-alpine with a simplified healthcheck.

Related PRs: None identified.

Suggested labels: test, refactor, filebeat

Suggested reviewers: None identified.

Poem
A rabbit hopped through Redis dens and MQTT streams,
Consolidating helpers into tidy testutil themes.
No more hostPort scattered, no more copy-paste,
Just shared pools and publishers, deployed with haste.
Seven-two-four-alpine now guards the healthcheck gate. 🐇

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

• 🛠️ Update Documentation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@filebeat/input/mqtt/testutil/testutil.go`:
- Around line 74-95: The failure handling in EmitMessages is unsafe because
require.Fail and require.NoError are called from the goroutine started by go
func(), which can leave the test hanging instead of failing cleanly. Update
EmitMessages to avoid require inside the background goroutine by using
assert-style checks or by sending publish errors back to the main test
goroutine; make sure the fix is applied at the Publish/WaitTimeout error
handling path and aligns with the downstream emitInputData/PublishMessage usage
pattern in the MQTT tests.

In `@filebeat/input/redis/testutil/testutil.go`:
- Around line 92-112: `EmitInputData` is calling `require.NoError` from inside
the background goroutine, which is unsafe because it can only stop that
goroutine and may panic after the test ends. Update the goroutine in
`testutil.EmitInputData` to use `assert.NoError` (or otherwise record the error
without `FailNow` semantics) for both `conn.Close()` and `conn.Do(...)`, while
keeping the surrounding `ticker`/`ctx.Done()` loop unchanged.

In `@x-pack/filebeat/tests/integration/otel_filebeat_input_test.go`:
- Around line 353-403: TestFilebeatOTelHTTPJSONInput starts a Beat instance
without matching cleanup, so add a deferred filebeat.Stop() right after
filebeat.Start() and register t.Cleanup to delete the data streams created by
the test using deleteDataStreamsFromES with the otelNamespace and fbNameSpace
patterns. Use the existing filebeat.Start(), filebeat.Stop(), and
deleteDataStreamsFromES helpers in this test so it matches the cleanup pattern
used by the other OTel integration tests.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 66961fe0-b4fa-45d0-bc7b-d3c698e82961

📥 Commits

Reviewing files that changed from the base of the PR and between be145e4 and 8f14f7b.

⛔ Files ignored due to path filters (3)

• testing/environments/docker/redis/certs/root-ca.pem is excluded by !**/*.pem
• testing/environments/docker/redis/certs/server-cert.pem is excluded by !**/*.pem
• testing/environments/docker/redis/certs/server-key.pem is excluded by !**/*.pem

📒 Files selected for processing (12)

• filebeat/docker-compose.yml
• filebeat/input/mqtt/mqtt_integration_test.go
• filebeat/input/mqtt/testutil/testutil.go
• filebeat/input/redis/_meta/Dockerfile
• filebeat/input/redis/redis_integration_test.go
• filebeat/input/redis/testutil/testutil.go
• testing/environments/docker/redis/Dockerfile
• testing/environments/docker/redis/Dockerfile-tls
• testing/environments/docker/redis/env
• testing/environments/docker/redis/redis.conf
• x-pack/filebeat/tests/integration/otel_filebeat_input_test.go
• x-pack/otel/oteltestcol/collector.go

💤 Files with no reviewable changes (1)

• filebeat/input/redis/_meta/Dockerfile

[github-actions]

TL;DR

Metricbeat: Run check/update is failing in go vet because metricbeat/mb/module/runner_test.go:173 calls module.NewWrapper with a *logp.Logger, but module.NewWrapper now expects a *beat.Info as its third argument. This is not from the Redis/MQTT refactor in this PR; the same scoped package fails on the checked-out base commit.

Remediation

• Change metricbeat/mb/module/runner_test.go:173 to pass &beat.Info{Logger: logptest.NewTestingLogger(t, "")} as the third module.NewWrapper argument, matching the existing calls at metricbeat/mb/module/runner_test.go:57 and metricbeat/mb/module/runner_test.go:85.
• Re-run cd metricbeat && go test ./mb/module or retry make -C metricbeat check update after the one-line test fix.

Investigation details

Root Cause

metricbeat/mb/module/wrapper.go:93 defines NewWrapper as:

func NewWrapper(config *conf.C, r *mb.Register, info *beat.Info, monitoring beatmonitoring.Monitoring, p *paths.Path, options ...Option) (*Wrapper, error)

But metricbeat/mb/module/runner_test.go:173 still passes the logger directly:

m, err := module.NewWrapper(config, mb.Registry, logptest.NewTestingLogger(t, ""), beatmonitoring.NewMonitoring(), paths.New(), module.WithMetricSetInfo())

The same file already has the correct pattern at metricbeat/mb/module/runner_test.go:57 and metricbeat/mb/module/runner_test.go:85:

&beat.Info{Logger: logptest.NewTestingLogger(t, "")}

Evidence

• Build: https://buildkite.com/elastic/beats/builds/48810
• Job/step: Metricbeat: Run check/update
• Command: make -C metricbeat check update && make check-no-changes
• Key log excerpt:

# github.com/elastic/beats/v7/metricbeat/mb/module_test
# [github.com/elastic/beats/v7/metricbeat/mb/module_test]
vet: mb/module/runner_test.go:173:51: cannot use logptest.NewTestingLogger(t, "") (value of type *logp.Logger) as *beat.Info value in argument to module.NewWrapper
Error: failed running go vet, please fix the issues reported: running "go vet ./..." failed with exit code 1

The PR changed 13 files under Filebeat Redis/MQTT test helpers and Docker setup; no metricbeat/mb/module files are included in the PR diff.

Verification

• Ran cd metricbeat && go test ./mb/module on checked-out base commit c2c559cdf0cf0855ccc4c303be0e9c5452cd47fc; it failed with the same compiler diagnostic:

mb/module/runner_test.go:173:51: cannot use logptest.NewTestingLogger(t, "") (value of type *logp.Logger) as *beat.Info value in argument to module.NewWrapper
FAIL	github.com/elastic/beats/v7/metricbeat/mb/module [build failed]

Follow-up

If this gets fixed on main first, rebase this PR and retry Buildkite. Otherwise, include the one-line runner_test.go update to unblock the Metricbeat check/update job.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[cmacknz]

Fine besides comment on the healthcheck in the redis container potentially getting slower. Approving pre-emptively.