[9.4](backport #51591) remove global paths from metricbeat

[mergify]

Proposed commit message

This change removes the use of global/singleton path state from Metricbeat and replaces it with per-instance paths, fixing a bug where multiple Metricbeat receivers could share the same data store.

Problem

Metricbeat was using global paths.Paths (a package-level singleton) when resolving file paths like the SQL cursor data store. In receiver mode, multiple Metricbeat instances share the same process, so they would all resolve to the same data path, causing them to share state incorrectly.

Changes

• metricbeat/mb/mb.go: Added a Paths *paths.Path field to BaseModule so each module carries its own path instance.
• metricbeat/mb/builders.go: Threaded the *paths.Path parameter through newBaseModuleFromConfig so it is assigned to the BaseModule at construction time.
• metricbeat/mb/lightmetricset.go: Updated LightMetricSet.baseModule() to pass the metric set's path to newBaseModuleFromConfig.
• x-pack/libbeat/cmd/instance/beat.go: Simplified NewBeatForReceiver by removing the filebeat-specific branch for path initialization — both filebeat and metricbeat now use the same path initialization path, constructing a fresh paths.Path instance per beat.
• x-pack/metricbeat/module/sql/module.go: Replaced paths.Resolve(paths.Data, "sql-cursor") (global) with m.Paths.Resolve(paths.Data, "sql-cursor") (per-instance), so each SQL module instance resolves its own data directory.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. Stand alone Metricbeat will function as before, only metricbeat receiver will have multiple beat instances in the same process. This change will allow the sql module to have unique data path per beat instance.

How to test this PR locally

cd x-pack/metricbeat/module/sql
go test .

Related issues

• Closes Metricbeat receivers share the same store and "corrupt" data elastic-agent#15154

Use cases

Screenshots

Logs

---

This is an automatic backport of pull request #51591 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 37f3d26 has failed:

On branch mergify/bp/9.4/pr-51591
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 37f3d269d.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1782499306-remove-global-paths-from-metricbeat.yaml
	modified:   metricbeat/mb/lightmetricset.go
	modified:   metricbeat/mb/lightmetricset_test.go
	modified:   metricbeat/mb/mb.go
	modified:   metricbeat/mb/mb_test.go
	modified:   x-pack/metricbeat/module/sql/module.go
	modified:   x-pack/metricbeat/module/sql/module_test.go
	modified:   x-pack/metricbeat/module/sql/query/cursor/store_test.go
	modified:   x-pack/metricbeat/module/sql/query/cursor_integration_test.go
	modified:   x-pack/metricbeat/module/sql/query/query_integration_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   metricbeat/mb/builders.go
	both modified:   x-pack/libbeat/cmd/instance/beat.go
	both modified:   x-pack/metricbeat/module/sql/query/query_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

TL;DR

Both failed Buildkite jobs stopped in pre-commit run --all-files because the PR head still contains unresolved cherry-pick conflict markers in three files. Resolve those conflict blocks with the 9.4 branch API, then rerun pre-commit.

Remediation

• In metricbeat/mb/builders.go:61, keep the 9.4 NewModule(config, r, p, logger) signature and resolve the call as newBaseModuleFromConfig(config, logger, p) so the new per-instance paths are passed without referencing undefined info.
• In x-pack/libbeat/cmd/instance/beat.go:116, remove the conflict block and initialize a fresh paths.Path once for all receivers: cfg.Unpack, paths.New(), p.InitPaths(&partialConfig.Path), then b.Info.Paths = p.
• In x-pack/metricbeat/module/sql/query/query_test.go:170, use the 9.4 call form mb.NewModule(c, mb.Registry, beatPaths, logptest.NewTestingLogger(t, "")) in the helper sites; do not use beat.Info on this branch.
• Validate with pre-commit run --all-files and cd x-pack/metricbeat/module/sql && go test ..

Investigation details

Root Cause

This is a backport conflict, not a flaky test or infrastructure issue. The Buildkite pre-commit hook found literal merge-conflict markers in the PR head:

• x-pack/libbeat/cmd/instance/beat.go:116, x-pack/libbeat/cmd/instance/beat.go:130, x-pack/libbeat/cmd/instance/beat.go:133
• metricbeat/mb/builders.go:61, metricbeat/mb/builders.go:63, metricbeat/mb/builders.go:65
• x-pack/metricbeat/module/sql/query/query_test.go:170, x-pack/metricbeat/module/sql/query/query_test.go:172, x-pack/metricbeat/module/sql/query/query_test.go:175

The incoming change also references the newer mb.NewModule(..., beat.Info{...}) shape, while the PR head on 9.4 still has func NewModule(config *conf.C, r *Register, p *paths.Path, logger *logp.Logger). The conflict resolution should therefore thread p/logger through the existing 9.4 signature instead of taking the incoming beat.Info hunk verbatim.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/49009
• Failed jobs:
  • x-pack/dockerlogbeat: Run pre-commit
  • x-pack/packetbeat: Run pre-commit
• Key log excerpt:

check for merge conflicts................................................Failed
- hook id: check-merge-conflict
- exit code: 1

x-pack/libbeat/cmd/instance/beat.go:116: Merge conflict string '<<<<<<<' found
x-pack/libbeat/cmd/instance/beat.go:130: Merge conflict string '=======' found
x-pack/libbeat/cmd/instance/beat.go:133: Merge conflict string '>>>>>>>' found
metricbeat/mb/builders.go:61: Merge conflict string '<<<<<<<' found
metricbeat/mb/builders.go:63: Merge conflict string '=======' found
metricbeat/mb/builders.go:65: Merge conflict string '>>>>>>>' found
x-pack/metricbeat/module/sql/query/query_test.go:170: Merge conflict string '<<<<<<<' found
x-pack/metricbeat/module/sql/query/query_test.go:172: Merge conflict string '=======' found
x-pack/metricbeat/module/sql/query/query_test.go:175: Merge conflict string '>>>>>>>' found

The PR head also shows x-pack/metricbeat/module/sql/query/query_test.go:170-175 split between the old paths.New() call and the incoming beat.Info{Paths: beatPaths, Logger: ...} call.

Verification

Not run locally: the failure is fully explained by the pre-commit merge-conflict check and the PR ref content, and the local workspace is not checked out at the PR head.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.