[9.3](backport #50191) otel: option to preserve at metadata field in log record body

[mergify]

Proposed commit message

The at metadata fields were removed in #42469 because it was deeemed unnecessary. But Logstash and Kafka require these fields for document compatibility with the standalone libbeat outputs.

We cannot include these fields unconditionally because the most commonly used exporters (Elasticsearch and OTLP for agentless) don't use them, which would incur additional allocations and CPU overhead. To address this, we add a configuration option to the Beats receiver to enable this behavior when needed. This can be used as a configuration knob by the EDOT runtime.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR locally

(cd x-pack/filebeat && mage docker:composeUp)

# tests
go test -v -tags integration -run 'TestFilebeatOTelKafka.*' ./x-pack/filebeat/tests/integration/ -count=1
./script/stresstest.sh --tags integration ./x-pack/filebeat/tests/integration 'TestFilebeatOTelKafka.*'
9m10s: 2056 runs so far, 0 failures, 32 active

# benches
go test -bench=BenchmarkOTelConsumerIncludeMetadata -benchmem -run='^$' -tags integration ./x-pack/filebeat/tests/integration

Related issues

• Relates otelconsumer: remove metadata and adjust timestamp format #42469
• For https://github.com/elastic/ingest-dev/issues/7389

---

This is an automatic backport of pull request #50191 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 2e949e6 has failed:

On branch mergify/bp/9.3/pr-50191
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 2e949e6c4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   x-pack/filebeat/tests/integration/otel_test.go
	modified:   x-pack/libbeat/cmd/instance/beat.go
	modified:   x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/beat/info.go
	both modified:   x-pack/filebeat/fbreceiver/config_test.go
	both modified:   x-pack/filebeat/tests/integration/otel_kafka_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

TL;DR

golangci-lint failed in lint (ubuntu-latest) because x-pack/libbeat/cmd/instance/beat.go still references the global paths.Paths, which is forbidden by the repo lint rule. Replace that assignment with a per-beat *paths.Path initialization and re-run lint.

Remediation

• In x-pack/libbeat/cmd/instance/beat.go, update the non-filebeat branch in NewBeatForReceiver to stop assigning b.Info.Paths = paths.Paths; initialize and assign a per-beat *paths.Path (same pattern used in the filebeat branch with p := paths.New(); p.InitPaths(...)).
• Re-run the same lint check locally (or rerun the workflow) after this change to confirm forbidigo passes.

Investigation details

Root Cause

The workflow failure is a forbidigo lint violation. The rule explicitly bans paths.Paths and requires a per-beat path instance.

Evidence

• Workflow: https://github.com/elastic/beats/actions/runs/24777155202
• Job/step: lint (ubuntu-latest) → golangci-lint
• Key log excerpt:

x-pack/libbeat/cmd/instance/beat.go:129:13: use of `paths.Paths` forbidden because "use a per-beat *paths.Path instance instead of the global paths.Paths" (forbidigo)
1 issues:
issues found

Validation

• Not run (read-only detective workflow; diagnosis based on workflow job logs).

Follow-up

• After updating the path assignment, trigger a rerun of golangci-lint for this PR.

---

What is this? | From workflow: PR Actions Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mauri870]

/test

[mauri870]

Blocked on the VM image fixes from #50254.

[mauri870]

/test

[mauri870]

@pierrehilbert This is stuck on the known issue with VM images. Can you force-merge this PR? Thanks.

[pierrehilbert]

Is it really needed in 9.3?
The kafka output will be enabled starting in 9.4.0.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @mauri870? 🙏

[mauri870]

Is it really needed in 9.3? The kafka output will be enabled starting in 9.4.0.

Fair point, 8.19 backport was closed as well. Thanks for pointing it out, closing this.