otel: option to preserve at metadata field in log record body

[mauri870]

Proposed commit message

The at metadata fields were removed in #42469 because it was deeemed unnecessary. But Logstash and Kafka require these fields for document compatibility with the standalone libbeat outputs.

We cannot include these fields unconditionally because the most commonly used exporters (Elasticsearch and OTLP for agentless) don't use them, which would incur additional allocations and CPU overhead. To address this, we add a configuration option to the Beats receiver to enable this behavior when needed. This can be used as a configuration knob by the EDOT runtime.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR locally

(cd x-pack/filebeat && mage docker:composeUp)

# tests
go test -v -tags integration -run 'TestFilebeatOTelKafka.*' ./x-pack/filebeat/tests/integration/ -count=1
./script/stresstest.sh --tags integration ./x-pack/filebeat/tests/integration 'TestFilebeatOTelKafka.*'
9m10s: 2056 runs so far, 0 failures, 32 active

# benches
go test -bench=BenchmarkOTelConsumerIncludeMetadata -benchmem -run='^$' -tags integration ./x-pack/filebeat/tests/integration

Related issues

• Relates otelconsumer: remove metadata and adjust timestamp format #42469
• For https://github.com/elastic/ingest-dev/issues/7389

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-metadata upstream/otel-metadata
git merge upstream/main
git push upstream otel-metadata

[github-actions]

TL;DR

Buildkite failed in x-pack/filebeat: Run check/update because go vet hit a compile-time test error in x-pack/filebeat/fbreceiver/config_test.go: the test still references cfg.IncludeMetadata, but that field was removed from Config.

Remediation

• Update x-pack/filebeat/fbreceiver/config_test.go to match the new Config shape after ef054f58d8 removed xpInstance.ReceiverConfig from fbreceiver.Config.
• Either:
  1. remove/replace TestUnmarshalIncludeMetadata, or
  2. reintroduce the embedded receiver config field in x-pack/filebeat/fbreceiver/config.go if keeping include_metadata in this config is intentional.
• Re-run make -C x-pack/filebeat check update && make check-no-changes.

Investigation details

Root Cause

x-pack/filebeat/fbreceiver/config.go:17-19 now defines:
type Config struct { Beatconfig map[string]any \mapstructure:",remain"` }`

So Config no longer has IncludeMetadata.

But x-pack/filebeat/fbreceiver/config_test.go:85 still has:
assert.True(t, cfg.IncludeMetadata)

That mismatch causes vet/compile failure.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/44442
• Job/step: x-pack/filebeat: Run check/update
• Key log excerpt:

vet: fbreceiver/config_test.go:85:21: cfg.IncludeMetadata undefined (type *Config has no field or method IncludeMetadata)
Error: failed running go vet, please fix the issues reported: running "go vet ./..." failed with exit code 1

• Related PR commit:
  • ef054f58d8 (simplify passing of the include_metadata option) removes embedded xpInstance.ReceiverConfig from fbreceiver.Config.

Verification

• Not run locally end-to-end here due environment/toolchain fetch overhead; diagnosis is from Buildkite logs plus source/commit inspection.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• otel: option to preserve at metadata field in log record body #50191 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50191 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6405a3b8-d475-487b-9f62-9a3e18b3ee5a

📥 Commits

Reviewing files that changed from the base of the PR and between 5afcabd and 227b33e.

📒 Files selected for processing (6)

• libbeat/beat/info.go
• libbeat/otel/otelconsumer/otelconsumer.go
• x-pack/filebeat/fbreceiver/config_test.go
• x-pack/filebeat/tests/integration/otel_kafka_test.go
• x-pack/filebeat/tests/integration/otel_test.go
• x-pack/libbeat/cmd/instance/beat.go

---

📝 Walkthrough

Walkthrough

This change introduces an IncludeMetadata configuration option to control whether metadata is enriched in OTel log records. The Info struct receives a new boolean field, which is read from receiver configuration in beat initialization. When enabled, the logsPublish function enriches each log record's body with an @metadata field populated from event metadata, beat information, and version. Tests validate configuration unmarshaling, metadata inclusion, and performance characteristics across different metadata settings.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

• 🛠️ Update Documentation: Commit on current branch
• 🛠️ Update Documentation: Create PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[rdner]

LGTM

Since we've added a benchmark in the PR, we should also include results before and after in the description.

[khushijain21]

LGTM!

We would have to update kafka and logstash translation logic in elastic-agent after this is merged. I can take that up cc: @mauri870

[github-actions]

@Mergifyio backport 8.19 9.3 9.4

[mergify]

backport 8.19 9.3 9.4

✅ Backports have been created

Details

• #50241 [8.19](backport #50191) otel: option to preserve at metadata field in log record body has been created for branch 8.19 but encountered conflicts

Cherry-pick of 2e949e6 has failed:

On branch mergify/bp/8.19/pr-50191
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 2e949e6c4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   x-pack/filebeat/tests/integration/otel_test.go
	modified:   x-pack/libbeat/cmd/instance/beat.go
	modified:   x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/beat/info.go
	both modified:   x-pack/filebeat/fbreceiver/config_test.go
	both modified:   x-pack/filebeat/tests/integration/otel_kafka_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #50242 [9.3](backport #50191) otel: option to preserve at metadata field in log record body has been created for branch 9.3 but encountered conflicts

Cherry-pick of 2e949e6 has failed:

On branch mergify/bp/9.3/pr-50191
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 2e949e6c4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   x-pack/filebeat/tests/integration/otel_test.go
	modified:   x-pack/libbeat/cmd/instance/beat.go
	modified:   x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/beat/info.go
	both modified:   x-pack/filebeat/fbreceiver/config_test.go
	both modified:   x-pack/filebeat/tests/integration/otel_kafka_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #50243 [9.4](backport #50191) otel: option to preserve at metadata field in log record body has been created for branch 9.4