[beatreceiver] Preserve metadata from OTel log body for LS exporter by khushijain21 · Pull Request #50145 · elastic/beats

khushijain21 · 2026-04-16T07:16:13Z

Proposed commit message

This PR preserves metadata from OTel log body going out of LS exporter

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None

Related issues

Note:
A follow up PR on elastic-agent will include_metadata:true for logstash output elastic/elastic-agent#13713

github-actions · 2026-04-16T07:16:26Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2026-04-16T07:16:51Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

khushijain21 · 2026-04-16T10:10:50Z

-// getEventMeta gives beat.Event.Meta from the context metadata
-// The value of `[@metadata][beat]` is taken from the `Index` option of logstash output.
-// In Elastic Agent, `Index` option is not available, hence, the value of `[@metadata][beat]` is derived from `IndexPrefix`
-func getEventMeta(ctx context.Context) map[string]any {


We don't need to add this metadata field. They are already being added here by logstash client in beats. https://github.com/khushijain21/beats/blob/logstash-metadata/libbeat/outputs/codec/json/event.go#L46-L51

Can also be proven by e2e tests when these are removed

khushijain21 · 2026-04-17T04:29:35Z

Blocked by. https://github.com/elastic/ingest-dev/issues/7389. Waiting on how we decide to preserve metadata going out of otelconsumer - changes here will proceed accodingly

cmacknz · 2026-04-21T15:21:01Z

Blocked by. https://github.com/elastic/ingest-dev/issues/7389. Waiting on how we decide to preserve metadata going out of otelconsumer - changes here will proceed accodingly

Let's fix this for Logstash so that it is completely functional and deal with generalizing to Kafka later. Logstash is much more widely used than Kafka right now.

cmacknz · 2026-04-21T15:21:55Z

The cost of rework here is something we can accept if we change the approach later, it's more important that the Logstash exporter works as it needs to.

cmacknz · 2026-04-21T15:24:23Z

Looks like #50191 is ready to merge already

elasticmachine · 2026-04-21T15:35:28Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

coderabbitai

♻️ Duplicate comments (1)

x-pack/otel/exporter/logstashexporter/internal/event.go (1)

25-35: ⚠️ Potential issue | 🟠 Major

Promote body @metadata into beat.Event.Meta before serialization.

libbeat/outputs/codec/json/event.go emits beat.Event.Meta as @metadata and inlines Fields; leaving source @metadata in Fields creates two writers for the same key. Also, deleting beat/version/type from the top-level body can drop user fields. Extract fields["@metadata"] into event.Meta, delete it from Fields, and trim generated keys from that metadata map only.

🐛 Proposed fix

-	removeRedundantMetadataFields(fields)
+	metadata := extractEventMetadata(fields)

 	timestamp, ok := parseEventTimestamp(fields)
 	if !ok {
 		timestamp = logRecord.ObservedTimestamp().AsTime()
 	}

 	event := beat.Event{
 		Timestamp: timestamp,
 		Fields:    fields,
+		Meta:      metadata,
 	}

+func extractEventMetadata(fields map[string]any) map[string]any {
+	rawMetadata, ok := fields["@metadata"].(map[string]any)
+	if !ok {
+		return nil
+	}
+
+	delete(fields, "@metadata")
+	removeRedundantMetadataFields(rawMetadata)
+	return rawMetadata
+}
+
 // removeRedundantMetadataFields removes certain metadata fields that will be generated again when the event is serialized.
 // See https://github.com/elastic/beats/blob/v9.3.3/libbeat/outputs/codec/json/event.go#L43-L54
 // Not removing these fields would create duplicate fields and bloat the final event size
-func removeRedundantMetadataFields(fields map[string]any) {
-	delete(fields, otelctx.MetadataBeatKey)
-	delete(fields, otelctx.MetadataVersionKey)
-	delete(fields, "type")
+func removeRedundantMetadataFields(metadata map[string]any) {
+	delete(metadata, otelctx.MetadataBeatKey)
+	delete(metadata, otelctx.MetadataVersionKey)
+	delete(metadata, "type")
 }

Also applies to: 62-68

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@x-pack/otel/exporter/logstashexporter/internal/event.go` around lines 25 -
35, The code must extract any source metadata from fields["@metadata"] into the
beat.Event.Meta before serialization to avoid duplicated `@metadata` keys; modify
the construction around removeRedundantMetadataFields / parseEventTimestamp /
beat.Event so that you look up v := fields["@metadata"], if present cast to
map[string]interface{} and assign it to event.Meta, then delete
fields["@metadata"] from Fields; ensure subsequent trimming/removal of generated
keys (like "beat","version","type") is applied only to that extracted meta map
rather than deleting top-level user fields in Fields; keep the existing
timestamp fallback logic from parseEventTimestamp as-is.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@x-pack/otel/exporter/logstashexporter/internal/event.go`:
- Around line 25-35: The code must extract any source metadata from
fields["@metadata"] into the beat.Event.Meta before serialization to avoid
duplicated `@metadata` keys; modify the construction around
removeRedundantMetadataFields / parseEventTimestamp / beat.Event so that you
look up v := fields["@metadata"], if present cast to map[string]interface{} and
assign it to event.Meta, then delete fields["@metadata"] from Fields; ensure
subsequent trimming/removal of generated keys (like "beat","version","type") is
applied only to that extracted meta map rather than deleting top-level user
fields in Fields; keep the existing timestamp fallback logic from
parseEventTimestamp as-is.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9165e98c-6cac-4a58-97d7-31613e6b7a57

📥 Commits

Reviewing files that changed from the base of the PR and between 741bd17 and beaa9c2.

📒 Files selected for processing (5)

x-pack/otel/exporter/logstashexporter/exporter.go
x-pack/otel/exporter/logstashexporter/exporter_test.go
x-pack/otel/exporter/logstashexporter/internal/batch.go
x-pack/otel/exporter/logstashexporter/internal/batch_test.go
x-pack/otel/exporter/logstashexporter/internal/event.go

cmacknz · 2026-04-21T19:00:22Z

I don't see a test proving that raw_index and input_id are preserved, those weren't in the existing tests either so they'd have to be new. The Elastic Agent tests will prove this but it would be nice to confirm here before we go through the dependency update process.

khushijain21 · 2026-04-22T04:00:23Z

I don't see a test proving that raw_index and input_id are preserved, those weren't in the existing tests either so they'd have to be new. The Elastic Agent tests will prove this but it would be nice to confirm here before we go through the dependency update process.

I added a similar test i.e system/metics with logstash output in beats. But looks like input_id and raw_index metadata are only added when run from inside elastic-agent. So we can only confirm inside elastic-agent

khushijain21 · 2026-04-22T04:32:47Z

/test

khushijain21 · 2026-04-22T11:02:55Z

I tested this changes in elastic-agent here and it works

github-actions · 2026-04-22T17:15:35Z

TL;DR

Three Linux FIPS jobs failed before tests ran due mage not being runnable in that environment (explicit in libbeat, and same failure signature in the two FIPS ECH jobs). The Windows auditbeat failure is a separate flaky test (TestLeak) where beat.db is still locked at temp-dir cleanup.

Remediation

Align FIPS jobs’ mage/toolchain versions: either install mage 1.15.0 on the FIPS image(s) or temporarily set ASDF_MAGE_VERSION back to a version present on the image.
Stabilize x-pack/auditbeat/abreceiver leak test on Windows by waiting for full shutdown/file handle release before temp-dir cleanup (or skipping this leak assertion on Windows until fixed).
Re-run the Buildkite jobs after the CI image/version alignment and test stabilization.

Investigation details

Root Cause

Configuration / CI environment mismatch (Linux FIPS jobs):
- Pipelines request ASDF_MAGE_VERSION: 1.15.0 in:
  - .buildkite/libbeat/pipeline.libbeat.yml:24
  - .buildkite/x-pack/pipeline.xpack.filebeat.yml:25
  - .buildkite/x-pack/pipeline.xpack.metricbeat.yml:25
- FIPS ECH test script executes mage directly: .buildkite/scripts/custom_fips_ech_test.sh:19 (GOEXPERIMENT=systemcrypto SNAPSHOT=true FIPS=true mage build fipsECHTest).
- Libbeat failing log explicitly shows missing preset:
  - /tmp/gh-aw/buildkite-logs/beats-libbeat-ubuntu-libbeat-ubuntu-x86_64-go-unit-tests-with-fips-provide.txt:125 No preset version installed for command mage
  - ...:131 asdf install mage 1.15.0
  - Job exits 126 at ...:140.
- The two FIPS ECH jobs also exit 126 with no test artifacts, matching the same pre-test toolchain failure pattern:
  - /tmp/gh-aw/buildkite-logs/beats-xpack-filebeat-ubuntu-x-packfilebeat-fips-ech-integration-tests.txt:140
  - /tmp/gh-aw/buildkite-logs/beats-xpack-metricbeat-ubuntu-x-packmetricbeat-fips-ech-integration-tests.txt:140
Test failure / flake (Windows auditbeat):
- /tmp/gh-aw/buildkite-logs/beats-xpack-auditbeat-windows-x-packauditbeat-win-2016-unit-tests.txt:130:
  - TempDir RemoveAll cleanup: unlinkat ...\data\beat.db: The process cannot access the file because it is being used by another process.
- Failing test: x-pack/auditbeat/abreceiver TestLeak (...:129).
- Relevant shutdown path in test:
  - x-pack/auditbeat/abreceiver/receiver_leak_test.go:96 calls rec.Shutdown(...)
  - temp dirs are configured in the same test (receiver_leak_test.go:42, :53, :58) and Windows cleanup races with lingering DB handle.

Evidence

Build: https://buildkite.com/elastic/beats/builds/44640
Failed jobs:
Key excerpts:
- No preset version installed for command mage
- Error: The command exited with status 126 (3 Linux jobs)
- TempDir RemoveAll cleanup ... beat.db ... used by another process (Windows job)

Verification

Not run locally in this workflow (analysis based on provided Buildkite failure logs and repository pipeline/test sources).

Follow-up

I could not complete a reliable prior-comment dedupe check because PR read endpoints are integrity-filtered in this runtime; this report is based on current logs and source evidence.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

[beatreceiver] Preserve metadata from OTel log body for LS exporter #50145 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#50145 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

cmacknz · 2026-04-22T19:22:20Z

Force merging, test failures are unrelated

…50145) (#50274) * when metadata is passed via otelconsumer * fix tests * add docker-compose back * remove metadta * include metadata in e2e test * fix event size * address comments --------- (cherry picked from commit 15c13a5) Co-authored-by: Khushi Jain <khushi.jain@elastic.co> Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>

…lastic#50145) * when metadata is passed via otelconsumer * fix tests * add docker-compose back * remove metadta * include metadata in e2e test * fix event size * address comments --------- Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>

when metadata is passed via otelconsumer

7e4a146

botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 16, 2026

mergify Bot assigned khushijain21 Apr 16, 2026

This comment has been minimized.

Sign in to view

khushijain21 commented Apr 16, 2026

View reviewed changes

khushijain21 added 2 commits April 16, 2026 15:46

fix tests

abab5ac

add docker-compose back

bc6cb01

khushijain21 changed the title ~~when metadata is passed via otelconsumer~~ [beatreceiver] Preserve metadata from OTel log body for LS exporter Apr 16, 2026

khushijain21 added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Apr 16, 2026

botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 16, 2026

khushijain21 added skip-changelog backport-9.4 labels Apr 16, 2026

This comment has been minimized.

Sign in to view

mauri870 reviewed Apr 17, 2026

View reviewed changes

Comment thread libbeat/otel/otelconsumer/otelconsumer.go Outdated

khushijain21 added 2 commits April 21, 2026 21:02

remove metadta

592bc71

Merge remote-tracking branch 'upstream/main' into logstash-metadata

7552c17

khushijain21 marked this pull request as ready for review April 21, 2026 15:34

khushijain21 requested a review from a team as a code owner April 21, 2026 15:34

khushijain21 requested review from AndersonQ and belimawr April 21, 2026 15:34

khushijain21 requested review from edmocosta and kaisecheng April 21, 2026 15:36

coderabbitai Bot reviewed Apr 21, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

VihasMakwana reviewed Apr 21, 2026

View reviewed changes

Comment thread x-pack/otel/exporter/logstashexporter/internal/event.go Outdated

khushijain21 added 2 commits April 22, 2026 09:31

address comments

aecf3c8

Merge branch 'main' into logstash-metadata

9b5e117

khushijain21 marked this pull request as draft April 22, 2026 10:52

khushijain21 marked this pull request as ready for review April 22, 2026 10:54

Merge branch 'main' into logstash-metadata

ebb7973

mauri870 approved these changes Apr 22, 2026

View reviewed changes

khushijain21 and others added 2 commits April 22, 2026 20:40

Merge branch 'main' into logstash-metadata

b884d2b

Merge branch 'main' into logstash-metadata

14cdd2a

This comment has been minimized.

Sign in to view

cmacknz merged commit 15c13a5 into elastic:main Apr 22, 2026
189 of 206 checks passed

mergify Bot mentioned this pull request Apr 22, 2026

[9.4](backport #50145) [beatreceiver] Preserve metadata from OTel log body for LS exporter #50274

Merged

6 tasks

This was referenced Apr 23, 2026

[bug-hunter] Logstash OTel exporter drops user fields named beat/version/type #50289

Closed

[breaking-change] Undocumented breaking changes detected (2026-04-23) #50292

Open

This was referenced Apr 24, 2026

x-pack/filebeat/input/entityanalytics: add entcollect adapter layer #49871

Merged

[metricbeat: Azure Applications Insights] Migrate azure/app_insights off the deprecated track-1 Azure SDK #50392

Merged

Uh oh!

Conversation

khushijain21 commented Apr 16, 2026 • edited by cmacknz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Disruptive User Impact

Related issues

Uh oh!

github-actions Bot commented Apr 16, 2026

🤖 GitHub comments

Uh oh!

mergify Bot commented Apr 16, 2026

Uh oh!

This comment has been minimized.

khushijain21 Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

khushijain21 commented Apr 17, 2026

Uh oh!

Uh oh!

cmacknz commented Apr 21, 2026

Uh oh!

cmacknz commented Apr 21, 2026

Uh oh!

cmacknz commented Apr 21, 2026

Uh oh!

elasticmachine commented Apr 21, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

cmacknz commented Apr 21, 2026

Uh oh!

This comment has been minimized.

Uh oh!

khushijain21 commented Apr 22, 2026

Uh oh!

khushijain21 commented Apr 22, 2026

Uh oh!

khushijain21 commented Apr 22, 2026

Uh oh!

This comment has been minimized.

github-actions Bot commented Apr 22, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

Uh oh!

cmacknz commented Apr 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

khushijain21 commented Apr 16, 2026 •

edited by cmacknz

Loading

khushijain21 Apr 16, 2026 •

edited

Loading