Skip to content

feat: add add_cloud_metadata processor to OTel Beat processor #48020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
andrzej-stencel merged 2 commits into from
Dec 11, 2025
Merged

feat: add add_cloud_metadata processor to OTel Beat processor #48020

andrzej-stencel merged 2 commits into from
Dec 11, 2025
+34 −12

Conversation

@andrzej-stencel
Copy link
Contributor

@andrzej-stencel andrzej-stencel commented Dec 10, 2025 •
edited
Loading

Proposed commit message

Add add_cloud_metadata processor to the OTel Beat processor.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
  • [ ] I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR locally

  1. Include the processor in Elastic Agent by adding it to https://github.com/elastic/elastic-agent/blob/89e0e9a5b0ebb376773de813a1fd2c0b2f30e74a/internal/pkg/otel/components.go.
  2. Build the Elastic Agent binary
  3. Run the Agent on a cloud instance, e.g. an AWS EC2 instance, with an OTel config that includes the Beat processor.

Example OTel config:

service:
  pipelines:
    logs:
      receivers:
        - filebeatreceiver
      processors:
        - beat
      exporters:
        - debug
receivers:
  filebeatreceiver:
    filebeat:
      inputs:
        - type: filestream
          id: logs-from-files
          paths:
            - /var/log/*.log 
    processors: []
    output:
      otelconsumer:
    path.data: /tmp/1210/data
    path.logs: /tmp/1210/logs
    queue.mem:
      flush.timeout: 0
processors:
  beat:
    processors:
      - add_cloud_metadata:
exporters:
  debug:
    sending_queue:
      enabled: false
    use_internal_logger: false
    verbosity: detailed

@andrzej-stencel andrzej-stencel requested a review from a team as a code owner December 10, 2025 13:59
@andrzej-stencel andrzej-stencel added enhancement release-note:skip The PR should be ignored when processing the changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-active-all Automated backport with mergify to all the active branches skip-changelog labels Dec 10, 2025
@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 10, 2025

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Dec 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@pierrehilbert pierrehilbert requested review from leehinman, mauiroma and mauri870 and removed request for khushijain21 December 11, 2025 08:29
@mauri870
Copy link
Member

mauri870 commented Dec 11, 2025

Can we update TestFilebeatOTelBeatProcessorE2E as well and add add_cloud_metadata to both the processor and receiver collector configurations there? This way we confirm that the cloud metadata was added to ingested events.

@andrzej-stencel andrzej-stencel force-pushed the add-cloud-processor-to-beats-processor branch from 6682559 to ea26721 Compare December 11, 2025 12:24
@andrzej-stencel
Copy link
Contributor Author

andrzej-stencel commented Dec 11, 2025

Can we update TestFilebeatOTelBeatProcessorE2E as well and add add_cloud_metadata to both the processor and receiver collector configurations there? This way we confirm that the cloud metadata was added to ingested events.

@mauri870 Added in c442f8f.

mauri870
mauri870 approved these changes Dec 11, 2025
View reviewed changes
Copy link
Member

@mauri870 mauri870 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for adding the processor check to integration tests.

@andrzej-stencel andrzej-stencel merged commit e1630be into elastic:main Dec 11, 2025
26 checks passed
@andrzej-stencel andrzej-stencel deleted the add-cloud-processor-to-beats-processor branch December 11, 2025 14:16
@github-actions
Copy link
Contributor

github-actions bot commented Dec 11, 2025

@Mergifyio backport 8.19 9.1 9.2

@mergify
Copy link
Contributor

mergify bot commented Dec 11, 2025 •
edited
Loading

backport 8.19 9.1 9.2

✅ Backports have been created

Details

mergify bot pushed a commit that referenced this pull request Dec 11, 2025
@andrzej-stencel @mergify
feat: add add_cloud_metadata processor to OTel Beat processor (#48020)
68b3860 
* feat: add `add_cloud_metadata` processor to OTel Beat processor

* test: improve coverage in integration test

(cherry picked from commit e1630be)

# Conflicts:
#	x-pack/filebeat/tests/integration/otel_test.go
mergify bot pushed a commit that referenced this pull request Dec 11, 2025
@andrzej-stencel @mergify
feat: add add_cloud_metadata processor to OTel Beat processor (#48020)
00fcd73 
* feat: add `add_cloud_metadata` processor to OTel Beat processor

* test: improve coverage in integration test

(cherry picked from commit e1630be)

# Conflicts:
#	x-pack/filebeat/tests/integration/otel_test.go
mergify bot pushed a commit that referenced this pull request Dec 11, 2025
@andrzej-stencel @mergify
feat: add add_cloud_metadata processor to OTel Beat processor (#48020)
72406ed 
* feat: add `add_cloud_metadata` processor to OTel Beat processor

* test: improve coverage in integration test

(cherry picked from commit e1630be)

# Conflicts:
#	x-pack/filebeat/tests/integration/otel_test.go
This was referenced Dec 11, 2025
Merged
Merged
andrzej-stencel added a commit that referenced this pull request Dec 12, 2025
@mergify @andrzej-stencel
[9.2](backport #48020) feat: add add_cloud_metadata processor to OT…
0db9bed 
…el Beat processor (#48043)

* feat: add `add_cloud_metadata` processor to OTel Beat processor (#48020)

* feat: add `add_cloud_metadata` processor to OTel Beat processor

* test: improve coverage in integration test

(cherry picked from commit e1630be)

# Conflicts:
#	x-pack/filebeat/tests/integration/otel_test.go

* fix conflicts

---------

Co-authored-by: Andrzej Stencel <[email protected]>
andrzej-stencel added a commit that referenced this pull request Dec 12, 2025
@mergify @andrzej-stencel
[8.19](backport #48020) feat: add add_cloud_metadata processor to O…
1e1484b 
…Tel Beat processor (#48041)

* feat: add `add_cloud_metadata` processor to OTel Beat processor (#48020)

* feat: add `add_cloud_metadata` processor to OTel Beat processor

* test: improve coverage in integration test

(cherry picked from commit e1630be)

# Conflicts:
#	x-pack/filebeat/tests/integration/otel_test.go

* fix conflicts

---------

Co-authored-by: Andrzej Stencel <[email protected]>
andrzej-stencel added a commit that referenced this pull request Dec 15, 2025
@mergify @andrzej-stencel
[9.1](backport #48020) feat: add add_cloud_metadata processor to OT…
fa9e782 
…el Beat processor (#48042)

* feat: add `add_cloud_metadata` processor to OTel Beat processor (#48020)

* feat: add `add_cloud_metadata` processor to OTel Beat processor

* test: improve coverage in integration test

(cherry picked from commit e1630be)

# Conflicts:
#	x-pack/filebeat/tests/integration/otel_test.go

* fix conflicts

---------

Co-authored-by: Andrzej Stencel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mauri870 mauri870 mauri870 approved these changes

@leehinman leehinman leehinman approved these changes

@AndersonQ AndersonQ Awaiting requested review from AndersonQ AndersonQ is a code owner automatically assigned from elastic/elastic-agent-data-plane

@mauiroma mauiroma Awaiting requested review from mauiroma

Assignees

@andrzej-stencel andrzej-stencel

Labels

backport-active-all Automated backport with mergify to all the active branches enhancement release-note:skip The PR should be ignored when processing the changelog skip-changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@andrzej-stencel @elasticmachine @mauri870 @leehinman