Allow using beats receivers for self-monitoring #8031

swiatekm · 2025-04-29T16:31:21Z

What does this PR do?

Adds the ability to use beats receivers for agent self-monitoring. To do so, we add a new configuration key to agent.monitoring named _runtime_experimental - identical to how you can currently switch inputs to the Otel runtime.

In terms of implementation, the changes are very straightforward. In the monitoring injection manager, we set the runtime manager for inputs we add, if it's set in the monitoring configuration.

Most of this PR's code changes lie in tests, and more specifically in the TestAgentMonitoring E2E test. This test compares the data collected by agent self-monitoring using beats processes to an equivalent Otel configuration of beats receivers in Hybrid mode. Instead of doing that, we can now just change agent.monitoring._runtime_experimental, so the test becomes much simpler conceptually.

I have simplified some of the test logic, but I haven't yet made it compare metrics. This should be doable now, but we have another PR (#8009 ) in-flight doing it, so I held off.

Why is it important?

We want to be able to use beats receivers for agent self-monitoring.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~- [ ] I have added an entry in ./changelog/fragments using the changelog tool~~
I have added an integration test or an E2E test

How to test this PR locally

Build the agent locally and use the following configuration:

outputs:
  default:
    type: elasticsearch
    hosts: [...]
    username: elastic
    password: "..."

inputs: []

agent:
  monitoring:
    metrics: true
    logs: true
    _runtime_experimental: otel

Looking at Kibana dashboards for the agent integration can prove the data is actually being ingested. You can verify that beats receivers are being used for self-monitoring by looking at their CPU usage - it should be 0.

Related issues

Closes https://github.com/elastic/ingest-dev/issues/3517

mergify · 2025-04-29T16:32:05Z

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

swiatekm · 2025-04-30T08:22:48Z

The test failures are due to the beats update. I'm going to do that in a separate PR for clarity: #8041.

mergify · 2025-04-30T10:13:41Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/self-monitoring-otel-runtime upstream/feat/self-monitoring-otel-runtime
git merge upstream/main
git push upstream feat/self-monitoring-otel-runtime

elasticmachine · 2025-04-30T15:35:02Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

testing/integration/beat_receivers_test.go

internal/pkg/core/monitoring/config/config.go

internal/pkg/agent/application/monitoring/v1_monitor.go

testing/integration/beat_receivers_test.go

# Conflicts: # internal/pkg/agent/application/monitoring/v1_monitor.go # Conflicts: # internal/pkg/otel/configtranslate/otelconfig.go

testing/integration/beat_receivers_test.go

cmacknz · 2025-05-05T20:37:12Z

This LGTM and testing locally I see the monitoring receivers running. It would be nice if we can make sure that only the beat receivers are used for monitoring, right now we have at least the beat receivers are used for monitoring in the tests.

sudo elastic-development-agent status --output=full
┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: 9293b312-f874-4866-bb1c-ebc21244c75c
   │  ├─ version: 9.1.0
   │  └─ commit: 3b2fe0010f4075f5dd47fad96a4b2c1dc5a97f52
   ├─ filestream-default
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '82070'
   │  ├─ filestream-default
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ filestream-default-your-input-id
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ system/metrics-default
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '82069'
   │  ├─ system/metrics-default
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ system/metrics-default-unique-system-metrics-input
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ pipeline:logs/_agent-component/beat/metrics-monitoring
   │  ├─ status: StatusOK
   │  ├─ exporter:elasticsearch/_agent-component/monitoring
   │  │  └─ status: StatusOK
   │  └─ receiver:metricbeatreceiver/_agent-component/beat/metrics-monitoring
   │     └─ status: StatusOK
   ├─ pipeline:logs/_agent-component/filestream-monitoring
   │  ├─ status: StatusOK
   │  ├─ exporter:elasticsearch/_agent-component/monitoring
   │  │  └─ status: StatusOK
   │  └─ receiver:filebeatreceiver/_agent-component/filestream-monitoring
   │     └─ status: StatusOK
   └─ pipeline:logs/_agent-component/http/metrics-monitoring
      ├─ status: StatusOK
      ├─ exporter:elasticsearch/_agent-component/monitoring
      │  └─ status: StatusOK
      └─ receiver:metricbeatreceiver/_agent-component/http/metrics-monitoring
         └─ status: StatusOK

elasticmachine · 2025-05-06T09:38:52Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

testing/integration/beat_receivers_test.go

Co-authored-by: Khushi Jain <[email protected]>

elastic-sonarqube · 2025-05-06T12:51:30Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-05-06T14:30:33Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 0f4bc74

Failed CI Steps

History

💛 Build #20913 was flaky 3b2fe00
💔 Build #20910 failed 183ff06
💚 Build #20774 succeeded 2d9c182
💔 Build #20767 failed 4ccd165
💔 Build #20761 failed 5f11db7
💔 Build #20726 failed 4b94fd4

cc @swiatekm

github-actions · 2025-05-07T09:24:56Z

@Mergifyio backport 9.0

mergify · 2025-05-07T09:25:07Z

backport 9.0

✅ Backports have been created

#8101 [9.0] (backport #8031) Allow using beats receivers for self-monitoring has been created for branch 9.0

* Handle nil case in monitoring config parsing * Allow using otel runtime for self-monitoring # Conflicts: # internal/pkg/agent/application/monitoring/v1_monitor.go # Conflicts: # internal/pkg/otel/configtranslate/otelconfig.go * Modify e2e test * Make the monitoring e2e test more restrictive * Revert "Handle nil case in monitoring config parsing" This reverts commit bb11a0f. * Check receiver statuses in e2e test * Send data from beats processes and receivers to different namespaces * Check all component statuses in E2E test * Fix typo Co-authored-by: Khushi Jain <[email protected]> --------- Co-authored-by: Khushi Jain <[email protected]> (cherry picked from commit a31d56f)

* Handle nil case in monitoring config parsing * Allow using otel runtime for self-monitoring # Conflicts: # internal/pkg/agent/application/monitoring/v1_monitor.go # Conflicts: # internal/pkg/otel/configtranslate/otelconfig.go * Modify e2e test * Make the monitoring e2e test more restrictive * Revert "Handle nil case in monitoring config parsing" This reverts commit bb11a0f. * Check receiver statuses in e2e test * Send data from beats processes and receivers to different namespaces * Check all component statuses in E2E test * Fix typo --------- (cherry picked from commit a31d56f) Co-authored-by: Mikołaj Świątek <[email protected]> Co-authored-by: Khushi Jain <[email protected]>

* upstream/main: Guard against `nil` pointer dereference (elastic#8107) Generate NOTICE.txt with only modules used by binaries (elastic#8053) Retry enrollment requests when an error is returned, add enrollment timeout (elastic#8056) Changelog for 8.17.6 version (elastic#8062) (elastic#8106) [main][Automation] Update versions (elastic#8098) Allow using beats receivers for self-monitoring (elastic#8031) Adding new configuration setting: `agent.upgrade.rollback.window` (elastic#8065) [Integration Testing] Allow tests to declare themselves as needing a FIPS environment (elastic#8083) fix(agentless): overcome SIGPIPE in agentless promotion pipeline (elastic#8094) ksm autosharing integration configuration update (elastic#8086)

mergify bot assigned swiatekm Apr 29, 2025

swiatekm added skip-changelog backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-8.19 Automated backport to the 8.19 branch enhancement New feature or request labels Apr 29, 2025

swiatekm force-pushed the feat/self-monitoring-otel-runtime branch from 4363785 to 4b94fd4 Compare April 29, 2025 17:15

swiatekm force-pushed the feat/self-monitoring-otel-runtime branch 3 times, most recently from 4ccd165 to 2d9c182 Compare April 30, 2025 13:13

swiatekm requested review from cmacknz, mauri870, leehinman and khushijain21 April 30, 2025 14:18

swiatekm marked this pull request as ready for review April 30, 2025 14:36

swiatekm requested a review from a team as a code owner April 30, 2025 14:36

swiatekm requested a review from ycombinator April 30, 2025 14:36

swiatekm changed the title ~~Allow using the otel runtime for self-monitoring~~ Allow using beats receivers for self-monitoring Apr 30, 2025

pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Apr 30, 2025

ycombinator reviewed May 1, 2025

View reviewed changes

testing/integration/beat_receivers_test.go Show resolved Hide resolved

ycombinator reviewed May 1, 2025

View reviewed changes

internal/pkg/core/monitoring/config/config.go Show resolved Hide resolved

ycombinator reviewed May 1, 2025

View reviewed changes

internal/pkg/agent/application/monitoring/v1_monitor.go Outdated Show resolved Hide resolved

ycombinator reviewed May 1, 2025

View reviewed changes

internal/pkg/agent/application/monitoring/v1_monitor.go Show resolved Hide resolved

cmacknz reviewed May 1, 2025

View reviewed changes

testing/integration/beat_receivers_test.go Outdated Show resolved Hide resolved

cmacknz reviewed May 1, 2025

View reviewed changes

testing/integration/beat_receivers_test.go Show resolved Hide resolved

swiatekm added 2 commits May 5, 2025 16:17

Handle nil case in monitoring config parsing

271a81d

Allow using otel runtime for self-monitoring

3239f89

# Conflicts: # internal/pkg/agent/application/monitoring/v1_monitor.go # Conflicts: # internal/pkg/otel/configtranslate/otelconfig.go

Send data from beats processes and receivers to different namespaces

3b2fe00

swiatekm requested review from ycombinator and cmacknz May 5, 2025 17:05

cmacknz reviewed May 5, 2025

View reviewed changes

testing/integration/beat_receivers_test.go Outdated Show resolved Hide resolved

cmacknz mentioned this pull request May 5, 2025

[otel]: Add e2e test for monitoring metrics in otel mode #8009

Open

8 tasks

pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label May 6, 2025

Check all component statuses in E2E test

62efd67

swiatekm requested a review from cmacknz May 6, 2025 09:42

khushijain21 reviewed May 6, 2025

View reviewed changes

testing/integration/beat_receivers_test.go Outdated Show resolved Hide resolved

Fix typo

0f4bc74

Co-authored-by: Khushi Jain <[email protected]>

swiatekm requested a review from khushijain21 May 6, 2025 11:50

ycombinator approved these changes May 6, 2025

View reviewed changes

khushijain21 approved these changes May 6, 2025

View reviewed changes

swiatekm merged commit a31d56f into main May 7, 2025
12 checks passed

swiatekm deleted the feat/self-monitoring-otel-runtime branch May 7, 2025 09:24

This was referenced May 7, 2025

[9.0] (backport #8031) Allow using beats receivers for self-monitoring #8101

Merged

[8.19] (backport #8031) Allow using beats receivers for self-monitoring #8102

Merged

swiatekm mentioned this pull request May 19, 2025

[testing/integration][otel] move configurations to different files for better readability #8008

Open

8 tasks

leehinman mentioned this pull request May 20, 2025

runtime_experimental: otel does not gather elastic_agent.filebeat_input datastream #8188

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow using beats receivers for self-monitoring #8031

Allow using beats receivers for self-monitoring #8031

Uh oh!

swiatekm commented Apr 29, 2025 •

edited

Loading

Uh oh!

mergify bot commented Apr 29, 2025

Uh oh!

swiatekm commented Apr 30, 2025

Uh oh!

mergify bot commented Apr 30, 2025

Uh oh!

elasticmachine commented Apr 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmacknz commented May 5, 2025 •

edited

Loading

Uh oh!

elasticmachine commented May 6, 2025

Uh oh!

Uh oh!

elastic-sonarqube bot commented May 6, 2025

Uh oh!

elasticmachine commented May 6, 2025

Uh oh!

Uh oh!

github-actions bot commented May 7, 2025

Uh oh!

mergify bot commented May 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

Allow using beats receivers for self-monitoring #8031

Allow using beats receivers for self-monitoring #8031

Uh oh!

Conversation

swiatekm commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Uh oh!

mergify bot commented Apr 29, 2025

Uh oh!

swiatekm commented Apr 30, 2025

Uh oh!

mergify bot commented Apr 30, 2025

Uh oh!

elasticmachine commented Apr 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmacknz commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented May 6, 2025

Uh oh!

Uh oh!

elastic-sonarqube bot commented May 6, 2025

Quality Gate passed

Uh oh!

elasticmachine commented May 6, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

History

Uh oh!

Uh oh!

github-actions bot commented May 7, 2025

Uh oh!

mergify bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

swiatekm commented Apr 29, 2025 •

edited

Loading

cmacknz commented May 5, 2025 •

edited

Loading

mergify bot commented May 7, 2025 •

edited

Loading