[8.19] (backport #8031) Allow using beats receivers for self-monitoring #8102
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
* Handle nil case in monitoring config parsing * Allow using otel runtime for self-monitoring # Conflicts: # internal/pkg/agent/application/monitoring/v1_monitor.go # Conflicts: # internal/pkg/otel/configtranslate/otelconfig.go * Modify e2e test * Make the monitoring e2e test more restrictive * Revert "Handle nil case in monitoring config parsing" This reverts commit bb11a0f. * Check receiver statuses in e2e test * Send data from beats processes and receivers to different namespaces * Check all component statuses in E2E test * Fix typo Co-authored-by: Khushi Jain <[email protected]> --------- Co-authored-by: Khushi Jain <[email protected]> (cherry picked from commit a31d56f)
mergify
bot
requested review from
kaanyalti and
andrzej-stencel
and removed request for
a team
github-actions
bot
added
enhancement
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
|
Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane) |
swiatekm
approved these changes
May 7, 2025
|
💛 Build succeeded, but was flaky
Failed CI Steps
Historycc @swiatekm |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds the ability to use beats receivers for agent self-monitoring. To do so, we add a new configuration key to
agent.monitoringnamed_runtime_experimental- identical to how you can currently switch inputs to the Otel runtime.In terms of implementation, the changes are very straightforward. In the monitoring injection manager, we set the runtime manager for inputs we add, if it's set in the monitoring configuration.
Most of this PR's code changes lie in tests, and more specifically in the
TestAgentMonitoringE2E test. This test compares the data collected by agent self-monitoring using beats processes to an equivalent Otel configuration of beats receivers in Hybrid mode. Instead of doing that, we can now just changeagent.monitoring._runtime_experimental, so the test becomes much simpler conceptually.I have simplified some of the test logic, but I haven't yet made it compare metrics. This should be doable now, but we have another PR (#8009 ) in-flight doing it, so I held off.
Why is it important?
We want to be able to use beats receivers for agent self-monitoring.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added an entry in./changelog/fragmentsusing the changelog toolHow to test this PR locally
Build the agent locally and use the following configuration:
Looking at Kibana dashboards for the agent integration can prove the data is actually being ingested. You can verify that beats receivers are being used for self-monitoring by looking at their CPU usage - it should be 0.
Related issues
This is an automatic backport of pull request #8031 done by [Mergify](https://mergify.com).