[otel]: Add e2e test for monitoring metrics in otel mode

[khushijain21]

What does this PR do?

This PR adds e2e tests for self-monitoring metrics exposed using beatreceivers. It also asserts document equivalency for metrics exposed by normal mode vs otel mode.

Why is it important?

Required to safely transition running elastic-agent in otel mode.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Related issues

• https://github.com/elastic/ingest-dev/issues/4877

[mergify]

This pull request does not have a backport label. Could you fix it @khushijain21? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[cmacknz]

Does this overlap with what #7622 intends to do?

[khushijain21]

Does this overlap with what #7622 intends to do?

#7622 adds test for metric inputs whereas this is for monitoring metrics collected by elastic-agent.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b e2emetrics upstream/e2emetrics
git merge upstream/main
git push upstream e2emetrics

[cmacknz]

Thanks for clarifying!

[cmacknz]

Once #8031 exists, instead of manually reproducing the configurations you could just install agent twice in the tests.

You can see an example of agent running twice in the same test in

elastic-agent/testing/integration/install_test.go

Lines 154 to 165 in 3b2fe00

	out, err := fixture.Install(ctx, &opts)
	if err != nil {
	t.Logf("install output: %s", out)
	require.NoError(t, err)
	}
	// Check that Agent was installed in the custom base path
	topPath := filepath.Join(basePath, "Elastic", "Agent")
	require.NoError(t, installtest.CheckSuccess(ctx, fixture, topPath, &installtest.CheckOpts{Privileged: opts.Privileged}))
	t.Run("check agent package version", testAgentPackageVersion(ctx, fixture, true))
	t.Run("check second agent installs with --namespace", testSecondAgentCanInstall(ctx, fixture, basePath, false, opts))

Then you will have a true side by side comparison with agent generating the configs itself. CC @mauri870 I think this idea probably applies to the test you are working on as well once the same thing is possible for non-monitoring metrics.

Reproducing the configs manually just decoupled you from having to wait for #8031, but once the feature flag to switch to beat receivers for monitoring exists it'll make sure we test with the latest config agent generates itself.

[cmacknz]

The --namespace feature is what is used to implement the --develop support in https://github.com/elastic/elastic-agent?tab=readme-ov-file#development-installations.

[ycombinator]

Looks like #8031 is merged now.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[leehinman]

take a look at:

main...leehinman:elastic-agent:4876_agent_monitoring_tests

I think it has some ideas you can use. Specifically the changes to the rawQuery including the sort, and the use of match_phrase over match, and the initial criteria to find the events. Doing that made the ignoredFields much smaller and the results were consistent.

[leehinman]

I found it was easier to to pick a field name that lead to a unique kind of metric. For example beat.stats.memstats.rss

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b e2emetrics upstream/e2emetrics
git merge upstream/main
git push upstream e2emetrics

[khushijain21]

investigating why memory related stats (such as beat.stats.memory.rss are not available with runtime_experimental:otel mode

[khushijain21]

http/metrics pulls process related metrics from beats and sends it to metrics-elastic_agent.elastic_agent.* index. These metrics are not applicable for beatreceivers - hence dropping these streams for this special case

[elastic-sonarqube]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1e2b700

Failed CI Steps

• x86_64:sudo: default
• x86_64:sudo: upgrade

History

• 💔 Build #21438 failed ea25080
• 💔 Build #21428 failed baf61f5
• 💛 Build #20992 was flaky d329105
• 💔 Build #20929 failed 86065ad
• 💔 Build #20901 failed 5020631

cc @khushijain21