[9.1] (backport #10153) Rewrite the beats receiver log test using common tooling

[mergify]

What does this PR do?

Re-enables the integration test for comparing logs between beats processes and beats receivers and rewrites it to use normal integration test tooling, making it simpler and more observable:

• Instead of uninstalling and reinstalling agent with new configuration, we install it once and just let it reload the configuration, verifying that this happened via status.
• Use the normal install process and let agent run as a service instead of running it manually.
• Fetches the logs by running elastic-agent logs instead of capturing agent output directly.
• Relax the status check to only verify what we care about. This test sets an ES host that doesn't exist, so the status will eventually be degraded, but the test doesn't really care about this.

Why is it important?

This test should be enabled and reliable.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Related issues

• Closes [Flaky Test]: TestBeatsReceiverLogs – Error: Condition never satisfied #9890

---

This is an automatic backport of pull request #10153 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 71ece0d has failed:

On branch mergify/bp/9.1/pr-10153
Your branch is up to date with 'origin/9.1'.

You are currently cherry-picking commit 71ece0de9.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   testing/integration/ess/beat_receivers_test.go

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request has not been merged yet. Could you please review and merge it @swiatekm? 🙏

[ebeahan]

@swiatekm CI failed on this backport - can you take a look?

[swiatekm]

Looks like this test wasn't originally ported to this branch, which seems like an oversight. I'll see if there's any obstruction to doing so.

[swiatekm]

There's currently some changes missing in 9.1 whose lack makes this test flaky, so I'll skip backporting it.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/9.1/pr-10153 upstream/mergify/bp/9.1/pr-10153
git merge upstream/9.1
git push upstream mergify/bp/9.1/pr-10153

[mergify]

This pull request has not been merged yet. Could you please review and merge it @swiatekm? 🙏

[swiatekm]

Looks like the windows tests are failing because Vihas' fix for retrying npipes in beats is not yet in this branch. And it looks like #10849 needs to go in first.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 235e442

History

• 💔 Build #29426 failed f4f6063
• 💔 Build #29365 failed 8881b8a
• 💔 Build #28719 failed dfb481d
• 💔 Build #28576 failed 70cf7df
• 💔 Build #28570 failed c94433d

cc @swiatekm