cleanup(otel): move otelbeat packages to x-pack/otel

[mauri870]

Proposed commit message

Move the remaining otelbeat packages to x-pack/otel so that all otel-related code is centralized.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR locally

cd x-pack/filebeat
mage goIntegTest

cd x-pack/metricbeat
mage goIntegTest

Related issues

• Relates otel: update integration tests to use an in-process testing collector #47338
• Relates otel: fix remaining tests to use in process testing collector #47772
• Relates chore(otel): remove otelbeat #47804
• Closes https://github.com/elastic/ingest-dev/issues/6207

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mauri870]

/test

[mauri870]

/test

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

Some intermittent agent lost failures are occurring in CI. It seems to be a widespread issue on Buildkite.

[mauri870]

/test

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[pierrehilbert]

@narph we will need someone from your team to review here please.

[efd6]

SSI file changes LGTM

[mauri870]

Moving to draft until I can fix the conflicts.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rellocate-xpack-otel upstream/rellocate-xpack-otel
git merge upstream/main
git push upstream rellocate-xpack-otel

[mauri870]

/test

[mauri870]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rellocate-xpack-otel upstream/rellocate-xpack-otel
git merge upstream/main
git push upstream rellocate-xpack-otel

[mauri870]

There's something causing CI to break. The x-pack/otel: run check step normally takes less than six minutes, but it's losing the agent around the 19-minute mark:

make -C x-pack/otel check && make check-no-changes
...
go: downloading github.com/Azure/go-autorest/autorest/validation v0.3.1
go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c
go: downloading github.com/kylelemons/godebug v1.1.0
Exited with status -1 (agent lost)

I ran the same command locally and it works, so I need to check which change in this PR caused the issue.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rellocate-xpack-otel upstream/rellocate-xpack-otel
git merge upstream/main
git push upstream rellocate-xpack-otel