Add filebeat receiver to otel mode

[leehinman]

What does this PR do?

Adds filebeat receiver as an otel receiver. This is a re-introduction of code that was added in #5672, that had to be reverted because the import of the beats code base was causing issues with command line parsing. The issues in the beats code base have been fixed, so we are re-introducing this change.

Why is it important?

This is the first beat receiver, and this will allow us to test the feasibility of running beats as otel receivers.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

should be 0 user impact of existing configurations

How to test this PR locally

Use something like the following as otel.yml, and run elastic-agent otel --config otel.yml
This will run the beat receiver, which will produce one event.
The event should be written to the path you specify in the file/no_rotation section.

receivers:
  filebeatreceiver:
    filebeat:
      inputs:
        - type: benchmark
          enabled: true
          count: 1
          message: "The quick brown fox jumped over the lazy dog"
    output:
      otelconsumer:
    logging:
      level: info
      selectors:
        - '*'
    path:
      home: /tmp/<state dir>
exporters:
  file/no_rotation:
    path: /tmp/<output file>
service:
  pipelines:
    logs:
      receivers: [filebeatreceiver]
      exporters: [file/no_rotation]

Related issues

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @leehinman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 41277_testing upstream/41277_testing
git merge upstream/main
git push upstream 41277_testing

[mauri870]

@leehinman FYI I edited your PR description to fix the otel config.

[mauri870]

LGTM, tested locally and was able to run the filebeat receiver and confirm the data is properly exported to the file output.

[blakerouse]

Excited to see this back! Looks good.

[jlind23]

The error we got in CI is:

Status: Downloaded newer image for docker.elastic.co/beats-dev/golang-crossbuild:1.22.8-main
  | >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64]
  | # github.com/elastic/elastic-agent
  | /usr/local/go/pkg/tool/linux_amd64/link: running aarch64-linux-gnu-gcc failed: exit status 1
  | /usr/lib/gcc-cross/aarch64-linux-gnu/6/../../../../aarch64-linux-gnu/bin/ld.gold: internal error in maybe_apply_stub, at ../../gold/aarch64.cc:5407
  | collect2: error: ld returned 1 exit status

Which looks similar to - elastic/beats#41270

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 41277_testing upstream/41277_testing
git merge upstream/main
git push upstream 41277_testing

[mauri870]

Triggered another test run. Let's see if the build succeeds this time around.

[mauri870]

@rdner Looks like this is missing an approval from you, would you mind taking a look? Thanks.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[flexitrev]

This is amazing!