[otel] add kafka partitioner extension

[VihasMakwana]

Proposed commit message

This PR adds a new extension for partitioning kakfa records. This extension will work with existing config.

The extension follows upstream RecordPartitionerExtension (See here), which is used to return underlying partition.

Ideally, we should use same logic as https://github.com/elastic/beats/blob/main/libbeat/outputs/kafka/partition.go, but upstream exporter uses https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo and beats uses sarama library and these two differ in terms of interfaces. Hence, the duplication.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR

• Build elastic-agent from this PR and use otel mode for the output.
• Start kafka server using following commands

docker pull apache/kafka:4.2.0
docker run -p 9092:9092 apache/kafka:4.2.0

• Create topic with couple of partitions:

bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092 --partitions 2 --replication-factor 1

• Install agent using following config:

outputs:
  default:
    type: kafka
    hosts: ["localhost:9092"]
    topic: "quickstart-events"
    partition.round_robin:

inputs:
  - type: filestream
    id: quickstart-filestream
    enabled: true
    streams:
      - id: quickstart-filestream-stream
        paths:
          - /path/to/logs

agent.monitoring:
  enabled: false

agent:
  internal:
    runtime:
      default: otel

• Write some sample logs. The logs should be distributed in round robin fashion. You can use following command to view logs arriving on first partition:

bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092 --partition 0

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka-partitioner upstream/kafka-partitioner
git merge upstream/main
git push upstream kafka-partitioner

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @VihasMakwana? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka-partitioner upstream/kafka-partitioner
git merge upstream/main
git push upstream kafka-partitioner

[github-actions]

TL;DR

The failures are mixed: one confirmed CI environment issue (mage version not available), one Windows test cleanup failure in TestLeak, and two FIPS ECH jobs that only expose terminal exit status 126 in the provided logs. Immediate action: fix the Windows test cleanup race and rerun; for the 126 jobs, capture the failing command output earlier in the step (current artifacts are truncated to teardown/epilogue).

Remediation

• Libbeat FIPS unit test (confirmed): ensure the FIPS agent image has the required mage preset (or installs it before mage goUnitTest).
• Auditbeat Windows unit test (confirmed): stabilize x-pack/auditbeat/abreceiver leak test by ensuring all handles to path.home/data/beat.db are closed before temp dir cleanup.
• x-pack/filebeat + x-pack/metricbeat FIPS ECH (inconclusive from provided logs): rerun with full command output retained for .buildkite/scripts/custom_fips_ech_test.sh (the attached logs only show terraform destroy + epilogue + exit status 126).

Investigation details

Root Cause

1. Libbeat FIPS unit tests failed due missing mage preset in the runtime environment (configuration/infrastructure), not a code assertion failure.
2. x-pack/auditbeat Windows tests failed in TestLeak with a file lock on beat.db during temp dir cleanup (test instability / resource cleanup timing).
3. x-pack/filebeat + x-pack/metricbeat FIPS ECH jobs report only generic exit status 126 in available logs; root cause cannot be proven from the provided excerpts.

Relevant code paths:

• .buildkite/scripts/custom_fips_ech_test.sh:19 runs GOEXPERIMENT=systemcrypto SNAPSHOT=true FIPS=true mage build fipsECHTest.
• x-pack/auditbeat/abreceiver/receiver_leak_test.go:27 defines TestLeak.
• x-pack/auditbeat/abreceiver/receiver_leak_test.go:53 sets "path.home": t.TempDir().
• x-pack/auditbeat/abreceiver/receiver_leak_test.go:96 calls rec.Shutdown(t.Context()) before cleanup.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/44639
• Job: :ubuntu: Libbeat: Ubuntu x86_64 Go Unit Tests with fips provider and requirefips build tag
  • /tmp/gh-aw/buildkite-logs/beats-libbeat-ubuntu-libbeat-ubuntu-x86_64-go-unit-tests-with-fips-provide.txt:125
    • No preset version installed for command mage
  • /tmp/gh-aw/buildkite-logs/beats-libbeat-ubuntu-libbeat-ubuntu-x86_64-go-unit-tests-with-fips-provide.txt:131
    • asdf install mage 1.15.0
  • /tmp/gh-aw/buildkite-logs/beats-libbeat-ubuntu-libbeat-ubuntu-x86_64-go-unit-tests-with-fips-provide.txt:142
    • user command error: exit status 126
• Job: :windows: x-pack/auditbeat: Win 2016 Unit Tests
  • /tmp/gh-aw/buildkite-logs/beats-xpack-auditbeat-windows-x-packauditbeat-win-2016-unit-tests.txt:129-130
    • === FAIL: x-pack/auditbeat/abreceiver TestLeak (4.07s)
    • TempDir RemoveAll cleanup: unlinkat ...\data\beat.db: The process cannot access the file because it is being used by another process.
• Jobs: :ubuntu: x-pack/filebeat: FIPS ECH Integration Tests and :ubuntu: x-pack/metricbeat: FIPS ECH Integration Tests
  • Logs currently show only terminal epilogue and user command error: exit status 126 without the failing command output.

Verification

• Not run locally (workflow is read-only and focused on log forensics).

Follow-up

• After applying the Windows test cleanup fix and CI image/preset fix, rerun the four failed jobs.
• If 126 persists for filebeat/metricbeat FIPS ECH, upload full step stdout/stderr before teardown so the failing executable line is captured.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #50226 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [otel] add kafka partitioner extension #50226 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[cmacknz]

Force merging, test failures are unrelated.