[9.5](backport #51772) filebeat kafka input: add group_instance_id for static group membership

[mergify]

Proposed commit message

filebeat kafka input: add group_instance_id for static group membership

The Kafka input now has a `group_instance_id` option wiring it to Consumer.Group.InstanceId (KIP-345): a member that rejoins within 
`session_timeout` keeps its partitions instead of rebalancing.
Static membership needs Kafka `version` >= 2.3.0, so the input
fails fast with a clear error when it is set on the default 2.1.0.
Unset by default; existing configs are unaffected.

Assisted-By: Claude Code

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None by default — the option is opt-in and unset out of the box; existing
configurations are unchanged. When enabled, operators should be aware:

• version must be >= 2.3.0; otherwise the input fails fast with a clear
  error.
• Each consumer instance sharing a group_id must use a unique
  group_instance_id.
• With static membership, a crashed/restarted member's partitions are not
  reassigned until session_timeout expires — that is the intended trade-off;
  tune session_timeout accordingly.

How to test this PR locally

Unit tests:

cd filebeat
go test -race -run TestNewSaramaConfig ./input/kafka/

Friendly version gate (no broker needed) — configure the input with
group_instance_id set but version left at the default and confirm startup
fails with group_instance_id requires 'version' >= 2.3.0 ....

End-to-end (optional, needs a broker >= 2.3.0): run two Filebeat instances that
share a group_id, each with a distinct group_instance_id and version: "2.3.0";
restart one instance and confirm the group does not rebalance within
session_timeout (no partition reassignment in the broker logs).

Related issues

• Closes [filebeat][kafka input] Expose Sarama static group membership (group_instance_id, KIP-345) #51768

Use cases

Multi-replica, containerized Filebeat consuming a high-throughput Kafka topic.
Setting a stable per-instance group_instance_id lets rolling restarts complete
without cascading rebalances, keeping consumption steady.

---

This is an automatic backport of pull request #51772 done by Mergify.

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

🔍 Preview links for changed docs

• docs/reference/filebeat/filebeat-input-kafka.md

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[github-actions]

TL;DR

The only failed job is x-pack/filebeat: Go fips140=only Integration Tests, and the captured output identifies TestUDPInputOTelE2E as the failed test artifact. This is unrelated to the Kafka group_instance_id changes in this backport; retry the Buildkite job first.

Remediation

• Retry Buildkite build 49284, specifically beats-xpack-filebeat build 34072.
• If it fails again, inspect the uploaded TEST-go-integration.out.json and TestUDPInputOTelE2E... artifacts for the assertion details; the console log does not include the failing assertion.

Investigation details

Root Cause

The failure is isolated to the UDP OTel end-to-end integration test, not the changed Kafka code. The job artifact list contains x-pack/filebeat/build/integration-tests/TestUDPInputOTelE2E687701074, while the PR changes filebeat/input/kafka/config.go and its unit tests. The test exercises fixed localhost ports 127.0.0.1:9042 and 127.0.0.1:9043 in x-pack/filebeat/tests/integration/otel_tcp_udp_test.go:50-63, then waits up to three minutes for both Elasticsearch documents in x-pack/filebeat/tests/integration/otel_tcp_udp_test.go:262-275.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/49284
• Job/step: x-pack/filebeat: Go fips140=only Integration Tests
• Key log excerpt: DONE 2880 tests, 11 skipped, 1 failure in 1009.707s (buildkite-failures log line 81), followed by Error: failed to execute go: exit status 1 (line 83).
• The artifact upload lists TestUDPInputOTelE2E687701074 (lines 93-119), but the captured console output contains no FAIL, assertion, or test error detail.

Verification

The affected Kafka configuration tests pass locally: go test ./filebeat/input/kafka -run TestNewSaramaConfig -count=1.

Follow-up

If the retry reproduces the failure, use the test artifact output to determine whether the UDP listener, OTel receiver, Elasticsearch document ingestion, or fixed-port setup is responsible before changing code.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[AndersonQ]

/test