filebeat kafka input: add group_instance_id for static group membership

[AndersonQ]

Proposed commit message

filebeat kafka input: add group_instance_id for static group membership

The Kafka input now has a `group_instance_id` option wiring it to Consumer.Group.InstanceId (KIP-345): a member that rejoins within 
`session_timeout` keeps its partitions instead of rebalancing.
Static membership needs Kafka `version` >= 2.3.0, so the input
fails fast with a clear error when it is set on the default 2.1.0.
Unset by default; existing configs are unaffected.

Assisted-By: Claude Code

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None by default — the option is opt-in and unset out of the box; existing
configurations are unchanged. When enabled, operators should be aware:

• version must be >= 2.3.0; otherwise the input fails fast with a clear
  error.
• Each consumer instance sharing a group_id must use a unique
  group_instance_id.
• With static membership, a crashed/restarted member's partitions are not
  reassigned until session_timeout expires — that is the intended trade-off;
  tune session_timeout accordingly.

How to test this PR locally

Unit tests:

cd filebeat
go test -race -run TestNewSaramaConfig ./input/kafka/

Friendly version gate (no broker needed) — configure the input with
group_instance_id set but version left at the default and confirm startup
fails with group_instance_id requires 'version' >= 2.3.0 ....

End-to-end (optional, needs a broker >= 2.3.0): run two Filebeat instances that
share a group_id, each with a distinct group_instance_id and version: "2.3.0";
restart one instance and confirm the group does not rebalance within
session_timeout (no partition reassignment in the broker logs).

Related issues

• Closes [filebeat][kafka input] Expose Sarama static group membership (group_instance_id, KIP-345) #51768

Use cases

Multi-replica, containerized Filebeat consuming a high-throughput Kafka topic.
Setting a stable per-instance group_instance_id lets rolling restarts complete
without cascading rebalances, keeping consumption steady.

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @AndersonQ? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/filebeat/filebeat-input-kafka.md

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[Copilot]

Pull request overview

Adds support for Kafka static group membership (KIP-345) to the Filebeat Kafka input by exposing a new group_instance_id setting and wiring it through to Sarama, including a version gate to fail fast when used with Kafka protocol versions older than 2.3.0.

Changes:

• Add group_instance_id to the Kafka input configuration and propagate it to sarama.Config.Consumer.Group.InstanceId.
• Enforce a minimum Kafka protocol version of >= 2.3.0 when group_instance_id is set, returning a clear error otherwise.
• Update reference configs, documentation, tests, and add a changelog fragment for the new option.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
x-pack/filebeat/filebeat.reference.yml	Documents the new group_instance_id option in the x-pack Filebeat reference config.
filebeat/input/kafka/config.go	Adds GroupInstanceID config wiring and an early version gate for static membership support.
filebeat/input/kafka/config_test.go	Adds unit tests for default behavior, successful propagation, version gating, and invalid IDs.
filebeat/filebeat.reference.yml	Documents the new group_instance_id option in the OSS Filebeat reference config.
filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl	Updates the reference config template so generated configs include group_instance_id docs.
docs/reference/filebeat/filebeat-input-kafka.md	Adds a new docs section describing group_instance_id, requirements, and usage.
changelog/fragments/1783416326-filebeat-kafka-group-instance-id.yaml	Adds a changelog fragment announcing the new Kafka input option.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[AndersonQ]

@orestisfl, @khushijain21, when you have some time, please review this PR.

[AndersonQ]

@orestisfl, @khushijain21 when you have some time, could you review this? thanks!

[khushijain21]

Looks straightforward to me!

[github-actions]

TL;DR

Build 49267 has two Filebeat Python integration failures, and neither is attributable to this PR. The Kafka module case timed out waiting for its Elasticsearch index with no Beat output captured; the system-auth case processed a different SSH journal event than the checked-in golden file. Retry the affected jobs first.

Remediation

• Retry both Filebeat Python integration jobs for build 49267.
• If the Kafka failure reproduces, collect the Beat output and Elasticsearch/service logs; the current log only shows the 20-second index wait expiring.
• If the system-auth failure reproduces, regenerate or reconcile filebeat/module/system/auth/test/debian-12.journal-expected.json only after confirming the Debian journal fixture/image is stable; do not change the Kafka implementation for this failure.

Investigation details

Root Cause

• Kafka module: tests/system/test_modules.py:295-297 waits for the index for each module fixture. The failing case was Test::test_fileset_file_145_kafka on controller-2.0.0.log; the wait expired after 20 seconds and the harness reported no Beat output file. This is insufficient evidence of a code defect and is isolated from the PR changes.
• System auth: tests/system/test_modules.py:403-406 compares each parsed event with its golden event. Test::test_fileset_file_018_system received Accepted password for vagrant from 192.168.42.119 port 55310 ssh2, while the golden event expects Accepted publickey for vagrant from 10.0.2.2 port 48274 ssh2: ED25519 SHA256:k1kjhwoH/H3w31MbGOIGd7qxrkSQJnoAN0eYJVHDmmI; it also omitted system.auth.ssh.signature. The differing SSH source/method/credentials indicate fixture or test-environment content drift, not a Kafka input change.

The PR diff changes only filebeat/input/kafka/config.go, Kafka config tests, Kafka documentation, and generated Kafka references. It does not modify filebeat/tests/system/test_modules.py, the Kafka/system module fixtures, or system-auth parsing code.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/49267
• Jobs: Filebeat: Python Integration Tests; Filebeat: Log input as Filestream Python Integration Tests
• Kafka log: Timeout waiting for condition 'indices present for /opt/buildkite-agent/builds/bk-agent-prod-gcp-1783929894995518090/elastic/filebeat/filebeat/module/kafka/log/test/controller-2.0.0.log' and Waited 20 seconds; 1 failed, 307 passed, 4 skipped.
• System-auth log: tests/system/test_modules.py:405 with four DeepDiff differences; 1 failed, 178 passed, 32 warnings.

Verification

• Not run locally: these are Docker/service-backed Filebeat Python integration tests, and Docker-in-Docker is unavailable in this environment.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[vishaangelova]

LGTM, thanks!

[github-actions]

@Mergifyio backport 9.5 9.4 9.3

[mergify]

backport 9.5 9.4 9.3

❌ No backport have been created

Details

• Backport to branch 9.5 failed

Git reported the following error:

To https://github.com/elastic/beats
 ! [rejected]            mergify/bp/9.5/pr-51772 -> mergify/bp/9.5/pr-51772 (fetch first)
error: failed to push some refs to 'https://github.com/elastic/beats'
hint: Updates were rejected because the remote contains work that you do not
hint: have locally. This is usually caused by another repository pushing to
hint: the same ref. If you want to integrate the remote changes, use
hint: 'git pull' before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

• #51886 [9.4](backport #51772) filebeat kafka input: add group_instance_id for static group membership has been created for branch 9.4
• #51887 [9.3](backport #51772) filebeat kafka input: add group_instance_id for static group membership has been created for branch 9.3