Skip to content

[9.3](backport #51772) filebeat kafka input: add group_instance_id for static group membership#51887

Merged
AndersonQ merged 1 commit into
9.3from
mergify/bp/9.3/pr-51772
Jul 13, 2026
Merged

[9.3](backport #51772) filebeat kafka input: add group_instance_id for static group membership#51887
AndersonQ merged 1 commit into
9.3from
mergify/bp/9.3/pr-51772

Conversation

@mergify

@mergify mergify Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

filebeat kafka input: add group_instance_id for static group membership

The Kafka input now has a `group_instance_id` option wiring it to Consumer.Group.InstanceId (KIP-345): a member that rejoins within 
`session_timeout` keeps its partitions instead of rebalancing.
Static membership needs Kafka `version` >= 2.3.0, so the input
fails fast with a clear error when it is set on the default 2.1.0.
Unset by default; existing configs are unaffected.

Assisted-By: Claude Code

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works.
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None by default — the option is opt-in and unset out of the box; existing
configurations are unchanged. When enabled, operators should be aware:

  • version must be >= 2.3.0; otherwise the input fails fast with a clear
    error.
  • Each consumer instance sharing a group_id must use a unique
    group_instance_id.
  • With static membership, a crashed/restarted member's partitions are not
    reassigned until session_timeout expires — that is the intended trade-off;
    tune session_timeout accordingly.

How to test this PR locally

Unit tests:

cd filebeat
go test -race -run TestNewSaramaConfig ./input/kafka/

Friendly version gate (no broker needed) — configure the input with
group_instance_id set but version left at the default and confirm startup
fails with group_instance_id requires 'version' >= 2.3.0 ....

End-to-end (optional, needs a broker >= 2.3.0): run two Filebeat instances that
share a group_id, each with a distinct group_instance_id and version: "2.3.0";
restart one instance and confirm the group does not rebalance within
session_timeout (no partition reassignment in the broker logs).

Related issues

Use cases

Multi-replica, containerized Filebeat consuming a high-throughput Kafka topic.
Setting a stable per-instance group_instance_id lets rolling restarts complete
without cascading rebalances, keeping consumption steady.


This is an automatic backport of pull request #51772 done by Mergify.

…ip (#51772)

* filebeat kafka input: add group_instance_id for static group membership

The Kafka input now has a `group_instance_id` option wiring it to Consumer.Group.InstanceId (KIP-345): a member that rejoins within
`session_timeout` keeps its partitions instead of rebalancing.
Static membership needs Kafka `version` >= 2.3.0, so the input
fails fast with a clear error when it is set on the default 2.1.0.
Unset by default; existing configs are unaffected.

Assisted-By: Claude Code
(cherry picked from commit 6811ebc)
@mergify mergify Bot requested review from a team as code owners July 13, 2026 10:10
@mergify mergify Bot added the backport label Jul 13, 2026
@mergify mergify Bot requested review from andrzej-stencel and orestisfl and removed request for a team July 13, 2026 10:10
@github-actions

Copy link
Copy Markdown
Contributor

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
  • /test : Run the Buildkite pipeline.

@botelastic botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 13, 2026
@github-actions github-actions Bot added enhancement Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Jul 13, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 13, 2026
@AndersonQ AndersonQ enabled auto-merge (squash) July 13, 2026 10:30
@AndersonQ AndersonQ merged commit a24a058 into 9.3 Jul 13, 2026
49 checks passed
@AndersonQ AndersonQ deleted the mergify/bp/9.3/pr-51772 branch July 13, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants