KAFKA-19244: Add support for kafka-streams-groups.sh options (reset-offsets) [1/N] #19646

aliehsaeedii · 2025-05-05T23:25:31Z

This PR implements all the options for --reset-offset, (supports
--execute, --dry-run, --export). Includes unit and integration
tests.

--reset-offsets --input-topic String: topics --shift-by Long
--reset-offsets --input-topic String: topics --to-offset Long
--reset-offsets --input-topic String: topics -to-latest
--reset-offsets --input-topic String: topics -to-earliest
--reset-offsets --all-input-topics --shift-by Long
--reset-offsets --all-input-topics --to-offset Long
--reset-offsets --all-input-topics -to-latest
--reset-offsets --all-input-topics -to-earliest
--reset-offsets --all-input-topics -to-earliest
--reset-offsets --from-file String:fileName
--reset-offsets --input-topic String: topics --by-duration String:
duration
--reset-offsets --all-input-topics --by-duration String: duration
--reset-offsets --input-topics <String: topics> --to-datetime String:
datetime
--reset-offsets --all-input-topics --to-datetime String: datetime
--reset-offsets --input-topics <String: topics> -to-current
--reset-offsets --all-input-topics -to-current

Reviewers: Lucas Brutschy [email protected], Bill Bejeck
[email protected]

bbejeck

Thanks for the PR @aliehsaeedii I made a pass over the non-testing code, I'll do more in a follow up review

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommand.java

bbejeck · 2025-05-07T17:20:06Z

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommand.java

+                        throw new IllegalArgumentException("Cannot shift offset for partition " + topicPartition + " since there is no current committed offset");
+                    }
+
+                    return currentOffset.offset() + shiftBy;


Do we want to do any validation on what the acceptable values for shiftBy?

I assume any long value should be accepted. Good point. I'll add some tests.

Seems shiftby 150 and -150 are already existing.

lucasbru

Hey @aliehsaeedii ! Thanks for the PR. Mostly looking good to me, left a few comments

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommand.java

lucasbru

Thanks for the PR! I left some comments. These are still relatively high-level things.

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommandOptions.java

tools/src/test/java/org/apache/kafka/tools/streams/StreamsGroupCommandTest.java

lucasbru · 2025-06-04T11:31:35Z

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommand.java

+                                    List<String> internalTopics = retrieveInternalTopics(List.of(groupId)).get(groupId);
+                                    if (internalTopics != null && !internalTopics.isEmpty()) {
+                                        try {
+                                            adminClient.deleteTopics(internalTopics).all().get();


Why do we delete internal topics here?

As of #19646 (comment)

lucasbru

One more comment @aliehsaeedii

tools/src/test/java/org/apache/kafka/tools/streams/StreamsGroupCommandTest.java

lucasbru

Hi, I left some commments on the documentation. Please make sure that --all-topics and --topics are not present in more places.

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommandOptions.java

lucasbru · 2025-06-10T08:42:11Z

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommandOptions.java

+    private static final String INPUT_TOPIC_DOC = "The input topic whose streams group information should be deleted or topic that should be included in the reset offset process. " +
+        "In `reset-offsets` case, partitions can be specified using this format: `topic1:0,1,2`, where 0,1,2 are the partition to be included in the process. " +
+        "Reset-offsets also supports multiple topic inputs.";
+    private static final String ALL_INPUT_TOPICS_DOC = "Consider all topics assigned to a group in the `reset-offsets` process.";


delete-offsets as well? Maybe use the format Supported operations: delete-offsets, reset-offsets. as in the other options. Also, topics aren't really assigned to a group. How about:

Consider all source topics used in the topology of the group.

Also, topics aren't really assigned to a group.

I meant associated:|

tools/src/main/java/org/apache/kafka/tools/streams/OffsetsUtils.java

tools/src/main/java/org/apache/kafka/tools/OffsetsUtils.java

clients/src/main/java/org/apache/kafka/clients/admin/ListStreamsGroupOffsetsSpec.java

Copilot

Pull Request Overview

This PR introduces full support for the --reset-offsets command in kafka-streams-groups.sh, refactors offset handling into OffsetsUtils, and updates related tests and utility classes.

Add new reset-offsets options (e.g., --to-offset, --by-duration, --export, etc.) in StreamsGroupCommandOptions.
Refactor ConsumerGroupCommand to delegate offset logic to OffsetsUtils.
Update tests in StreamsGroupCommandTest and DescribeStreamsGroupTest to cover new behavior and require explicit --group for describe; make CSV and spec methods public.

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tools/src/main/java/.../StreamsGroupCommandOptions.java	Add `reset-offsets` CLI options, docs, and validation logic.
tools/src/main/java/.../ConsumerGroupCommand.java	Replace inline offset logic with `OffsetsUtils` delegation.
tools/src/test/java/.../StreamsGroupCommandTest.java	Update tests to use shared `ADMIN_CLIENT` mock and new helpers.
tools/src/test/java/.../DescribeStreamsGroupTest.java	Require `--group` for all describe invocations in tests.
tools/src/main/java/.../consumer/group/CsvUtils.java	Make readerFor/writerFor methods public.
tools/src/main/java/.../consumer/group/ConsumerGroupCommand.java	(Reviewed above)
clients/src/main/java/.../ListStreamsGroupOffsetsSpec.java	Make topicPartitions methods public.

Comments suppressed due to low confidence (2)

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommandOptions.java:227

Add a similar check to ensure exportOpt is only allowed when resetOffsetsOpt is present (e.g., only --reset-offsets may be used with --export).

if ((options.has(dryRunOpt) || options.has(executeOpt)) && !options.has(resetOffsetsOpt))

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommandOptions.java:233

Include deleteOffsetsOpt in the allowed set for inputTopicOpt, since deleting offsets also uses --input-topic. E.g., minus(allStreamsGroupLevelOpts, resetOffsetsOpt, deleteOffsetsOpt).

CommandLineUtils.checkInvalidArgs(parser, options, inputTopicOpt, minus(allStreamsGroupLevelOpts, resetOffsetsOpt));

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommandOptions.java

lucasbru

Hi @aliehsaeedii ! I left some comments

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommand.java

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommandOptions.java

lucasbru

Hi @aliehsaeedii ! I left some comments

chia7712 · 2025-07-12T16:54:35Z

tools/src/main/java/org/apache/kafka/tools/OffsetsUtils.java

+
+    public static class Unknown implements LogOffsetResult { }
+
+    public static class Ignore implements LogOffsetResult { }


Excuse me, what is the purpose of Ignore? will it be used in next PR?

aliehsaeedii added 6 commits April 28, 2025 11:39

initial impl

78c61b2

impl reset offset and all tests

1acb651

Merge branch 'trunk' into offsets

53f953a

refactor and adding tests

42bfde4

revert irrelavent changes

204cd03

clean-up

52d709b

github-actions bot added triage PRs from the community tools clients labels May 5, 2025

mjsax added streams KIP-1071 PRs related to KIP-1071 ci-approved and removed triage PRs from the community labels May 5, 2025

bbejeck reviewed May 7, 2025

View reviewed changes

aliehsaeedii changed the title ~~KAFKA-19244: Add support for kafka-streams-groups.sh options (delete group, offset-related APIs) [1/2]~~ KAFKA-19244: Add support for kafka-streams-groups.sh options (delete group, offset-related APIs) [1/N] May 19, 2025

aliehsaeedii mentioned this pull request May 19, 2025

KAFKA-19244: Add support for kafka-streams-groups.sh options (delete all groups) [2/N] #19758

Merged

aliehsaeedii added 2 commits May 26, 2025 12:48

address reviews

557a76c

correct comments/msgs

ab7d858

lucasbru changed the title ~~KAFKA-19244: Add support for kafka-streams-groups.sh options (delete group, offset-related APIs) [1/N]~~ KAFKA-19244: Add support for kafka-streams-groups.sh options (offset-related APIs) [1/N] May 27, 2025

lucasbru reviewed May 27, 2025

View reviewed changes

aliehsaeedii added 5 commits June 2, 2025 12:44

reset offsets of src topics

78aed57

backup

a5fe8c8

address reviews

12c9f07

fix utests

35d319e

throw exception

cd88721

lucasbru reviewed Jun 4, 2025

View reviewed changes

aliehsaeedii changed the title ~~KAFKA-19244: Add support for kafka-streams-groups.sh options (offset-related APIs) [1/N]~~ KAFKA-19244: Add support for kafka-streams-groups.sh options (reset-offsets) [1/N] Jun 4, 2025

aliehsaeedii added 3 commits June 4, 2025 18:11

topic to input-topic

125f33a

merge

e7baf58

rebase

626b0df

aliehsaeedii added 4 commits June 4, 2025 19:16

fix cs issue

b9be3eb

backup

03240c5

add input to options + rebase

b002400

refactore

abf8acb

lucasbru reviewed Jun 6, 2025

View reviewed changes

tools/src/test/java/org/apache/kafka/tools/streams/StreamsGroupCommandTest.java Outdated Show resolved Hide resolved

ListGroupsResult should be package private

024d3fe

lucasbru reviewed Jun 10, 2025

View reviewed changes

tools/src/main/java/org/apache/kafka/tools/streams/OffsetsUtils.java Outdated Show resolved Hide resolved

aliehsaeedii added 4 commits June 11, 2025 19:53

complete refactoring

def165f

merge

2c7fb00

fix cs bug

cac2732

fix doc

a6e1925

aliehsaeedii commented Jun 11, 2025

View reviewed changes

tools/src/main/java/org/apache/kafka/tools/OffsetsUtils.java Show resolved Hide resolved

aliehsaeedii commented Jun 11, 2025

View reviewed changes

clients/src/main/java/org/apache/kafka/clients/admin/ListStreamsGroupOffsetsSpec.java Show resolved Hide resolved

keep options package private

a7be157

lucasbru requested a review from Copilot June 12, 2025 14:35

Copilot AI reviewed Jun 12, 2025

View reviewed changes

tools/src/main/java/org/apache/kafka/tools/streams/StreamsGroupCommandOptions.java Outdated Show resolved Hide resolved

aliehsaeedii added 2 commits June 19, 2025 08:32

introduce delete internal topics options

33cf7d3

fix utest

45fb98c

lucasbru reviewed Jun 19, 2025

View reviewed changes

address reviews

73b54b3

lucasbru approved these changes Jun 20, 2025

View reviewed changes

lucasbru merged commit a797d64 into apache:trunk Jun 20, 2025
22 checks passed

chia7712 reviewed Jul 12, 2025

View reviewed changes


		public static class Unknown implements LogOffsetResult { }

		public static class Ignore implements LogOffsetResult { }

KAFKA-19244: Add support for kafka-streams-groups.sh options (reset-offsets) [1/N] #19646

KAFKA-19244: Add support for kafka-streams-groups.sh options (reset-offsets) [1/N] #19646

Uh oh!

Conversation

aliehsaeedii commented May 5, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bbejeck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucasbru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lucasbru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucasbru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lucasbru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

lucasbru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lucasbru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aliehsaeedii commented May 5, 2025 •

edited by github-actions bot

Loading