Skip to content

Kafka Connect: Add option to fail connector task after max retries #13268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

dttung2905
Copy link
Contributor

Hi team,

I add a few more configs:

  private static final String COMMIT_MAX_RETRIES_PROP = "iceberg.control.commit.max-retries";
  private static final int COMMIT_MAX_RETRIES_DEFAULT = 3;
  private static final String COMMIT_MIN_RETRY_WAIT_MS_PROP = "iceberg.control.commit.min-retry-wait-ms";
  private static final int COMMIT_MIN_RETRY_WAIT_MS_DEFAULT = 100;
  private static final String COMMIT_MAX_RETRY_WAIT_MS_PROP = "iceberg.control.commit.max-retry-wait-ms";
  private static final int COMMIT_MAX_RETRY_WAIT_MS_DEFAULT = 60_000;
  private static final String COMMIT_TOTAL_RETRY_TIME_MS_PROP = "iceberg.control.commit.total-retry-time-ms";
  private static final int COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT = 300_000; // 5 minutes
  public static final String FAIL_ON_MAX_COMMIT_RETRIES = "fail.on.max.commit.retries";
  public static final boolean FAIL_ON_MAX_COMMIT_RETRIES_DEFAULT = false;

if we set fail.on.max.commit.retries=true, the connector task will fails after some retries. The default value of fail.on.max.commit.retries is false to maintain backward compatibility

Fixes #13117

Signed-off-by: dttung2905 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Kafka-connect : many commit Timeout in a row never fail the connector
1 participant