HTTP183: consolidate http configuration between sink and lookup by jonathanlehto · Pull Request #184 · getindata/flink-http-connector

Jonathan Lehto (jonathanlehto) · 2025-10-30T18:26:19Z

Description

consolidate http configuration between sink and lookup

Resolves 183

PR Checklist

Tests added
Changelog updated

Jonathan Lehto (jonathanlehto) · 2025-10-31T15:56:22Z


-            httpPostRequestCallback.call(
-                optResponse.orElse(null), sinkRequestEntry, endpointUrl, headerMap);
+            httpPostRequestCallback.call(optResponse.orElse(null), sinkRequestEntry, endpointUrl, headerMap);


I think it would be helpful to also include the responseChecker in the callback, but that should be done in another pull request. The HttpResponseChecker is critical to properly understand request response information, and could be valuable, especially with logging.

David Radley (davidradl) · 2025-11-04T11:44:37Z

+    }
+
+    public boolean isErrorCode(int httpStatusCode) {
+        return !isTemporalError(httpStatusCode) && !isSuccessful(httpStatusCode);


if we do checks like this can we ensure that the same status code cannot occur in multiple places - with an early config validation.

The HttpResponseChecker does validate that the combination of success and ignore codes are not contained within temporal codes. This happens on creation during the validate call under this function https://github.com/getindata/flink-http-connector/pull/184/files/ea116a6e703edcbdcafe1a3e6b51bad53a4166bd#diff-c35b6c5d289edf15e9e5773c77632b091bc633e825a5da7112fb5d4434e5655cR73. Currently, with the lookup table, you can have ignore codes and success codes overlap, so a check there would potentially break backwards compability. Is that what check you are referring to?

interesting. If there overlap which has precedence? I would prefer that we police that they do not overlap. Thius my break people who have probably incorrectly coded the overlaps.

David Radley (davidradl) · 2025-11-05T13:36:08Z

+
+The sink categorizes HTTP responses into groups:
+- Success codes (`gid.connector.http.sink.success-codes`): Expected successful responses.  `1XX, 2XX, 3XX` are defaults
+- Retry codes (`gid.connector.http.sink.retry-codes`): Transient errors that trigger automatic retries when using `at-least-once` delivery guarantee. `500, 503, 504` are defaults


500s should not be retried - these are internal server errors - not retriable codes. I do not think we should have any defaults.

The 500 default was done to match the lookup table defaults. I'm okay with removing 500, but, ~~should we do the same to lookup as well?~~ (probably outside the bounds of this pull request) I'm not sure if we need the two to match exactly as the use case is a bit different

David Radley (davidradl) · 2025-11-05T13:37:39Z

+- Ignored responses (`gid.connector.http.sink.ignored-response-codes`): Responses whose content is ignored but treated as successful.
+- Error codes: Any response code not classified in the above groups.
+
+Parameters support whitelisting and blacklisting: `2XX,404,!203` means all codes from 200-299, plus 404, except 203.


I suggest not using whitelisting and blacklisting as the language is questionable- I have made this change in the apache version.

Okay, I will switch it to denylist & allowlist, but please let me know if you had another naming convention in my mind!

David Radley (davidradl) · 2025-11-05T13:38:21Z

-| gid.connector.http.sink.error.code.exclude              | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.sink.error.code` list, separated with comma.                                                                                                                      |
+| gid.connector.http.sink.error.code `DEPRECATED`         | optional | List of HTTP status codes that should be treated as errors by HTTP Sink, separated with comma.                                                                                                                                                   |
+| gid.connector.http.sink.error.code.exclude `DEPRECATED` | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.sink.error.code` list, separated with comma.                                                                                                                      |
+| gid.connector.http.sink.success-codes                   | optional | Comma separated http codes considered as success response. Use [1-5]XX for groups and '!' character for excluding. Defaults are `1XX,2XX,3XX`                                                                                                    |


I think the default should be 200, as was the case for the lookup.

1XX,2XX,3XX was done to preserve backwards compatibility to the existing/old configuration. I am okay with just 2XX, but should we place 1XX and 3XX into a default ignore code set?

I see 2XX as the default for lookup success-codes.

David Radley (davidradl) · 2025-11-05T13:39:27Z

+
+public enum ResponseItemStatus {
+    SUCCESS("success"),
+    TEMPORAL("temporal"),


I think RETRIABLE would be better than TEMPORAL.

Okay! I used to TEMPORAL to match the lookup table's language. Eventually, we may want to change it their as well

makes sense

David Radley (davidradl) · 2025-12-04T11:24:56Z

+    public static HttpResponseChecker createHttpResponseChecker(Properties properties) {
+        try {
+            String deprecatedIgnoreExpr = properties.getProperty(
+                    HttpConnectorConfigConstants.HTTP_ERROR_SINK_CODE_WHITE_LIST,


I am thinking we eitehr use the old way or the new way -so we should validate that if we are configured with HTTP_ERROR_SINK_CODE_WHITE_LIST then we do not accept ignore, success or retry codes and do not honour the defaults of them.

And if we have ignore, success or retry codes configured we do not allow HTTP_ERROR_SINK_CODE_WHITE_LIST and its exclude.

David Radley (davidradl) · 2026-02-25T11:16:53Z

Jonathan Lehto (@jonathanlehto) Sorry for the long delay; this and the other 2 PRs are capabilities that we (IBM) would like to get merged. I think the current state of the code is good enough to merge, but to be extra diligent I would like to have this ported to the Apache HTTP connector and have Ferenc review it. Are you ok to bring this code up to date here and port to the Apache HTTP connector - under 3 Apache Flink Jiras. Once it is reviewed in the apache repo we can implement any feedback and merge in both places. If you port the fix these will be recognised Apache Flink contributions. If you are no long wanting to work on this let me know and I will take over when it gets to the top of my priority list.

fyi Grzegorz Kołakowski (@grzegorz8)

Jonathan Lehto (jonathanlehto) · 2026-02-27T18:43:48Z

Jonathan Lehto (@jonathanlehto) Sorry for the long delay; this and the other 2 PRs are capabilities that we (IBM) would like to get merged. I think the current state of the code is good enough to merge, but to be extra diligent I would like to have this ported to the Apache HTTP connector and have Ferenc review it. Are you ok to bring this code up to date here and port to the Apache HTTP connector - under 3 Apache Flink Jiras. Once it is reviewed in the apache repo we can implement any feedback and merge in both places. If you port the fix these will be recognised Apache Flink contributions. If you are no long wanting to work on this let me know and I will take over when it gets to the top of my priority list.

fyi Grzegorz Kołakowski (@grzegorz8)

I can't seem to find the three Jiras, I can see https://issues.apache.org/jira/browse/FLINK-38245. Would you have any specific FLINK ids?

Jonathan Lehto (jonathanlehto) mentioned this pull request Oct 31, 2025

HTTP182: Add retry logic with delivery guarantees, retry all failed requests #185

Open

2 tasks

Jonathan Lehto (jonathanlehto) commented Oct 31, 2025

View reviewed changes

Comment thread src/main/java/com/getindata/connectors/http/internal/status/HttpResponseChecker.java

Jonathan Lehto (jonathanlehto) commented Oct 31, 2025

View reviewed changes

Jonathan Lehto (jonathanlehto) mentioned this pull request Nov 3, 2025

HTTP-182 HTTP sink retries #181

Open

2 tasks

David Radley (davidradl) reviewed Nov 4, 2025

View reviewed changes

David Radley (davidradl) reviewed Nov 5, 2025

View reviewed changes

David Radley (davidradl) reviewed Dec 4, 2025

View reviewed changes

Jonathan Lehto (jonathanlehto) added 3 commits February 27, 2026 13:29

HTTP183: consolidate http configuration between sink and lookup

c6308da

fix readme comment, consolidate tests, fix formatting in tests

c458719

addressing code review comments

d8a31ce

Jonathan Lehto (jonathanlehto) force-pushed the http183 branch from ea116a6 to d8a31ce Compare February 27, 2026 18:30

handle null response

c72210d

Conversation

Jonathan Lehto (jonathanlehto) commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

PR Checklist

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

David Radley (davidradl) Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jonathan Lehto (jonathanlehto) Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

David Radley (davidradl) Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

David Radley (davidradl) commented Feb 25, 2026

Uh oh!

Jonathan Lehto (jonathanlehto) commented Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Jonathan Lehto (jonathanlehto) commented Oct 30, 2025 •

edited

Loading

David Radley (davidradl) Nov 4, 2025 •

edited

Loading

Jonathan Lehto (jonathanlehto) Nov 5, 2025 •

edited

Loading

David Radley (davidradl) Dec 4, 2025 •

edited

Loading