Random connection closed before response was received in connector executor

[loicgreffier]

Reproduced on localhost using the demo docker-compose.

During connector synchronization, when collecting connectors from a Kafka Connect instance, Ns4Kafka randomly encounters the following error:

22:59:26.563 [connect-nioEventLoopGroup-4-3] ERROR c.m.n.s.e.ConnectorAsyncExecutor - Error during connectors synchronization for Kafka cluster local and Kafka Connect local: Client 'kafka-connect': Connection closed before response was received.

The stack trace is:

io.micronaut.http.client.exceptions.ResponseClosedException: Client 'kafka-connect': Connection closed before response was received
	at io.micronaut.http.client.netty.Http1ResponseHandler$ReaderState.channelInactive(Http1ResponseHandler.java:114)
	at io.micronaut.http.client.netty.Http1ResponseHandler.channelInactive(Http1ResponseHandler.java:85)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:250)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
	at io.netty.handler.codec.http.HttpContentDecoder.channelInactive(HttpContentDecoder.java:241)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:250)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:416)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:412)

This error is triggered at the following line:

ns4kafka/src/main/java/com/michelin/ns4kafka/service/executor/ConnectorAsyncExecutor.java

Line 166 in 29c556c

log.error(

When this happens, Kafka Connect sends a FIN request instead of responding normally:

139851	1552.898391	127.0.0.1	127.0.0.1	HTTP	187	GET /connectors?expand=info&expand=status HTTP/1.1 
139852	1552.898430	127.0.0.1	127.0.0.1	TCP	44	8083 → 52884 [ACK] Seq=732 Ack=787 Win=64512 Len=0
139853	1552.899288	127.0.0.1	127.0.0.1	TCP	44	8083 → 52884 [FIN, ACK] Seq=732 Ack=787 Win=64512 Len=0

A normal communication is like this:

137373	1522.897797	127.0.0.1	127.0.0.1	HTTP	187	GET /connectors?expand=info&expand=status HTTP/1.1 
137374	1522.897840	127.0.0.1	127.0.0.1	TCP	44	8083 → 52884 [ACK] Seq=623 Ack=644 Win=64768 Len=0
137375	1522.899693	127.0.0.1	127.0.0.1	HTTP/JSON	153	HTTP/1.1 200 OK , JSON (application/json)
137376	1522.899741	127.0.0.1	127.0.0.1	TCP	44	52884 → 8083 [ACK] Seq=644 Ack=732 Win=64768 Len=0

In the Kafka Connect logs, the call is not logged at all:

[2025-12-13 22:51:56,562] INFO 172.18.0.1 - - [13/Dec/2025:22:51:56 +0000] "GET /connectors?expand=info&expand=status HTTP/1.1" 200 2 "-" "-" 1 (org.apache.kafka.connect.runtime.rest.RestServer)

[2025-12-13 22:52:10,380] INFO 172.18.0.7 - - [13/Dec/2025:22:52:10 +0000] "GET /connectors HTTP/1.1" 200 2 "-" "armeria/1.32.5" 0 (org.apache.kafka.connect.runtime.rest.RestServer)

[2025-12-13 22:52:12,868] INFO [AdminClient clientId=compose-connect-group-shared-admin] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)

[2025-12-13 22:52:27,948] INFO 172.18.0.7 - - [13/Dec/2025:22:52:27 +0000] "GET /connectors HTTP/1.1" 200 2 "-" "armeria/1.32.5" 2 (org.apache.kafka.connect.runtime.rest.RestServer)

[2025-12-13 22:52:49,564] INFO 172.18.0.1 - - [13/Dec/2025:22:52:49 +0000] "GET / HTTP/1.1" 200 94 "-" "-" 0 (org.apache.kafka.connect.runtime.rest.RestServer)

[2025-12-13 22:52:51,849] INFO 172.18.0.7 - - [13/Dec/2025:22:52:51 +0000] "GET /connectors HTTP/1.1" 200 2 "-" "armeria/1.32.5" 1 (org.apache.kafka.connect.runtime.rest.RestServer)

[2025-12-13 22:52:56,564] INFO 172.18.0.1 - - [13/Dec/2025:22:52:56 +0000] "GET /connectors?expand=info&expand=status HTTP/1.1" 200 2 "-" "-" 1 (org.apache.kafka.connect.runtime.rest.RestServer)

[2025-12-13 22:53:10,079] INFO 172.18.0.7 - - [13/Dec/2025:22:53:10 +0000] "GET /connectors HTTP/1.1" 200 2 "-" "armeria/1.32.5" 1 (org.apache.kafka.connect.runtime.rest.RestServer)

[2025-12-13 22:53:26,564] INFO 172.18.0.1 - - [13/Dec/2025:22:53:26 +0000] "GET /connectors?expand=info&expand=status HTTP/1.1" 200 2 "-" "-" 1 (org.apache.kafka.connect.runtime.rest.RestServer)

In the logs above, a GET /connectors?expand=info&expand=status call is missing at 22:52:26.

Each time this happens, there appears to be a change in the source port (on the Ns4Kafka side):

488843	5797.922238	127.0.0.1	127.0.0.1	TCP	45	[TCP Keep-Alive] 8083 → 53861 [ACK] Seq=420 Ack=394 Win=65024 Len=1
488844	5797.922364	127.0.0.1	127.0.0.1	TCP	56	[TCP Keep-Alive ACK] 53861 → 8083 [ACK] Seq=394 Ack=421 Win=65024 Len=0 SLE=420 SRE=421
490120	5812.910034	127.0.0.1	127.0.0.1	HTTP	187	GET /connectors?expand=info&expand=status HTTP/1.1 
490121	5812.910076	127.0.0.1	127.0.0.1	TCP	44	8083 → 53861 [ACK] Seq=421 Ack=537 Win=64768 Len=0
490122	5812.910580	127.0.0.1	127.0.0.1	TCP	44	8083 → 53861 [FIN, ACK] Seq=421 Ack=537 Win=64768 Len=0
490123	5812.910610	127.0.0.1	127.0.0.1	TCP	44	53861 → 8083 [ACK] Seq=537 Ack=422 Win=65024 Len=0
490124	5812.910748	127.0.0.1	127.0.0.1	TCP	44	53861 → 8083 [FIN, ACK] Seq=537 Ack=422 Win=65024 Len=0
490125	5812.910783	127.0.0.1	127.0.0.1	TCP	44	8083 → 53861 [ACK] Seq=422 Ack=538 Win=64768 Len=0
491706	5831.205939	::1	::1	TCP	76	61890 → 8083 [SYN] Seq=0 Win=65535 Len=0 MSS=65475 WS=256 SACK_PERM
491707	5831.206009	::1	::1	TCP	76	8083 → 61890 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=65475 WS=256 SACK_PERM
491708	5831.206053	::1	::1	TCP	64	61890 → 8083 [ACK] Seq=1 Ack=1 Win=65280 Len=0
491709	5831.206641	::1	::1	HTTP	322	GET /connectors?expand=info&expand=status HTTP/1.1 
491710	5831.206669	::1	::1	TCP	64	8083 → 61890 [ACK] Seq=1 Ack=259 Win=65024 Len=0
491711	5831.209995	::1	::1	HTTP/JSON	173	HTTP/1.1 200 OK , JSON (application/json)
491712	5831.210036	::1	::1	TCP	64	61890 → 8083 [ACK] Seq=259 Ack=110 Win=65280 Len=0

In the example above, Kafka Connect responds on port 53861. Subsequent requests come from port 61890.

The issue seems to be that Ns4Kafka sometimes reuses a connection while it is being closed by Kafka Connect, resulting in a FIN request and a ResponseClosedException.

[loicgreffier]

Two solutions seem to do the trick:

1. Setting the following configuration:

micronaut:
  http:
    services:
      kafka-connect:
        connection-pool-idle-timeout: "20s"

This ensures that Ns4Kafka expires connections to Kafka Connect before Kafka Connect does (it appears to do so at around 30 seconds).

2. Lower the connector synchronization interval to less than 30 seconds:

Flux.interval(Duration.ofSeconds(12), Duration.ofSeconds(20))

This way, when the next synchronization is triggered (originally at +30s after the previous one), it will not encounter a closing connection.

Both solutions separately did not trigger any errors over multiple hours.