Skip to content

turms-service failed to connect to turms-gateway #1617

@yueyangw

Description

@yueyangw

First, it is found that the number of online turms-admin is always 0, and the query log finds that turms-service will try to reconnect to the turms-gateway every 15s, but all fail.

turms-service log:

2026-05-07 02:53:36.260  INFO S gyqjqleb                     parallel-3        i.t.s.c.i.c.s.c.ConnectionService : [Client] Connecting to the member: {id=rfvbpcqr, host=turms-gateway, port=7610}. Retry times: 0
2026-05-07 02:53:36.260  INFO S gyqjqleb                     turms-node-connection-client-io-4-4        i.t.s.c.i.c.s.c.ConnectionService : [Client] Connected to the member: {id=rfvbpcqr, host=turms-gateway, port=7610}
2026-05-07 02:53:36.261  INFO S gyqjqleb                     turms-node-connection-client-io-4-4        i.t.s.c.i.c.s.c.ConnectionService : [Client] Sending an opening handshake request to the member: {id=rfvbpcqr, host=turms-gateway, port=7610}
2026-05-07 02:53:36.261 ERROR S gyqjqleb                     turms-node-connection-client-io-4-4        i.t.s.c.i.c.s.c.ConnectionService : [Client] Failed to complete the opening handshake with the member: {id=rfvbpcqr, host=turms-gateway, port=7610}. Closing connection to reconnect
java.lang.RuntimeException: Failure code: 2
	at im.turms.server.common.infra.cluster.service.connection.ConnectionService.lambda$connectMemberUntilSucceedOrRemoved0$1(ConnectionService.java:304)
	at reactor.core.publisher.LambdaMonoSubscriber.onNext(LambdaMonoSubscriber.java:171)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.MonoMetrics$MetricsSubscriber.onNext(MonoMetrics.java:125)
	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:181)
	at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2834)
	at reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
	at reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
	at im.turms.server.common.infra.cluster.service.rpc.RpcEndpoint.resolveRequest(RpcEndpoint.java:138)
	at im.turms.server.common.infra.cluster.service.rpc.RpcEndpoint.handleResponse(RpcEndpoint.java:126)
	at im.turms.server.common.infra.cluster.service.rpc.RpcService$1.onResponseReceived(RpcService.java:235)
	at im.turms.server.common.infra.cluster.service.rpc.RpcService$1.onDataReceived(RpcService.java:145)
	at im.turms.server.common.infra.cluster.service.connection.ConnectionService.lambda$onMemberConnectionAdded$0(ConnectionService.java:490)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:185)
	at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:377)
	at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:444)
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:115)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:361)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at reactor.netty.channel.AbstractChannelMetricsHandler.channelRead(AbstractChannelMetricsHandler.java:148)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(unknown)

2026-05-07 02:53:36.261  INFO S gyqjqleb                     turms-node-connection-client-io-4-4        i.t.s.c.i.c.s.c.ConnectionService : [Client] The connection to the member {id=rfvbpcqr, host=turms-gateway, port=7610} has been closed
2026-05-07 02:53:36.261  INFO S gyqjqleb                     turms-node-connection-client-io-4-4        i.t.s.c.i.c.s.c.ConnectionService : [Client] Try to reconnect the member {id=rfvbpcqr, host=turms-gateway, port=7610} after 15000 millis
2026-05-07 02:53:45.000  INFO S gyqjqleb                     turms-task-manager            i.t.s.d.o.s.StatisticsService : The current online users number is: 0

turms-gateway log:

2026-05-07 02:53:36.261  INFO G rfvbpcqr                     turms-node-connection-server-worker-4-2        i.t.s.c.i.c.s.c.ConnectionService : [Server] Connected to the member
2026-05-07 02:53:36.261  WARN G rfvbpcqr                     turms-node-connection-server-worker-4-2        i.t.s.c.i.c.s.c.ConnectionService : [Server] The connection to an unknown member has been closed unexpectedly
2026-05-07 02:53:36.261  INFO G rfvbpcqr                     turms-node-connection-server-worker-4-2        i.t.s.c.i.c.s.c.ConnectionService : [Server] Stop to connect the member because the local node is server

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions