Is it possible to apply retry strategy?

[gigi]

The context

Apply the retry, backoff strategy per RPC call

The question

Sometimes we get an error using your library as gRPC client and Armeria as gRPC server. We saw just failed unexpected logs failed: Connection refused.

gRPC client Armeria implementation has the ability to configure retries for injected client https://line.github.io/armeria/client-retry.html

gRPC has experimental proposal https://github.com/grpc/proposal/blob/master/A6-client-retries.md

Stacktraces and logs

Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: ADDRHERE/IPHERE:50051
	Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
	at io.grpc.netty.shaded.io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)`

Additional information

So, is it possible to achieve retry behavior using your library? Thank you

[ST-DDT]

Yes, that is theoretically possible using a GrpcChannelConfigurer and similar configurers.

However, I couldn't find any useful documentation on the retry in grpc-java, so I'm not sure what else needs to be done.

@Bean
public GrpcChannelConfigurer retryChannelConfigurer() {
    return (channelBuilder, name) -> {
        channelBuilder
                    .enableRetry()
                    .maxRetryAttempts(42);
    };
}

This code only enables the retry subsystem, but it doesn't actually retry according to : https://stackoverflow.com/questions/55428539/grpc-java-setting-enableretry-and-maxretryattempt-is-not-retrying . Do you have good source-code/documentation for retry logic in grpc-java itself?
The inject-able stub classes are final, so there is nothing I could do to implement the retry on a stub level.

[gigi]

Sorry that I did not reply for a while)
Unfortunately, I don't have any good reference except we can find by links above :)

Some references
The class that holds retry logic in GrpcChannelConfigurer
https://github.com/grpc/grpc-java/blob/master/core/src/main/java/io/grpc/internal/AbstractManagedChannelImplBuilder.java#L138

Armeria wraps grpc stub somehow and uses client with retry logic
https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/client/retry/RetryingRpcClient.java#L97

Some discussion about service config. I can try to configure NettyChannelBuilder later
grpc/grpc-java#5724

[freezingWu]

the retry feature , i think it's easy to reuse the spring-retry module.

[helloworlde]

the retry feature , i think it's easy to reuse the spring-retry module.

@freezingWu Could you please provide a sample of spring-retry?

[freezingWu]

hi bro,
here comes the spring-boot retry module sample ,pls track it from : https://my.oschina.net/wangxincj/blog/827221

[trustin]

@gigi Does Armeria's gRPC client work OK in this case? Out of curiosity, would you mind sharing your reasoning behind using grpc-spring-boot-starter's gRPC client while using Armeria on the other side?

[ST-DDT]

@trustin This library does not provide its own client. It just manages the injection of grpc-java clients/stubs. (I assume they are the same as those used by armeria, just without any additional wrapping).

There are many ways to implement retry logic. This question is about using grpc-java's own logic to avoid the problems with the recurring unavailability of the Armeria server.

[helloworlde]

Actually, grpc-java support retry from 1.20.0, so there need a entrance config to open it. Please reference RetryingHelloWorldClient

[ST-DDT]

Thanks for the info.
I'll add a corresponding entry to the documentation for easy reference.