Make least request available by default (remove experimental environment variable)

[atollena]

Is your feature request related to a problem?

The least request load balancer has been implemented and released behind an environment variable since July 2022 (grpc-java 1.48.0). To my knowledge it has been used in production by spotify for about as long. The last significant change is in https://github.com/grpc/grpc-java/releases/tag/v1.63.0 to fix a race introduced a few versions before.

Describe the solution you'd like

Remove the GRPC_EXPERIMENTAL_ENABLE_LEAST_REQUEST environment variable, so that least request is always available.

Same issue in Go: grpc/grpc-go#8228

[ejona86]

For gRFCs that aren't planning to have cross-language testing, I think the bar is "someone has actually run it and it seems to work," which would be per-language. That doesn't require production usage. I do believe Spotify is likely to have used it, but I'm not sure we ever got confirmation. You actually using it in Go for a year should be well beyond what is necessary to stabilize it in grpc-go.

However, the gRFC didn't actually specify the qualification for "stable." So I'd like to confirm with Doug/Mark, and both of them are out-of-office this week. So I'll talk to them next week about it.

[atollena]

Maybe @yannickepstein can comment regarding Spotify usage?

[yannickepstein]

Sorry for the late response. I was out of office.

Maybe @yannickepstein can comment regarding Spotify usage?

We have been running least request for a few high volume services (around 20, but some with significant load) but we are mostly using our internal load balancing algorithm ELS.
We haven't encountered any bugs with the grpc-java implementation of least request after multiple years of using it.

For the most part, least request performed well for us, but we have run into a few edge cases where tail latencies were elevated by least request compared to our internal LB. I don't think this is particularly problematic but just something to be aware of when using the algorithm.