Timeout errors to LaunchDarkly with Redis Sentinel enabled

[andrewwyee]

Describe the bug
I am running an ld-relay service from the ld-relay prebuilt docker image. When I start up the service with my SDK key configured via env, the service works fine, but when I point it towards a redis sentinel host, I get a the Timeout encountered waiting for LaunchDarkly client initialization warning in the docker logs.

I am able to set key values into sentinel via the Predis php client. I am unsure why the redis sentinel affects the connection to launch darkly

To reproduce

1. Start the ld-relay docker image (v6) with environment variables defined for:

LD_ENV_*={SDK_KEY}
USE_REDIS=1
REDIS_TLS=true
REDIS_HOST={SENTINEL_HOST}
REDIS_PORT={SENTINEL_PORT}
CACHE_TTL=30s
LOG_LEVEL=debug

2. Tail the docker logs

Expected behavior
LD-relay should successfully connect to sentinel and set the environment's feature flag keys.

Logs

2020/11/30 21:27:18.424679 INFO: Starting LaunchDarkly relay version 6.1.0 with configuration from environment variables
2020/11/30 21:27:18.424962 INFO: Using Redis feature store: rediss://REDACTED_HOST:REDACTED_PORT with prefix:
2020/11/30 21:27:18.426017 INFO: [env: ...1387] Starting LaunchDarkly client 5.0.2
2020/11/30 21:27:18.426028 INFO: RedisDataStore: Using URL: rediss://REDACTED_HOST:REDACTED_PORT
2020/11/30 21:27:18.426283 INFO: [env: ...1387] Starting LaunchDarkly streaming connection
2020/11/30 21:27:18.426296 INFO: [env: ...1387] Waiting up to 10000 milliseconds for LaunchDarkly client to start...
2020/11/30 21:27:18.426314 INFO: [env: ...1387] Connecting to LaunchDarkly stream
2020/11/30 21:27:18.427054 DEBUG: [env: ...1387] Sending diagnostic event: {"kind":"diagnostic-init","id":{"diagnosticId":"67c9d367-e916-409d-a7b8-ada0928a2fed","sdkKeySuffix":"**"},"creationDate":1606771638426,"sdk":{"name":"go-server-sdk","version":"5.0.2"},"configuration":{"startWaitMillis":10000,"connectTimeoutMillis":3000,"customBaseURI":false,"diagnosticRecordingIntervalMillis":900000,"usingRelayDaemon":false,"userKeysFlushIntervalMillis":300000,"customEventsURI":false,"socketTimeoutMillis":3000,"eventsCapacity":10000,"eventsFlushIntervalMillis":5000,"allAttributesPrivate":false,"usingProxy":false,"streamingDisabled":false,"customStreamURI":false,"reconnectTimeMillis":1000,"dataStoreType":"custom","inlineUsersInEvents":false,"userKeysCapacity":1000},"platform":{"name":"Go","goVersion":"go1.15.2","osName":"Linux","osArch":"amd64"}}
2020/11/30 21:27:18.427154 INFO: Starting server listening on port 8030
2020/11/30 21:27:19.154240 DEBUG: [env: ...1387] Received all feature flags
2020/11/30 21:27:28.465941 WARN: [env: ...1387] Timeout encountered waiting for LaunchDarkly client initialization
2020/11/30 21:27:28.469060 ERROR: Error initializing LaunchDarkly client for "REDACTED": timeout encountered waiting for LaunchDarkly client initialization

Relay version
https://hub.docker.com/layers/launchdarkly/ld-relay/v6/images/sha256-2a3bdc8d8adf774a1631d7f13bacac5a8e9e89cdb2ce84f662104a3bb4be11fa?context=explore

[andrewwyee]

I was able to get it working by connecting to a redis host directly. Is it possible that ld-relay just doesn't support sentinel redis connections?

[hroederld]

Hi @andrewwyee

Relay does not support Redis Sentinel. We use https://github.com/gomodule/redigo for Redis which would require an additional library to specifically handle Sentinel.

[andrewwyee]

@hroederld Thanks.

Do you think it would be possible to build in sentinel support in ld-relay? I was under the impression that it was supported as the php-sdk allows you to provide an existing predis instance, which I had configured to point to sentinel

[eli-darkly]

@andrewwyee We're looking into switching to https://github.com/go-redis/redis which does support Sentinel, but we haven't tested this yet.

The PHP SDK isn't relevant to this. Each of our SDKs that has some type of Redis support is relying on third-party libraries to handle the Redis connection; we can't guarantee that all of those libraries support all of the same Redis configurations.

[andrewwyee]

Thanks @eli-darkly.

Is there an estimation as to when you guys would make the decision to utilize go-redis/redis or not?

Also do you have recommendations for redis HA support in the meantime with ld-relay? I'm aware that the ld-relay docs recommend tinkering with the CACHE_TTL option, however the LD PHP sdk will solely read through redis, so if redis is down we essentially cannot evaluate any keys.

[joe-udwin-lrn]

This is also a blocker for us purchasing LD. @eli-darkly Any update when this might be implemented?

[hroederld]

Hi @Joe-U-Questionmark I don't think we have any updates about sentinel support but as to HA reads:

You can configure Redis replication without Relay being aware (1). Predis supports configuring a leader and multiple followers for reads without sentinel (2). You can pass a configured Predis client to the SDK bypassing the standard options(3).

1. https://redis.io/topics/replication
2. https://github.com/predis/predis#replication
3. https://github.com/launchdarkly/php-server-sdk/blob/master/src/LaunchDarkly/Integrations/PHPRedis.php#L30.

I have not tested this configuration but I think this would provide HA reads if the leader node goes down, without any modification to the SDK or Relay.

[joe-udwin-lrn]

@hroederld

Thanks for the reply and confirming that you think that the PHP SDK can work with a Sentinel.
However, this thread is discussing the lack of sentinel support in the LD Relay which is a Go application. Do you have an update on when Sentinel support in the Relay proxy might be implemented?

[eli-darkly]

@Joe-U-Questionmark I believe @hroederld's reply was actually meant for @andrewwyee, who had specifically mentioned the PHP SDK.

[eli-darkly]

@Joe-U-Questionmark And I realize that that didn't answer your question, but on the question of time frame I have to defer to management who determines priorities; I've brought this to their attention. I just wanted to avoid confusion on the PHP point.

[bwoskow-ld]

Hi all,

I work with @eli-darkly and @hroederld and am partially responsible for prioritizing the SDK roadmap. While we can't commit to any particular timelines at this time, we are aware of the feature request for adding Redis Sentinel support to Relay and will keep you posted on any relevant updates we have.

Cheers,
@bwoskow-ld

[joe-udwin-lrn]

@bwoskow-ld are we talking about this year? If I found a Go developer would you accept a PR?

[eli-darkly]

@Joe-U-Questionmark This can't (or shouldn't) be done with a single PR to the Relay Proxy code itself. The Redis integration is not in this repo, it's an add-on to the Go SDK, and as we described in other comments above, what was needed was for us to write a new one that uses a different Redis client. That's not hard to do and we had already mostly done it, but a large number of other SDK team priorities intervened. The obstacle hasn't been the lack of a Go developer but just the development pipeline in general, which wouldn't be speeded up by having to review an external PR for work we had already started. I'm not sure if "are we talking about this year" was meant as a serious question, but you can see that we haven't abandoned development on Relay Proxy or the Go SDK as we fairly recently released major version rewrites of both.

[eli-darkly]

@Joe-U-Questionmark Another thing that was delaying the release of a new Go SDK Redis integration is that the point of it was to support both clustered Redis and Redis Sentinel, but those things are not as straightforward to test against as a simple Redis instance. Constructing automated tests for this functionality is a somewhat larger task than actually writing the integration, and we try to avoid relying on manual tests.