[BUG] Connecting to the DB takes a very long time

[stheiven]

Describe the bug
We have noticed that at times connecting to the DB take a very long time (e.g., 100s of milliseconds). I have attached an image of a Honeycomb trace where the telemetry data shows that the connect span takes 403 milliseconds.

Expected behavior
I would expect that the connect takes 10s of milliseconds not 100s,

Environment (please complete the following information, because it helps us investigate better):

• OS: linux
• Version 1.1.1
• DB: Postgresql

[tolgaOzen]

Hi @stheiven , could you provide a bit more information about the environment? Is the Permify instance in the same region as the database?

[stheiven]

@tolgaOzen Yes, they are in the Permify instance is in the same region as the DB. The connect times were reduced once we put a RDS proxy in front of the DB, but we still some spikes where the connect times are 100s of ms. We have increased the max connections on the Permify pods as well to see if this helps (still testing this), but I suspect that the connection spikes will still remain after this change.

[tolgaOzen]

@stheiven , could you please share your config file after hiding confidential fields?

[stheiven]

@tolgaOzen I've added attached manifest for the pod. We had the best results with the default setting of 20 for PERMIFY_DATABASE_MAX_OPEN_CONNECTIONS

pod.txt