apache airflow client connections are failing to connect to pgbouncer sporadically

[BalaMuralibi]

We are facing occassional errors while airflow workers and scheduler connecting to pgbouncer. We have tried using external postgres server and bitnami postgres available from airflow helm chart, in both cases we are having challeneges.

We do not see any connection drops between pgbouncer and postgres, the problem is limited to airflow pods connecting to pgbouncer. Our airflow is hosted on kubernetes env, please advise on potential problem and fix.

Dags producing following error

((psycopg2.OperationalError) connection to server at "airflow-pgbouncer.airflow.svc.cluster.local" (172.xx.xx.xx), port 6543 failed: Connection refused Is the server running on that host and accepting TCP/IP connections?

In corelation to dag error, we observe following errors in pgbouncer logs

closing because: client unexpected eof (age=132s)

pgbouncer.ini

[databases]
airflow = host=xxxxxxxxxxxxxxxxxxxxx.postgres.database.azure.com dbname=airflow port=5432 pool_size=200

[pgbouncer]
pool_mode = transaction
listen_port = 6543
listen_addr = *
auth_type = md5
auth_file = /etc/pgbouncer/users.txt
stats_users = airflow_user
ignore_startup_parameters = extra_float_digits
max_client_conn = 2000
verbose = 0
log_disconnections = 1
log_connections = 1
admin_users = airflow_user
server_tls_sslmode = prefer
server_tls_ciphers = normal
log_stats = 0
log_pooler_errors = 1
server_idle_timeout = 300
server_connect_timeout = 15