Controller and server not starting after upgrade from 4.0.3 to 4.1.2

[martijnremmen]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

I upgraded to the following images

• quay.io/argoproj/workflow-controller:v4.1.2
• quay.io/argoproj/argocli:v4.1.2

Now I get the following error from the controller and server: level=ERROR msg="failed to create initial database session: failed to create upper/db session: pq: no pg_hba.conf entry for host \"172.28.6.51\", user \"argo_test\", database \"argo_test\", no encryption (28000)"

It seems very similar to #16707 but im using postges instead of mysql.

Version(s)

v4.1.2

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

persistence: |
  archive: true
  postgresql:
    host: my-azure-postgresql-server.postgres.database.azure.com
    database: argo_test
    tableName: argo_test
    userNameSecret:
      name: argo-postgres-config
      key: user
    passwordSecret:
      name: argo-postgres-config
      key: password
    sslMode: require

Logs from the workflow controller

level=ERROR msg="failed to create initial database session: failed to create upper/db session: pq: no pg_hba.conf entry for host \"172.28.6.51\", user \"argo_test\", database \"argo_test\", no encryption (28000)"

Logs from in your workflow's wait container

-

[martijnremmen]

@Joibel Is it possible this issue is similar to #16707?

[Joibel]

@Joibel Is it possible this issue is similar to #16707?

It looks it. That is also about postgres and not mysql - I am not sure why you mention mysql.

You can prove it by testing :latest. It has a fix.

I intend to do a release next week when I return from holiday

[martijnremmen]

@Joibel Is it possible this issue is similar to #16707?

It looks it. That is also about postgres and not mysql - I am not sure why you mention mysql.

You can prove it by testing :latest. It has a fix.

I intend to do a release next week when I return from holiday

I tried with the current :latest (quay.io/argoproj/argocli@sha256:531c97befef68d9a115c9e543f6e446973c2af531177fb292d7451c3b2e057cb) but i am still getting the same error:

failed to create initial database session: failed to create upper/db session: pq: no pg_hba.conf entry for host \"172.28.6.54\", user \"argo_test\", database \"argo_test\", no encryption (28000)"