migrate pgsql installation from rpm to container-based

[archanaserver]

This PR transitions the pgsql installation from an RPM-based installation to a containerized based.

[archanaserver]

idk what is wrong with image, it is failing everytime reading manifest in registry.access.redhat.io/rhel9/postgresql-16:latest?

[archanaserver]

also i do not like doing this way, should i go with creating a separate role for postgresql and do the containerization work there? would it be good? any thoughts?

[ehelms]

idk what is wrong with image, it is failing everytime reading manifest in registry.access.redhat.io/rhel9/postgresql-16:latest?

This is an official Red Hat image locked behind subscriptions, only ubi is available for wide usage but I think we've realized it does not come in flavors like the rhel images.

[ehelms]

also i do not like doing this way, should i go with creating a separate role for postgresql and do the containerization work there? would it be good? any thoughts?

100% -- give it a role like all the other services

[archanaserver]

@ehelms how can we re run the workflow?

edit: got it!

[ehelms]

With Candlepin, the thing to usually check is the candlepin.log file:

vi /var/log/candlepin/candlepin.log

And the output I see is:

2025-02-06 13:59:51,400 [thread=main] [=, org=, csid=] ERROR org.candlepin.database.DatabaseConnectionManager - Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

[ekohl]

Per https://www.postgresql.org/docs/current/sql-createdatabase.html you can set the owner on creation

Suggested change

	CREATE DATABASE {{ db.name }};
	ALTER DATABASE {{ db.name }} OWNER TO {{ db.owner }};
	CREATE DATABASE {{ db.name }} OWNER {{ db.owner }};

[ekohl]

I was debating using createuser but reading the documentation I wonder how to properly deal with passing in the password.

Then my other thought was whether if it's better to create the users in a single query:

Suggested change

	{% for user in postgresql_users %}
	psql -v ON_ERROR_STOP=1 --username "postgres" <<-EOSQL
	CREATE USER {{ user.name }} WITH PASSWORD '{{ user.password }}';
	EOSQL
	{% endfor %}
	psql -v ON_ERROR_STOP=1 --username "postgres" <<-EOSQL
	{% for user in postgresql_users %}
	CREATE USER {{ user.name }} WITH PASSWORD '{{ user.password }}';
	{% endfor %}
	EOSQL

And if you take that latter step to the extreme: should there just be a single script that creates the users and the databases.

[ekohl]

These files will be world readable, meaning any user on the system can read the admin password. This should be supplied via a secret.

Having said that: why do we need an admin password? https://github.com/sclorg/postgresql-container/tree/master/13#postgresql-admin-account says you can still use local connections. Or is that insecure because we use the host network?

[archanaserver]

These files will be world readable, meaning any user on the system can read the admin password. This should be supplied via a secret.

That's right! To address this i've moved the password storage to a dedicated secrets file and restrict file permission to 0600 so only container process can now access it. But I'm trying to do this with Podman secrets.

Having said that: why do we need an admin password? https://github.com/sclorg/postgresql-container/tree/master/13#postgresql-admin-account says you can still use local connections. Or is that insecure because we use the host network?

While the container allows local connections without a password here by default, but host networking exposes PostgreSQL to all network interfaces I believe(not just localhost). So without a password, anyone on the network could access the database with postgres privileges. And admin password is that is gonna adds a layer of security here, even if local connections are allowed. If we switch to a non-host network later, maybe the password isn't needed, but for now, it's safer.

[ekohl]

By using init you don't guarantee the password is correct. Perhaps not relevant now, but for the long term: do we need to provide a mechanism to rotate passwords?

Another thing I'm worried about: this will contain plain text passwords in the container. Will that be a security risk?

By the default the container already contains code to create a single user (POSTGRESQL_USER, POSTGRESQL_PASSWORD, POSTGRESQL_DATABASE) which could be provided via secrets. It makes me wonder what would be a good pattern. Should we have a PostgreSQL container for every database?

[archanaserver]

By using init you don't guarantee the password is correct. Perhaps not relevant now, but for the long term: do we need to provide a mechanism to rotate passwords?

It only works for initial step for now and isn't idempotent. I suppose we can add separate tasks to manage users/passwords or maybe we have Ansible modules to handle this, not sure. For now, yes we need that.
https://docs.ansible.com/ansible/latest/collections/community/postgresql/postgresql_user_module.html

Another thing I'm worried about: this will contain plain text passwords in the container. Will that be a security risk?

After looking at it that's valid point. Here the init script run once during container initialization, but the file is in postgresql-init/ could expose secrets. So I restricted now dir permission to owner access but better would be do it with Podman secrets or file based, so that's another to-do.

By the default the container already contains code to create a single user (POSTGRESQL_USER, POSTGRESQL_PASSWORD, POSTGRESQL_DATABASE) which could be provided via secrets. It makes me wonder what would be a good pattern. Should we have a PostgreSQL container for every database?

Can't think of much but If we want strict isolation then we should adopt multiple containers, each with its own secrets and we gonna definitely need a private podman network in that case. Or I guess starting with a single container would be good but with a private network so we can revisit it later for multi-container isolation?

[evgeni]

You can omit the when here, if the array is empty, the loop will not be executed :)

[evgeni]

You can omit the when here, if the array is empty, the loop will not be executed :)

[evgeni]

If I am reading https://github.com/sclorg/postgresql-container/blob/master/13/root/usr/share/container-scripts/postgresql/common.sh#L228-L240 correctly, we don't need to configure hba in the container case, as the container is already configured as "everyone can connect"

[evgeni]

You dropped the hba editing, so I think this test can be removed (and then the re import too)

[archanaserver]

Done!

[evgeni]

You created a secret for that above, so let's use it

Suggested change

	env:
	POSTGRESQL_ADMIN_PASSWORD: "{{ postgresql_admin_password }}"
	secrets:
	- 'postgresql_admin_password,target=POSTGRESQL_ADMIN_PASSWORD,type=env'

[evgeni]

This can be removed now (and so can the setting of it in deploy.yaml)

[ehelms]

Great to see this working! How much data do we want on how upgrades work (#66) before we merge this change?

[evgeni]

I would see those as separate problems (= no data required to merge this).

It seems the container does support running pg_upgrade if it starts with existing data and POSTGRESQL_UPGRADE variable was set:

• https://github.com/sclorg/postgresql-container/blob/e65d040374930106672d88a26b275ee02e56ab37/13/root/usr/bin/run-postgresql#L27
• https://github.com/sclorg/postgresql-container/blob/e65d040374930106672d88a26b275ee02e56ab37/13/root/usr/share/container-scripts/postgresql/common.sh#L408

So we might be rather lucky here and be able to orchestrate updates in a very similar way we do on base rhel

[ehelms]

I'll do a final review of this PR then. @evgeni could you add those upgrade notes to #66 ?

[ehelms]

@archanaserver Do you know if this container setup satisfies #106 ?

[archanaserver]

@archanaserver Do you know if this container setup satisfies #106 ?

No it doesn't, currently container setup by default uses md5. So I can take look into adding explici settings for scram-sha-256.

[ehelms]

The SCRAM update can be a follow up.