Move co-located containers onto shared foreman-core-network by Alleny244 · Pull Request #728 · theforeman/foremanctl

Alleny244 · 2026-08-05T13:53:04Z

Problem Statement

Embedded PostgreSQL defaults to listen_addresses = '*', exposing port 5432 on public interfaces when containers use host networking.

https://redhat.atlassian.net/browse/SAT-48503

Changes

Add a shared Podman bridge network foreman-core-network (10.130.0.0/24) via the foreman_core_network role, created early in deploy.
Move co-located containers off host networking onto that network (PostgreSQL, Valkey, Candlepin, Pulp, Foreman/Dynflow, IOP services).
Reach services by container name on the shared network (e.g. postgresql, valkey, candlepin, iop-core-gateway).
Publish host access on loopback only where needed (e.g. 127.0.0.1:5432, 127.0.0.1:6379, Candlepin/IOP gateway ports) so ports are not exposed on 0.0.0.0 / public interfaces.
Remove the IOP-only iop_network / iop-core-network; IOP uses foreman-core-network.
Update TLS SANs and Smart Proxy URL so Foreman can reach Candlepin and the IOP gateway by container name.
Leave foreman_proxy on host networking (edge/DHCP/DNS/TFTP/REX).
Add/update tests to ensure PostgreSQL (and Valkey) are not published on 0.0.0.0:5432 / [::]:5432 (and equivalent for Valkey).

Tests

./forge test --pytest-args="tests/postgresql_test.py -vv"

Checklist

Tests added/updated
Documentation updated (docs/iop.md)

ehelms · 2026-08-06T17:43:41Z

  ansible.builtin.include_role:
    name: iop_kafka

+# Kafka is the first container on iop-core-network, so the gateway IP


This, as you are discovering, is tricky and the same issue I ran into when I tried to do this before. This creates a rather fragile experience.

ehelms · 2026-08-07T01:58:19Z

Here's two other ideas:

Put all the containers on the same network.

The iop network was created back in the foreman-installer for isolation, to reduce the need for TLS everywhere and because we were learning. Then we copied the idea here to start with parity. However, back when we implemented iop we always said this should get easier to manage once everything is in containers because we can normalize.

Put postgresql on both networks and have it listen to both.

Alleny244 · 2026-08-07T06:18:58Z

Here's two other ideas:

Put all the containers on the same network.

The iop network was created back in the foreman-installer for isolation, to reduce the need for TLS everywhere and because we were learning. Then we copied the idea here to start with parity. However, back when we implemented iop we always said this should get easier to manage once everything is in containers because we can normalize.

Put postgresql on both networks and have it listen to both.

I’ll go with PostgreSQL on both networks here. Putting all containers on the same network would require broader changes, so a separate PR would be right?

ehelms · 2026-08-07T13:30:58Z

I’ll go with PostgreSQL on both networks here. Putting all containers on the same network would require broader changes, so a separate PR would be right?

Yes and no, you are ultimately trying to solve this broader problem of running only what's needed on the public interface. I'd rather us solve the architecture question than patch it and change it again right after.

I was reading on how pods accomplish the networking issue, and they create a lightweight "infra" container that basically does nothing but is the first container to always start and is present on the network to ensure it's available. I do not think we should use pods here as that will complicate things, but perhaps we can learn from this idea. The trickiest part would be getting the ordering right. But I do think if they are all on the same network, things get easier and I do not see a reason anymore to split them across networks.

Alleny244 · 2026-08-07T13:44:23Z

I’ll go with PostgreSQL on both networks here. Putting all containers on the same network would require broader changes, so a separate PR would be right?

Yes and no, you are ultimately trying to solve this broader problem of running only what's needed on the public interface. I'd rather us solve the architecture question than patch it and change it again right after.

I was reading on how pods accomplish the networking issue, and they create a lightweight "infra" container that basically does nothing but is the first container to always start and is present on the network to ensure it's available. I do not think we should use pods here as that will complicate things, but perhaps we can learn from this idea. The trickiest part would be getting the ordering right. But I do think if they are all on the same network, things get easier and I do not see a reason anymore to split them across networks.

Yeah, makes sense. I’ll look into the single network approach and see how we can handle the startup ordering.

Alleny244 · 2026-08-10T08:12:44Z

I’ll go with PostgreSQL on both networks here. Putting all containers on the same network would require broader changes, so a separate PR would be right?

Yes and no, you are ultimately trying to solve this broader problem of running only what's needed on the public interface. I'd rather us solve the architecture question than patch it and change it again right after.

I was reading on how pods accomplish the networking issue, and they create a lightweight "infra" container that basically does nothing but is the first container to always start and is present on the network to ensure it's available. I do not think we should use pods here as that will complicate things, but perhaps we can learn from this idea. The trickiest part would be getting the ordering right. But I do think if they are all on the same network, things get easier and I do not see a reason anymore to split them across networks.

Should I reuse the existing iop-core-network or create a new general purpose network ( if naming conventions are being looked into) for the containers

ehelms · 2026-08-10T11:38:11Z

Should I reuse the existing iop-core-network or create a new general purpose network ( if naming conventions are being looked into) for the containers

New general purpose network name that is more descriptive.

stejskalleos · 2026-08-11T08:04:18Z

-foreman_development_database_host: "localhost"
+# Prefer 127.0.0.1 over localhost: Postgres is published on IPv4 only, and
+# "localhost" often tries ::1 first (connection refused).
+foreman_development_database_host: "127.0.0.1"


The commit title says:

Restrict PostgreSQL listen_addresses to localhost and IOP gateway

But I don't see any change in the listen_addresses. Please update the commit message & PR title/description to reflect the current state of the PR.

Yeah, sure. I’ll update it. Also, the current implementation was based on putting PostgreSQL on both networks. Based on the discussion above, we decided to move forward with creating a new network and placing all the containers on it instead. I’ve been working on that implementation and will push the changes shortly.

stejskalleos · 2026-08-11T08:08:22Z

+# Optional parameters - can use defaults.
+# Use 127.0.0.1: Postgres is published on IPv4 only (not ::1), and TCP
+# requires a password now that it is no longer on host networking.
+iop_fdw_database_host: "127.0.0.1"


The iop.md architecture diagram (mermaid) still draws Postgres under "Host System", outside the iop-core-network subgraph, while the prose now says Postgres is reached via postgresql:5432 on that network.

Should we update that as well?

Yeah, it's updated now

Why do some places use 127.0.0.1 and others use postgresql ?

stejskalleos · 2026-08-11T08:11:06Z

    name: postgres_fdw
    login_db: "{{ iop_fdw_database_name }}"
    login_user: postgres
+    login_password: "{{ iop_fdw_login_password }}"


This will log the password value in the logs, especially when using -vvv. Should we use no_log: true or something similar (secrets?) to mask the value of the password?

Yes, added no_log property here

stejskalleos · 2026-08-11T08:12:49Z


-iop_database_host: host.containers.internal
+# Co-located Postgres joins iop-core-network; IOP reaches it by container name.
+iop_database_host: postgresql


Suggestion from AI:

Pre-existing (not introduced by this PR) architectural gap: iop_database_host in src/vars/database.yml is unconditionally set to postgresql (a container name only resolvable on iop-core-network), with no branching on database_mode. Combining --database-mode external with the iop feature was already questionable before this PR (nothing in features.yaml or obsah metadata blocks that combination), and this PR doesn't change that — just flagging it's still an open gap, not something to necessarily fix in this PR.

stejskalleos · 2026-08-11T08:15:03Z

    password: "{{ iop_vulnerability_database_password }}"
    feature: iop

+


Suggested change

stejskalleos · 2026-08-12T08:09:01Z

Given the amount of surface area both PRs touch, it'd be good for a maintainer to pick one direction (or explicitly merge the best of both) before either goes further, rather than have both mature in parallel and conflict at merge time.

@ehelms @ekohl we need a decision on which way (PR) we want to go.

stejskalleos · 2026-08-12T10:13:44Z

 - name: Issue server certificate
  when:
-    - (certificates_source != 'custom_server') or (certificates_hostname == 'localhost')
+    # localhost/candlepin are always issued (service-name TLS on the shared network).


I don't understand why these changes are necessary for this PR. Can you please explain?

Since the services are now on the same network, they communicate using the container names instead of localhost. So we need to add those names to the certificate SANs as well, else TLS hostname verification will fail, right?

Why does this role need to know this though? Just like localhost, can candlepin not be specified in the list of hosts in the playbook to generate certificates for?

Here -- https://github.com/theforeman/foremanctl/blob/master/src/vars/base.yaml#L2

Oh yes, it's updated now

ekohl · 2026-08-12T12:19:21Z

I'd like to avoid introducing additional networks at this point. It's a major architectural change and currently I'm much more interested in actually having feature parity with our existing installation. That will allow us to deprecate the package-based installations, which in turn unlocks dropping Ruby 3.1 support, which in turn allows upgrading Ruby on Rails to at least Rails 7.2. That is needed to get CVE fixes.

At this point in the release cycle even this change is IMHO very risky. If we're going to introduce networks, I think we should do it properly and well thought out.

IMHO neither this nor #403 do what I would expect: have 1 network per service.

When I say service, I mean:

Candlepin
Foreman
Foreman Proxy
Pulp
IoP

But then there are more complexities, like do you run 1 instance of Valkey for both Foreman and Pulp or isolate them from each other? PostgreSQL is in a similar boat.

pablomh · 2026-08-13T10:33:38Z

On "1 network per service" — I think the tricky part is exactly the Valkey/Postgres question you raised.

If Foreman, Candlepin and Pulp all need the same Postgres, there are really only two ways to handle it: hook Postgres up to all three networks, or give each service its own Postgres.

The multi-homing option doesn't end up buying much isolation though — Postgres is reachable from all three networks either way, so "who can reach the DB" looks basically the same as just having one foreman-db network with those three on it. Same picture, just more bridges to draw it.

Separate Postgres/Valkey per service would give real isolation, but that feels like a much bigger change than what we're actually trying to fix (containers listening on 0.0.0.0).

That's the thinking behind going with networks per workflow rather than per named service: foreman-db/foreman-cache are the DB/cache-access boundary — one shared instance, one network, and membership is controlled per consumer. foreman-app covers Foreman/Candlepin/Pulp/httpd, since those already need to talk to each other directly regardless of how many networks exist. And foreman-proxy is split off on its own, since it's the one piece here talking to arbitrary remote hosts and probably shouldn't have a path to the database.

That proxy split feels like the boundary that matters most. The rest ends up as one network because those services are already tightly coupled, and splitting them further wouldn't really stop anything from reaching anything — just add more bridges for the same result.

ekohl · 2026-08-13T10:54:38Z

And foreman-proxy is split off on its own, since it's the one piece here talking to arbitrary remote hosts and probably shouldn't have a path to the database.

#611 would disagree with you.

ehelms · 2026-08-13T14:02:52Z

I'll give both PRs a re-read, but the thing I am missing is why or what value individual networks per service gives us.

Networks aren't for free, they will require our users to ensure that address space is available and the more introduce the more we potentially clash with any existing configuration they might have. We should be thoughtful about this trade-off when considering what additional networks bring us in value given most of our services run locally.

The biggest reason I pushed for a single, custom podman network is to allow easy container-to-container DNS name resolution as this is not available on the default podman network.

ekohl · 2026-08-13T15:12:49Z

I'll give both PRs a re-read, but the thing I am missing is why or what value individual networks per service gives us.

My initial thought was that you would have a good isolation. However, given all the cross-service talk it's questionable how much benefit we would actually get.

Networks aren't for free, they will require our users to ensure that address space is available and the more introduce the more we potentially clash with any existing configuration they might have. We should be thoughtful about this trade-off when considering what additional networks bring us in value given most of our services run locally.

If you make the network IPv6-only then the address space is less of a concern. Also note that with systemd socket activation for Foreman and Pulp they don't have any ingress, only egress.

But sure, it's also additional complexity that can break.

ehelms · 2026-08-18T02:15:30Z

 iop_advisor_database_user: advisor_user
 iop_advisor_database_password: "{{ undef(hint='Set a secure database password') }}"
-iop_advisor_database_host: host.containers.internal
+iop_advisor_database_host: "{{ iop_database_host | default('postgresql') }}"


Suggested change

iop_advisor_database_host: "{{ iop_database_host | default('postgresql') }}"

iop_advisor_database_host: postgresql

I think you get the pattern by now. While I get the temptation to use a variable, the whole point of defaults is to default to something that is meaningful. And not to be reliant on some variable to exist somewhere external.

yeah agreed, have updated all to use default postgresql

ehelms · 2026-08-18T02:17:24Z

+postgresql_networks:
+  - "{{ foreman_core_network_name | default('foreman-core-network') }}"
+postgresql_publish:
+  - "127.0.0.1:5432:5432"


Does this need to be published if everything is on the network that needs it?

For host side consumers, they still require port publication right?

What host side consumers are you expecting to exist?

Mainly for deploy-dev workflow which runs forman on the host itself

Just a doubt here, please correct me if I’m wrong, since the Ansible task community.postgresql.postgresql_user uses login_host: 127.0.0.1, doesn't PostgreSQL need to have its port published on the host for this task to work?

Fair point, I suppose that is the best we can do right now.

ehelms · 2026-08-18T02:26:47Z

This raises the question of:

Do we need TLS for Candlepin?
Do we need TLS for the iop-gateway?

These are both services that are only internally communicated with by other services on the network aka the foreman container. We currently do not have TLS between any of the other services (e.g. iop-*, postgresql, redis).

Alleny244 · 2026-08-18T14:37:54Z

This raises the question of:

Do we need TLS for Candlepin?

Do we need TLS for the iop-gateway?

These are both services that are only internally communicated with by other services on the network aka the foreman container. We currently do not have TLS between any of the other services (e.g. iop-*, postgresql, redis).

TLS here is mainly because Candlepin is currently configured to use HTTPS, and the IOP gateway is used as a Smart Proxy, which Foreman expects to access over HTTPS right?. It’s not required by the shared network itself

ehelms · 2026-08-18T14:58:50Z

TLS here is mainly because Candlepin is currently configured to use HTTPS, and the IOP gateway is used as a Smart Proxy, which Foreman expects to access over HTTPS right?. It’s not required by the shared network itself

Fair point on the IOP gateway.
My point with Candlepin is -- should it continue to be? I suppose we do not have to answer that now, but it's a valid question to consider. We do not, for example, lock down postgresql over TLS.

The dedicated network does offer some security enhancements.

Alleny244 · 2026-08-18T15:13:40Z

TLS here is mainly because Candlepin is currently configured to use HTTPS, and the IOP gateway is used as a Smart Proxy, which Foreman expects to access over HTTPS right?. It’s not required by the shared network itself

Fair point on the IOP gateway. My point with Candlepin is -- should it continue to be? I suppose we do not have to answer that now, but it's a valid question to consider. We do not, for example, lock down postgresql over TLS.

The dedicated network does offer some security enhancements.

yeah maybe we could leave TLS posure as a follow up?

ehelms · 2026-08-20T19:14:04Z

Can you introduce a new document that lays out the architecture of the network? Take a look at #771 as I'd like it to go into that folder structure.

Alleny244 · 2026-08-21T08:34:05Z

Can you introduce a new document that lays out the architecture of the network? Take a look at #771 as I'd like it to go into that folder structure.

Sure I ll take a look into that and formulate one

Alleny244 · 2026-08-21T11:43:02Z

One thing that I noticed was , TLS issue affects both Candlepin and IOP: their certs are issued for candlepin / iop-core-gateway, but deploy-dev Foreman connects via localhost. Currently we are using those hostnames instead of localhost so hostname verification succeeds for production. For the host Foreman, I guess we’d also need to add both names to /etc/hosts pointing to 127.0.0.1 for TLS

pr-processor Bot added the Not yet reviewed label Aug 5, 2026

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from c6c2b37 to de1fe85 Compare August 5, 2026 14:19

Alleny244 marked this pull request as draft August 5, 2026 14:22

Alleny244 force-pushed the fix/postgresql-listen-localhost branch 4 times, most recently from 6bf8eb0 to a7edb41 Compare August 6, 2026 09:16

Alleny244 marked this pull request as ready for review August 6, 2026 11:47

ehelms reviewed Aug 6, 2026

View reviewed changes

pr-processor Bot removed the Not yet reviewed label Aug 6, 2026

Alleny244 force-pushed the fix/postgresql-listen-localhost branch 5 times, most recently from 3b4d9d7 to 6f86341 Compare August 7, 2026 11:47

stejskalleos self-requested a review August 11, 2026 07:57

stejskalleos requested changes Aug 11, 2026

View reviewed changes

pr-processor Bot added the Waiting on contributor label Aug 11, 2026

Alleny244 changed the title ~~Restrict PostgreSQL listen_addresses to localhost and IOP gateway~~ Move co-located containers onto shared foreman-core-network Aug 11, 2026

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from 6f86341 to d65b220 Compare August 11, 2026 10:53

pr-processor Bot added Needs re-review and removed Waiting on contributor labels Aug 11, 2026

Alleny244 force-pushed the fix/postgresql-listen-localhost branch 2 times, most recently from 5c5760a to 456b6e3 Compare August 11, 2026 13:16

stejskalleos reviewed Aug 12, 2026

View reviewed changes

pablomh mentioned this pull request Aug 13, 2026

Switch container networking from host to named bridge networks #403

Closed