Redesign container network topology for better service isolation

Replace the three-network model (foreman-db, foreman-cache, foreman-app)
with a per-service network topology derived from actual service
communication requirements:

foreman (not internal, isolated)
  Foreman owns this network. Provides internet access for outbound
  calls. PostgreSQL and Redis attach here for Foreman data access.
  Dynflow, Candlepin, Pulp, and Foreman Proxy also attach to enable
  direct communication with Foreman.

pulp (not internal, isolated)
  Pulp owns this network. Provides internet access for content
  synchronisation. PostgreSQL and Redis attach here for Pulp data
  access. Foreman and Dynflow also attach to reach Pulp directly.

candlepin (internal, isolated)
  Candlepin owns this network. PostgreSQL attaches here for
  Candlepin data access. Foreman attaches to reach Candlepin.
  Candlepin also joins the foreman network so that its hostname is
  resolvable from Foreman's primary DNS without cross-bridge routing.
  Pulp and Foreman Proxy have no route to Candlepin.

foreman-proxy (not internal, not isolated)
  Connects Foreman and Foreman Proxy. Not isolated because Foreman
  reaches the proxy at quadlet.example.com:8443 via host-gateway
  DNAT, which crosses the foreman-to-foreman-proxy bridge boundary.
  Netavark isolation rules would block this cross-bridge DNAT
  forwarding. External managed hosts reach the proxy via the
  published port from the physical network, unaffected by isolation.

Additionally:
- postgresql_network and redis_network renamed to postgresql_networks
  and redis_networks (lists) to support multiple network attachments.
  Updated development and remote-database playbooks accordingly.
- Foreman Proxy container attaches only to foreman-proxy network,
  not to foreman, keeping it isolated from candlepin and pulp.
- Updated docs/deployment.md to reflect the new topology.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: pablomh

development/playbooks/deploy-dev/deploy-dev.yaml

@@ -17,7 +17,7 @@
     - role: certificates
     - role: postgresql
       vars:
-        postgresql_network: host
+        postgresql_networks: host
         postgresql_databases:
           - name: "{{ candlepin_database_name }}"
             owner: "{{ candlepin_database_user }}"
@@ -37,7 +37,7 @@
             password: "{{ pulp_database_password }}"
     - role: redis
       vars:
-        redis_network: host
+        redis_networks: host
     - role: candlepin
       vars:
         candlepin_networks: host

docs/deployment.md

@@ -241,90 +241,87 @@ There is a desire to allow deployments where a single `foremanctl` control node
 
 All containers are connected to one or more named Podman bridge networks instead of sharing the host network namespace, limiting lateral movement: a container can only reach the services it is explicitly connected to.
 
+The topology is designed top-down from service communication requirements. Each of the four application services (Foreman, Pulp, Candlepin, Foreman Proxy) owns a network that carries its private backend traffic. Services join each other's networks only where communication is required, making the allowed paths explicit and auditable.
+
 ### Networks
 
-#### `foreman-db`
+#### `foreman`
 
-**Properties:** `internal: true`, `isolate: true`
+**Properties:** `internal: false`, `isolate: true`
 
-The database network. Only containers that need to read or write persistent data are attached.
+Foreman's primary network. Provides outbound internet access for Foreman's external API calls. PostgreSQL and Redis attach here for Foreman's data access. Candlepin, Pulp, Dynflow, and Foreman Proxy also attach so that Foreman can reach each of them directly, and they can reach Foreman.
 
-- `internal: true` removes the default gateway, so no container on this network can initiate outbound internet connections. Database servers have no reason to reach the internet, and clients that need internet access (e.g. for content sync) are multi-homed and use a different network for that.
-- `isolate: true` prevents containers on this network from forwarding packets to containers on other bridge networks, closing off lateral movement paths between network segments.
+`isolate: true` prevents containers on this network from forwarding packets to containers on other bridge networks, so Candlepin cannot reach Pulp via this network even though both are attached.
 
 | Container | Role |
 |-----------|------|
-| `postgresql` | Server — listens on port 5432 (internal DB only) |
-| `foreman` | Client |
+| `postgresql` | Server — Foreman database |
+| `redis` | Server — Foreman cache and Dynflow queue |
+| `candlepin` | Server — also joins to make its hostname resolvable from Foreman's primary DNS |
+| `foreman` | Owner |
 | `dynflow-sidekiq@*` | Client |
-| `foreman-recurring@*` | Client |
-| `candlepin` | Client (internal DB only) |
-| `pulp-api` | Client |
-| `pulp-content` | Client |
-| `pulp-worker@*` | Client |
+| `pulp-api` | Server — reachable from Foreman |
+| `pulp-content` | Server — reachable from Foreman |
+| `pulp-worker@*` | Worker |
+| `foreman-proxy` | Server — reachable from Foreman |
 
-Ansible's `community.postgresql.*` modules reach the database during deployment via a Unix socket: `/var/run/postgresql` is bind-mounted from the host into the container so the socket is accessible on the host without publishing a TCP port.
+`foreman`, `pulp-api`, and `pulp-content` publish their respective ports to `127.0.0.1` so that the `httpd` reverse proxy running on the host can reach them.
 
-#### `foreman-cache`
+#### `pulp`
 
-**Properties:** `internal: true`, `isolate: true`
+**Properties:** `internal: false`, `isolate: true`
 
-The cache network. Only containers that need to reach Redis are attached. The same rationale as `foreman-db` applies: cache servers have no business reaching the internet, and the `isolate` flag prevents bridge pivoting.
+Pulp's private network. Provides outbound internet access for content synchronisation. PostgreSQL and Redis attach here for Pulp's data access. Foreman and Dynflow also attach to reach Pulp directly.
 
 | Container | Role |
 |-----------|------|
-| `redis` | Server — listens on port 6379 |
-| `foreman` | Client — app cache and Dynflow queue |
-| `dynflow-sidekiq@*` | Client — job queue |
-| `foreman-recurring@*` | Client — job queue |
-| `pulp-api` | Client |
-| `pulp-content` | Client |
-| `pulp-worker@*` | Client |
+| `postgresql` | Server — Pulp database |
+| `redis` | Server — Pulp cache and task queue |
+| `pulp-api` | Owner |
+| `pulp-content` | Owner |
+| `pulp-worker@*` | Owner |
+| `foreman` | Client |
+| `dynflow-sidekiq@*` | Client |
 
-Redis does not publish any port to the host: it is a purely internal service with no legitimate consumers outside the container network.
+#### `candlepin`
 
-#### `foreman-app`
+**Properties:** `internal: true`, `isolate: true`
 
-**Properties:** none (`internal: false`, `isolate: false`)
+Candlepin's private network. `internal: true` removes the default gateway: Candlepin has no reason to initiate outbound internet connections. PostgreSQL attaches here for Candlepin's data access. Foreman attaches to reach Candlepin directly. Pulp and Foreman Proxy have no route to Candlepin.
 
-The application network. Containers that need to communicate with each other at the application layer, or that need outbound internet access (e.g. for content synchronisation), are attached here.
+Candlepin also joins the `foreman` network so that its hostname is registered in Foreman's primary DNS zone, avoiding cross-bridge DNAT for the Foreman-to-Candlepin connection.
 
 | Container | Role |
 |-----------|------|
-| `candlepin` | Server — Tomcat (23443) and Artemis STOMP broker (61613) |
-| `foreman` | Client to Candlepin; serves Foreman Proxy requests |
-| `dynflow-sidekiq@*` | Client |
-| `foreman-recurring@*` | Client |
-| `pulp-api` | Server — API (24817); needs internet for content sync |
-| `pulp-content` | Server — content (24816); needs internet for content sync |
-| `pulp-worker@*` | Worker — needs internet for content sync |
+| `postgresql` | Server — Candlepin database |
+| `candlepin` | Owner — Tomcat (23443) and Artemis STOMP broker (61613) |
+| `foreman` | Client |
 
-Candlepin does not publish any ports to the host: `foreman` reaches it directly over the bridge using its DNS name. `foreman`, `pulp-api`, and `pulp-content` publish their respective ports to `127.0.0.1` so that the `httpd` reverse proxy running on the host can reach them.
+Candlepin does not publish any ports to the host: Foreman reaches it directly over the bridge using its DNS name.
 
-#### `foreman-proxy-net`
+#### `foreman-proxy`
 
-**Properties:** none (`internal: false`, `isolate: false`)
+**Properties:** `internal: false`, `isolate: false`
 
-The proxy network, used exclusively for communication between Foreman and Foreman Proxy. Keeping this traffic on a dedicated network makes it straightforward to apply stricter controls in future without affecting the rest of the application.
+The proxy network, used for communication between Foreman and Foreman Proxy. Not isolated because Foreman reaches the proxy at `quadlet.example.com:8443` via host-gateway DNAT, which crosses the `foreman`-to-`foreman-proxy` bridge boundary; netavark isolation rules would block this cross-bridge DNAT forwarding.
 
 | Container | Role |
 |-----------|------|
-| `foreman-proxy` | Server — listens on 0.0.0.0:8443 (external) |
+| `foreman-proxy` | Owner — listens on `0.0.0.0:8443` |
 | `foreman` | Client |
 
-`foreman-proxy` publishes port `0.0.0.0:8443` so that remote Foreman Proxies and clients can register and communicate with it from outside the host.
+`foreman-proxy` publishes port `0.0.0.0:8443` so that external managed hosts can communicate with it, and so that Foreman on the `foreman` network can call back to it via host-gateway.
 
 ### Network membership summary
 
-| Container | foreman-db | foreman-cache | foreman-app | foreman-proxy-net |
-|-----------|:----------:|:-------------:|:-----------:|:-----------------:|
-| `postgresql` | ✓ | | | |
-| `redis` | | ✓ | | |
-| `candlepin` | ✓ (internal DB) | | ✓ | |
+| Container | `foreman` | `pulp` | `candlepin` | `foreman-proxy` |
+|-----------|:---------:|:------:|:-----------:|:---------------:|
+| `postgresql` | ✓ | ✓ | ✓ | |
+| `redis` | ✓ | ✓ | | |
+| `candlepin` | ✓ | | ✓ | |
 | `foreman` | ✓ | ✓ | ✓ | ✓ |
-| `dynflow-sidekiq@*` | ✓ | ✓ | ✓ | |
-| `foreman-recurring@*` | ✓ | ✓ | ✓ | |
+| `dynflow-sidekiq@*` | ✓ | ✓ | | |
 | `foreman-proxy` | | | | ✓ |
-| `pulp-api` | ✓ | ✓ | ✓ | |
-| `pulp-content` | ✓ | ✓ | ✓ | |
-| `pulp-worker@*` | ✓ | ✓ | ✓ | |
+| `pulp-api` | ✓ | ✓ | | |
+| `pulp-content` | ✓ | ✓ | | |
+| `pulp-worker@*` | ✓ | ✓ | | |

src/playbooks/deploy/deploy.yaml

@@ -24,20 +24,20 @@
         certificate_checks_ca: "{{ ca_certificate }}"
     - role: deploy_network
       vars:
-        deploy_network_name: foreman-db
-        deploy_network_internal: true
+        deploy_network_name: foreman
         deploy_network_isolate: true
     - role: deploy_network
       vars:
-        deploy_network_name: foreman-cache
-        deploy_network_internal: true
+        deploy_network_name: pulp
         deploy_network_isolate: true
     - role: deploy_network
       vars:
-        deploy_network_name: foreman-app
+        deploy_network_name: candlepin
+        deploy_network_internal: true
+        deploy_network_isolate: true
     - role: deploy_network
       vars:
-        deploy_network_name: foreman-proxy-net
+        deploy_network_name: foreman-proxy
     - role: postgresql
       when:
         - database_mode == 'internal'

src/roles/candlepin/defaults/main.yml

@@ -15,7 +15,9 @@ candlepin_ciphers:
 candlepin_container_image: quay.io/foreman/candlepin
 candlepin_container_tag: "4.4.14"
 candlepin_registry_auth_file: /etc/foreman/registry-auth.json
-candlepin_networks: "{{ (['foreman-db'] if database_mode == 'internal' else []) + ['foreman-app'] }}"
+candlepin_networks:
+  - candlepin
+  - foreman
 
 candlepin_database_host: postgresql
 candlepin_database_port: 5432

src/roles/foreman/tasks/main.yaml

@@ -102,10 +102,10 @@
     state: quadlet
     sdnotify: true
     network:
-      - foreman-db
-      - foreman-cache
-      - foreman-app
-      - foreman-proxy-net
+      - foreman
+      - pulp
+      - candlepin
+      - foreman-proxy
     ports:
       - "127.0.0.1:3000:3000"
     hostname: "{{ ansible_facts['fqdn'] }}"
@@ -144,9 +144,8 @@
     state: quadlet
     sdnotify: true
     network:
-      - foreman-db
-      - foreman-cache
-      - foreman-app
+      - foreman
+      - pulp
     hostname: "{{ ansible_facts['fqdn'] }}"
     volume:
       - 'foreman-data-run:/var/run/foreman:z'
@@ -199,9 +198,8 @@
     image: "{{ foreman_container_image }}:{{ foreman_container_tag }}"
     sdnotify: false
     network:
-      - foreman-db
-      - foreman-cache
-      - foreman-app
+      - foreman
+      - pulp
     hostname: "{{ ansible_facts['fqdn'] }}"
     command: "foreman-rake {{ item.rake }}"
     volume:
@@ -252,7 +250,7 @@
       - bin/rails db:migrate && bin/rails db:seed
     detach: false
     rm: true
-    network: "{{ ['foreman-db'] if database_mode == 'internal' else ['foreman-app'] }}"
+    network: foreman
     env:
       FOREMAN_ENABLED_PLUGINS: "{{ foreman_plugins | join(' ') }}"
     secrets:

src/roles/foreman_proxy/tasks/main.yaml

@@ -18,7 +18,8 @@
     image: "{{ foreman_proxy_container_image }}:{{ foreman_proxy_container_tag }}"
     state: quadlet
     sdnotify: true
-    network: foreman-proxy-net
+    network:
+      - foreman-proxy
     ports:
       - "0.0.0.0:8443:8443"
     hostname: "{{ ansible_facts['fqdn'] }}"

src/roles/postgresql/defaults/main.yml

@@ -3,7 +3,10 @@ postgresql_container_image: quay.io/sclorg/postgresql-13-c9s
 postgresql_container_tag: "latest"
 postgresql_registry_auth_file: /etc/foreman/registry-auth.json
 postgresql_container_name: postgresql
-postgresql_network: foreman-db
+postgresql_networks:
+  - foreman
+  - pulp
+  - candlepin
 postgresql_socket_dir: /var/run/postgresql
 postgresql_restart_policy: always
 

src/roles/postgresql/tasks/main.yml

@@ -36,7 +36,7 @@
     state: quadlet
     healthcheck: pg_isready
     sdnotify: healthy
-    network: "{{ postgresql_network }}"
+    network: "{{ postgresql_networks }}"
     volumes:
       - "{{ postgresql_data_dir }}:/var/lib/pgsql/data:Z"
       - "{{ postgresql_socket_dir }}:{{ postgresql_socket_dir }}:Z"

src/roles/pulp/defaults/main.yaml

@@ -36,14 +36,13 @@ pulp_database_user: pulp
 pulp_database_host: postgresql
 pulp_redis_url: "redis://redis:6379/8"
 pulp_networks:
-  - foreman-db
-  - foreman-cache
-  - foreman-app
+  - pulp
 pulp_api_ports:
   - "127.0.0.1:24817:24817"
 pulp_content_ports:
   - "127.0.0.1:24816:24816"
-pulp_migration_networks: "{{ ['foreman-db'] if database_mode == 'internal' else ['foreman-app'] }}"
+pulp_migration_networks:
+  - pulp
 pulp_database_port: 5432
 pulp_database_ssl_mode: disabled
 pulp_database_ssl_ca:

src/roles/redis/defaults/main.yml

@@ -2,4 +2,6 @@
 redis_container_image: quay.io/sclorg/redis-6-c9s
 redis_container_tag: "latest"
 redis_registry_auth_file: /etc/foreman/registry-auth.json
-redis_network: foreman-cache
+redis_networks:
+  - foreman
+  - pulp