Add Foreman development environment

[ehelms]

This adds a way to deploy a development environment similar to how the devel boxes in Forklift work. This deploys the backend services as containers to match production, and clones the source code directly to the VM and installs and configures.

In my testing, timing with this to get to a working development setup:

Forklift: 29 minutes
foremanctl: 16 minutes

I think this is a good starting baseline, as it provides a way to develop using our container-based installation that is similar to how developers work today. I'll be exploring other development setups that can be follow-ons to this method. The important bit for me is getting an environment that works within the context of this repository and our containers.

I did try running a development container and mounting the code into the development container. Within the VM environment this proved to be incredibly slow, likely due to the amount of files being shared and the I/O overhead.

[ehelms]

Updates support broker hosts now, for example:

ANSIBLE_ASK_PASS=true ./forge deploy-dev --target-host <host> --foreman-development-user ehelms

I opened a change to Obsah to support the password ask naturally: theforeman/obsah#91

[adamruzicka]

If anyone wants to follow along and doesn't have a properly set up machine (like me), here's a handy scriptlet to run this within a container to set up a remote machine (remote from the container's point of view, chances are it could be the machine hosting the container)

Spawn a fedora 42 container

podman run -ti --rm fedora:42 bash

Inside

USER=<remote_ssh_username>
HOST=<remote_hostname>

dnf install -y git python3 python3-virtualenv make

git clone https://github.com/theforeman/foremanctl
cd foremanctl
git fetch origin refs/pull/259/head
git checkout FETCH_HEAD

echo "$HOST ansible_user=$USER" >inventories/$HOST

./setup-environment
source .venv/bin/activate
ANSIBLE_ASK_PASS=true ./forge deploy-dev \
    --target-host $HOST \
    --foreman-development-enabled-plugins="katello,foreman_remote_execution,foreman_ansible,foreman_openscap" \
    --foreman-development-user $USER

[ekohl]

On a related note, I've looked at using Puma directly to serve HTTPS (https://community.theforeman.org/t/setting-up-ssl-for-running-foreman-on-https/36168/4) and I still think that's an interesting model: keep all services able to function fully standalone, claiming the entire hostname & port combination.

IMHO this works towards the goal of getting rid of a mandatory Apache in front of all services, allowing you to compose it all from independent services.

[ehelms]

• What targets do we want to support here? In the vagrant case we default to CentOS Stream 9 boxes, which "just work" for a lot of things. But you also add a target-host that basically allows any arbitrary host to become one. And then you add some RHEL-isms that we can't verify in CI here.

I was thinking of starting with documenting where we have tested / developers are known to use this t deploy to. I'd love to be as formal as possible by testing combinations, but I also know that with development setups there is a lot more creativity applied.

[ehelms]

• What is the purpose of the plugin settings YAMLs you provide, without any way to adjust them? (I see why we need a katello.yml, but otherwise?)

Fair question. I asked myself the same of the Foreman settings.yaml and which values are actually needed in there. To be honest, I had not audited any of that yet, I translated from what was known to work in puppet-katello_devel.

I think that's part of the question for me -- do we merge a working version and then edit it or do we edit and narrow this down and then merge.

[evgeni]

• What targets do we want to support here? In the vagrant case we default to CentOS Stream 9 boxes, which "just work" for a lot of things. But you also add a target-host that basically allows any arbitrary host to become one. And then you add some RHEL-isms that we can't verify in CI here.

I was thinking of starting with documenting where we have tested / developers are known to use this t deploy to. I'd love to be as formal as possible by testing combinations, but I also know that with development setups there is a lot more creativity applied.

How about we write in the development setup readme something like:

## Prerequisites

- A running EL9 virtual machine, and inventory that contains knowledge of the VM. For example, using `./forge vms start`.
  - CentOS Stream 9 is recommended and tested
  - Other EL9 variants should work too. Please report if they do not.
- Run `./setup-environment` and activate the virtual environment

[evgeni]

• What is the purpose of the plugin settings YAMLs you provide, without any way to adjust them? (I see why we need a katello.yml, but otherwise?)

Fair question. I asked myself the same of the Foreman settings.yaml and which values are actually needed in there. To be honest, I had not audited any of that yet, I translated from what was known to work in puppet-katello_devel.

I think that's part of the question for me -- do we merge a working version and then edit it or do we edit and narrow this down and then merge.

Ideally, the templates here and in prod are identical (if they can be). I am fine merging as is and converging later, but we need to do that, as otherwise we'll be in "dev and prod diverge too much" land quickly again.

[evgeni]

I tested locally (with vagrant) with the following patch and things work as expected (I enabled foreman_ansible etc)

diff --git development/playbooks/deploy-dev/metadata.obsah.yaml development/playbooks/deploy-dev/metadata.obsah.yaml
index 49f925a..ebe86e6 100644
--- development/playbooks/deploy-dev/metadata.obsah.yaml
+++ development/playbooks/deploy-dev/metadata.obsah.yaml
@@ -3,10 +3,11 @@ help: |
   Deploy and manage Foreman development environment with git-based Foreman and containerized backend services.
 
 variables:
-  foreman_development_enabled_plugin:
+  foreman_development_enabled_plugins:
     help: Plugin to enable (can be used multiple times)
     action: append
     default: [katello, foreman_remote_execution]
+    parameter: --foreman-development-enabled-plugin
   target_host:
     help: Target hostname or IP address for deployment
     action: store
diff --git development/roles/foreman_development/tasks/main.yaml development/roles/foreman_development/tasks/main.yaml
index 3eb8465..8d54a8e 100644
--- development/roles/foreman_development/tasks/main.yaml
+++ development/roles/foreman_development/tasks/main.yaml
@@ -149,6 +149,14 @@
   loop: "{{ foreman_development_enabled_plugins }}"
   loop_control:
     loop_var: plugin_item
+  vars:
+    foreman_development_plugin_config: "{{ foreman_development_plugin_registry[plugin_item] }}"
+    foreman_development_plugin_name: "{{ foreman_development_plugin_config.name.split('/')[1] }}"
+    foreman_development_plugin_org: "{{ foreman_development_plugin_config.name.split('/')[0] }}"
+    foreman_development_plugin_repo_url: "https://github.com/{{ foreman_development_plugin_config.name }}.git"
+    foreman_development_plugin_manage_repo: "{{ foreman_development_plugin_config.manage_repo | default(true) }}"
+    foreman_development_plugin_settings_template: "{{ foreman_development_plugin_config.settings_template | default('') }}"
+    foreman_development_plugin_extra_gemfiles: "{{ foreman_development_plugin_config.extra_gemfiles | default([]) }}"
 
 - name: Install Ruby dependencies
   ansible.builtin.command:
diff --git development/roles/foreman_development/tasks/plugin.yaml development/roles/foreman_development/tasks/plugin.yaml
index 5bd8419..5fb2215 100644
--- development/roles/foreman_development/tasks/plugin.yaml
+++ development/roles/foreman_development/tasks/plugin.yaml
@@ -1,17 +1,4 @@
 ---
-- name: Resolve plugin configuration
-  ansible.builtin.set_fact:
-    foreman_development_plugin_config: "{{ foreman_development_plugin_registry[plugin_item] }}"
-
-- name: Set plugin variables
-  ansible.builtin.set_fact:
-    foreman_development_plugin_name: "{{ foreman_development_plugin_config.name.split('/')[1] }}"
-    foreman_development_plugin_org: "{{ foreman_development_plugin_config.name.split('/')[0] }}"
-    foreman_development_plugin_repo_url: "https://github.com/{{ foreman_development_plugin_config.name }}.git"
-    foreman_development_plugin_manage_repo: "{{ foreman_development_plugin_config.manage_repo | default(true) }}"
-    foreman_development_plugin_settings_template: "{{ foreman_development_plugin_config.settings_template | default('') }}"
-    foreman_development_plugin_extra_gemfiles: "{{ foreman_development_plugin_config.extra_gemfiles | default([]) }}"
-
 - name: Clone plugin repository  # noqa latest[git]
   ansible.builtin.git:
     repo: "{{ foreman_development_plugin_repo_url }}"

I did not test remote deployments with broker etc, but I think it's fine enough for now to go in and be polished later.

[evgeni]

two nitpicks, but otherwise LGTM