Add custom server certificate support

[ehelms]

Why are you introducing these changes? (Problem description, related links)

Allow users to provide their own server certificates via --tls-server-certificate, --tls-server-key, and --tls-server-ca-certificate flags on foremanctl deploy. Custom certificates are copied into the canonical /root/certificates/ structure, while client certificates and localhost certificates continue to be managed by the internal CA.

What are the changes introduced in this pull request?

• Adds new user parameters for providing server certificate, key and CA
• Includes custom certificates in the test matrix

How to test this pull request

Steps to reproduce:

• Perform a standard setup for testing and stop before foremanctl deploy
• Run ./forge custom-certs to generate custom certs on the host
• Deploy with custom certs: ./foremanctl --tls-server-certificate /root/custom-certificates/certs/quadlet.example.com.crt --tls-server-key /root/custom-certificates/private/quadlet.example.com.key --tls-server-ca-certificate /root/custom-certificates/certs/server-ca.crt

Checklist

• Tests added/updated (if applicable)
• Documentation updated (if applicable)

[evgeni]

Which meaning of persist is used here?

The parameter values are stored (persisted) in params.yaml, so the files need to persist on disk as otherwise things will go bad (the same as in the old installer).
And this also effectively means you don't need to pass them on update if the paths don't change.

(Yes, I am picking words, but I want to make sure the implementation matches the intent)

[evgeni]

But you pass custom in CI

[evgeni]

@ehelms I tried to make minimal changes to the tests so they match the docs (no "custom" source, just "default" with additional params) and picked the one change I had in #421 for "don't re-gen the server cert", please have a look if you agree

[ekohl]

Zooming out: what is the server CA certificate? I know we're in a messy place, but can we at least better define this?

Today we really mix a few concerns so I'm going to start with what certbot creates because I think it's a really great setup. For every certificate it creates files /etc/letsencrypt/live/$domain:

• cert.pem only the issued certificate
• chain.pem the intermediate certificates between the issued certificate and the root
• fullchain.pem the issued certificate AND the intermediate certificates (the same as cert.pem and chain.pem combined)
• privkey.pem the private key

Note it doesn't write the actual root CA anywhere and it assumes that's already in the trust stores somewhere.

You can configure Apache in 2 ways correctly serve to clients. What we do today:

SSLCertificateFile /etc/letsencrypt/live/$domain/cert.pem
SSLCertificateChainFile /etc/letsencrypt/live/$domain/chain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/$domain/privkey.pem

But quoting https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslcertificatechainfile

SSLCertificateChainFile became obsolete with version 2.4.8, when SSLCertificateFile was extended to also load intermediate CA certificates from the server certificate file.

These days it's recommended to do:

SSLCertificateFile /etc/letsencrypt/live/$domain/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/$domain/privkey.pem

Note that if you use multiple algorithms (typically RSA, DSA, ECC, but with PQC also ML-DSA) you need to repeat the SSLCertificateFile statement. How it exactly matches certs and keys is unclear to me. This also implies the foremanctl --tls-server-certificate parameters needs to be repeatable.

In Foreman we also need the actual root CA because we want to distribute that to clients. It's also used to only allow certificates signed by a specific CA to do client authentication. If you use a third party CA that is already well known (like Let's Encrypt) then is there really a point to specifying it? Right now our tooling expects it so I can accept we'll do that for now.

As we're moving to a whole new installer, I think this is the moment to properly define the inputs and what we expect. As we're also looking at PQC then I think the fullchain.pem approach is really the only sane approach but not 100% sure all of our tooling supports that today.

[evgeni]

We only use the server_ca for Apache as a "server" thing, so that part should work.
We also pass it to Foreman Proxy, and Hammer to validate the connection to the server, but there it's already used as a "CA file" which can contain multiple certs, and one more cert in that file won't hurt.

[ehelms]

As we're moving to a whole new installer, I think this is the moment to properly define the inputs and what we expect. As we're also looking at PQC then I think the fullchain.pem approach is really the only sane approach but not 100% sure all of our tooling supports that today.

Yea, we'd have to audit and potentially update any component to work this same way that uses public facing certificates. And we'd be putting the onus on the user to concatenate their CA and server certificate together. They are capable of this, but we'd have to do error checking and what not to ensure the user does it correctly. I'm hesitant to go that route, at least right now. Especially with not having resolved our installer certificate strategy. I am hopeful #421 is the right approach and that would allow us to then consider this route.

[ekohl]

I'm wondering what we should do. I'd like to have a GA quality with Foreman 3.19, but we can "migrate" in Foreman 3.20. But at that point I'd really want to have finalized the meaning of parameters for at least some time.

Looking at https://community.theforeman.org/t/foreman-3-19-schedule-and-planning/45747 we have prep week next week so need to decide soon. Not saying we can't merge this in 3.19, but then we also need to plan follow up issues.

[ehelms]

I'm wondering what we should do. I'd like to have a GA quality with Foreman 3.19, but we can "migrate" in Foreman 3.20. But at that point I'd really want to have finalized the meaning of parameters for at least some time.

Does GA quality meaning supporting upgrades from release to release?

[ekohl]

IMHO yes