[bug] Built-in KMS encryption doesn't work with a custom CA chain

[Clockwork-Muse]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Omni cannot be used as an automatic KMS system when Omni has a certificate from a custom root chain; a loop of "untrusted certificate errors" is printed to the logs. Any attached nodes are effectively bricked, and require re-installation.

(sorry this is an image, but it's a remote console view)

Note that installation of a cluster with custom root chains over the entire process otherwise works, via specifying the trusted roots config on initial ISO boot and in cluster config during install.

Expected Behavior

The automatic KMS setup includes the certificates from Omni in the setup. Or alternatively, provides a way to specify one.

Steps To Reproduce

1. On Ubuntu (or any other host), generate a custom root certificate and chain (or get one signed by a non-standard certificate chain).
2. Install the certificate root to the host CA store.
   • On Ubuntu, move the new certificate chain into /usr/local/share/ca-certificates, then run sudo update-ca-certificates
3. Follow the standard instructions for self-hosting. Instead of using the directions for a normal trusted cert, mount in the created certs.
   • In addition to mounting in the custom certificates, also mount the host certificate chains by adding the following mounts, eg:

   # For Ubuntu host, but should be similar on many distros
   docker run 
   ...
    -v /etc/ssl/certs:/etc/ssl/certs:ro \
    -v /usr/share/ca-certificates:/usr/share/ca-certificates:ro \
   ...
   ghcr.io/siderolabs/omni:v0.46.3
   ...
   

4. Access the Omni console as normal and register a new machine;
   • When downloading installation media, specify the kernel parameter talos.config=metal-iso
   • Follow the directions to generate the required iso, the contents (at least partially) being the TrustedRootsConfig config
   • Make sure to also add the TrustedRootsConfig as a machine config patch after it has registered itself.
5. Attempt to spin up a new cluster with Omni as the KMS system;
   • Add the TrustedRootsConfig as a cluster config patch
   • Also select the checkbox for "Encrypt Disks":

When you hit "Create Cluster", observe that Talos Linux installs, but after the machine reboots it fails to decrypt the STATE partition (which isn't entirely surprising given the warnings about any network config).

What browsers are you seeing the problem on?

No response

Anything else?

There could be multiple ways that it may have such a chain;

• Corporate MITM proxy for SaaS solution
• On-prem hosting with custom trust roots for security or other reasons

[smira]

This is quite expected, as you try to encrypt with the secret (trusted CA) being inside the encrypted partition.

There is no good way here except for not encrypting STATE, or using a different encryption method for it

[Clockwork-Muse]

... you can't set that up, though: you only get a blanket "everything is KMS" config applied. The to-be-applied config is also not surfaced, so you can't edit it to do that either.
I'm not sure what would happen if you supplied a machine config for the setup and then clicked on "Encrypt disks"...

Obviously the system has to trust some set of certificates, or the feature wouldn't work at all - my assumption is I could probably build a custom talos image that baked in the certificate chains, but that would be a heavier lift. It would also make rolling the root certificates more annoying.

[smira]

Your issue doesn't have to anything with KMS probably - as you rolled self-signed Omni cert, Talos won't establish Omni connection it until it can load the trusted CA list.

The easiest way is to have a proper cert for Omni, tons of problems gone.

[Clockwork-Muse]

I do have a working talos instance with a custom chain. I can even use other forms of encryption (if I enable it during install). But the built-in easy button doesn't work with custom chains.