RFC: weaning off cockpit web console from self-signed certificates

[martinpitt]

Hello all,

the Cockpit team is currently investigating a redesign of Cockpit's web server, which becomes slowly but surely more pressing. See details if you are interested. A very interesting possibility is to replace much of it with a web service worker, which would allow us to make the architecture dramatically simpler, more robust, and more secure (see prototype).

The main concern with this approach is that web service workers don't work with unknown TLS certificates ("secure context"), in particular for the self-signed certificates that cockpit creates on startup if you didn't supply your own. I. e. they don't support the usual TOFU model with "click here to accept this unknown certificate anyway" that browsers show for "normal" web pages in this case.

So I would like to do a little survey and request for comments about deprecating or changing the handling of self-signed TLS certs in Cockpit. This is an annoyingly hard problem: On one hand we really want to build a world where TLS works and your services are trustworthy (especially if you are typing in your root password as the next thing!), but for many home or even small business situations even LetsEncrypt etc. is impractical/impossible (no DNS, etc.).

My questions:

• Are you currently accessing Cockpit with the builtin fallback self-signed certificate?

• If so, would you have a better option like getting a proper cert, or joining the machine to a FreeIPA domain (which requests a valid cert from the IPA server), or running Cockpit behind a reverse proxy which already has a proper certificate? Or using the Cockpit Client flatpak? If any of these don't work for you, can you describe your situation?

• If the above isn't possible, could you live with running the cockpit web server on a machine which has proper TLS setup, and connect to your target machines with SSH?

• As last resort, Cockpit could show an error page (after accepting the invalid certificate in the browser) which explains the situation and offers you to download the CA certificate for importing into your browser as trusted CA. This includes the case that the default self-signed certificate is not valid for the hostname you are using to connect to cockpit (i.e. not "localhost" or the server's unqualified host name) - then you need to regenerate the certificate with the additional name.

  This could then be used to actually establish some trust to the certificate, by something like ssh remote cockpit-show-cert which would show you the fingerprint or similar -- then you have something actually meaningful to check in the browser.

  Would that be acceptable for you?

• Are you working on a different web-based project and already have some experience with handling proper certificates?

Notes:

• This only affects remote connections -- if you connect to your own machine on http://localhost:9090, you can use unencrypted HTTP and service workers are fine.

• We are aware of and use sscg if it's installed (cockpkit-ws.rpm Recommends: it). We can change the non-sscg fallback to also create a separate CA and cert, instead of the single self-signed cert. So that's not a problem, just a bit of work.

Thank you!

[AliveDevil]

Are you currently accessing Cockpit with the builtin fallback self-signed certificate?

Yes, in a home/lab setup (so breaking every and all best-practices there are) without public access.

If so, would you have a better option like […] running Cockpit behind a reverse proxy which already has a proper certificate? Or using the Cockpit Client flatpak? If any of these don't work for you, can you describe your situation?

Currently Cockpit is running on manage.lab.xc, I'd have to change to some public domain and can't use "lab.xc" anymore. I don't have a CA setup, and it's unfeasible to change that (just a bunch of end-devices, I don't want to deploy CA certificates to) in a Home-environment (so, not Active Directory/LDAP setup).

Cockpit Client flatpak: Windows, macOS, iOS, I don't have a homogeneous Linux desktop workstation setup.

If the above isn't possible, could you live with running the cockpit web server on a machine which has proper TLS setup, and connect to your target machines with SSH?

I'm using the "Add Hosts"-method of managing like 24 hosts, all registered to one GUI-instance (the others just have cockpit-ws installed), installing cockpit GUI on all of these hosts is cumbersome, and SSHing into these doesn't make sense, when I want to have quick bookmarks into machines. I.e. I have one "admin"-account on the manage-instance, and all other hosts trust the ssh key of that account.
I'd like to keep using one single entrypoint into managing multiple hosts. <- I know that this is discouraged, and you want to move away from this, but this is a huge selling point for managing multiple hosts, without remembering all the accounts, ssh keys, host names and stuff.

As last resort, Cockpit could show an error page (after accepting the invalid certificate in the browser) which explains the situation and offers you to download the CA certificate for importing into your browser as trusted CA. This includes the case that the default self-signed certificate is not valid for the hostname you are using to connect to cockpit (i.e. not "localhost" or the server's unqualified host name)

FreeIPA/oVirt had this, if I remember correctly. There is/was a link to the CA certificate, which could be downloaded and installed manually. (see comment, last part)
This could be an option, though then you need to regenerate the certificate with the additional name. should have a simple command, which just takes the hostname of the current machine, and generate the certificate automatically (I don't remember all the switches I have to pass to openssl to get a self-signed certificate working).

[martinpitt]

@AliveDevil Thanks for you reply! Indeed "Add hosts" is deprecated, but you can also ssh-connect to other machines directly from the login page (and bookmark these even). Of course we'll provide a simple command to (re)generate a webserver cert with additional hosts (we already have something like it, /usr/libexec/cockpit-certificate-ensure -- we'd generalize/extend it a bit and probably put it into $PATH).

[AliveDevil]

ssh-connect to other machines directly from the login page

Which doesn't allow me to use one central login (the login of the gateway Cockpit), and use that users ssh key on all other machines - this would work for LDAP/AD setups, but not with "just a bunch of hosts", which happen to have been given the same SSH authorized key during cloud-init. As ssh-connect doesn't support ssh-keys, I'd have to use root/password login.

(and bookmark these even)

Not sure about the state here, as I'm running on Debian 12, and haven't kept up here (not further than update 322, that is): There's not a dropdown on the login page with known remote hosts, but I have to bookmark every host manually for every client device I want to access these hosts. So, I'd have to setup bookmark for all of the 24 hosts on e.g. a mac Mini, my Windows PC, my Linux laptop, a macBook Air, on my iPhone?

I don't want to derail the discussion from the certificate topic here too much, so I don't expect an answer to the above, if it is out-of-scope/not helping the discussion.

course we'll provide a simple command to (re)generate a webserver cert with additional hosts

Yeah, something like this. Just for clarification: The architecture is kept the same: Cockpit acts as a jump-host between client device and connected to machine, correct? So client device doesn't have to have IP routable access to the machine.

[martinpitt]

Right, we don't plan any observable change with the ws rewrite, other than this self-signed cert problem. The network visibility/architecture/port/ssh connectivity graph is supposed to stay the same.

[jwildeboer]

@AliveDevil Thanks for you reply! Indeed "Add hosts" is deprecated, but you can also ssh-connect to other machines directly from the login page (and bookmark these even).

Not really, though ;) I just tried from my RHEL 9 box to connect via ssh (which listens on a non-standard port) to one of my homelab machines and got the wonderful

SELinux is preventing /usr/libexec/cockpit-ssh from name_connect access on the tcp_socket port 2XXX.

;)

[martinpitt]

@jwildeboer ah, assuming that this works with standard ssh(1) (which uses a different SELinux policy), this should be fixed in 9.6 --- we dropped cockpit-ssh and made it use ssh(1).

[abbra]

FreeIPA developer here.

In Fedora/RHEL/Debian/etc we have certmonger as a tool to handle issuance and refresh of the certificates from different CAs.

On every system where certmonger is present, you already have a self-signed CA that can issue certificates using the same mechanism you are using for FreeIPA integration. In fact, for the localkdc (Kerberos KDC on each machine for local needs to replace insecure crypto exchanges like NTLM) integration we plan to use it by default as well, so it will be in use already.

Here is an example of using it to generate a certificate from the local self-signed CA:

    # Prepare Anonymous PKINIT configuration for the realm
    ${LOCAL_GETCERT} request \
        -K "krbtgt/${LOCALKDC_REALM}@${LOCALKDC_REALM}" \
        -N "cn=${LOCALKDC_FQDN}" \
        -F /etc/pki/ca-trust/source/anchors/localkdc-ca.crt \
        -k "${LOCALSTATE_DIR}/kdc.key" \
        -f "${LOCALSTATE_DIR}/kdc.crt" \
        -U{id-kp-serverAuth,id-pkinit-KPKdc} \
        -u{digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment}

On RHEL/Fedora one can copy the certificate (/etc/pki/ca-trust/source/anchors/localkdc-ca.crt above) to your system and run update-ca-trust to trust it. After this all certificates issued by that CA will become trusted. This is pretty much something you can do over SSH.

I think it would be good if Cockpit would use this self-sign CA to request the certificates instead of issuing them as self-signed certs.

An alternative to retrieving the CA chain through alternative mechanisms (such as ssh), could be something we do in FreeIPA: we expose HTTP end-point to get access to the CA certificate from the clients that currently don't yet have trust to the CA.

[martinpitt]

Hello @abbra Thanks -- we are aware of certmonger and sscg, etc. -- The problem isn't the mechanics of building a certificate, but (1) establishing trust to it, (2) getting it into the client browsers, and (3) the fact that it's really hard (and impossible to do/guess automatically) to specify the domain name/IP restrictions of that CA. We don't want the user to import a CA which is valid for any and all domains/IPs (that would be a major danger of abusing the local CA).

We already have an endpoint in Cockpit to retrieve the CA (had to do this years ago as a workaround for some Safari websocket bug). But of course that has no established trust, so we had the idea of using ssh to validate the fingerprint.

[abbra]

Each local self-signed CA is unique. If you are controlling the options to use for the getcert request, you can specify both DNS SAN names for the machine and IP addresses, and that means your certificate will always be limited by those. You can also embed the SSH figerprint into the certificate you requested as one of additional fields (or as a SAN name). On the other hand, it maybe possible to extend certmonger with a restriction for self-sign CA to only sign domains/IPs associated with the host configuration.

The trust establishing issue is a common problem and since you have to distribute CA content elsewhere, it has to be done through a different channel anyway. This is not a problem specific to Cockpit and self-signed certificates. For any system which wants to trust something produced externally, one needs to solve this issue. It just happens to be solved for CAB Forum-provided CA chains by browsers who embed CAB Forum's root certificate chains. I do not see this as a problem unique to Cockpit and something that Cockpit would have to desperately to drop self-signed certs for.

[martinpitt]

@abbra Right, thanks, that matches my understanding. It's indeed not a cockpit specific problem, we are just facing it now as we decide whether we can use web service workers -- if untrusted certs are something which is too hard to get rid of, then we can't use them, and need to keep a more "classic" web server.

[jmbit]

Are you currently accessing Cockpit with the builtin fallback self-signed certificate?

Yes, this is a setup I have deployed on multiple locations

If so, would you have a better option like getting a proper cert, or joining the machine to a FreeIPA domain (which requests a valid cert from the IPA server), or running Cockpit behind a reverse proxy which already has a proper certificate? Or using the Cockpit Client flatpak? If any of these don't work for you, can you describe your situation?

None of the sites where I have set up Cockpit on a system have a FreeIPA domain (At best, they have an AD domain). The people accessing it tend to be doing so from Windows PCs, and for the reverse proxy the issue stays the same. They also tend to access Cockpit via the IP, not fqdn.

If the above isn't possible, could you live with running the cockpit web server on a machine which has proper TLS setup, and connect to your target machines with SSH?

These environments usually only have one or two systems with a cockpit web UI.

Are you working on a different web-based project and already have some experience with handling proper certificates?

I do, and in an ideal environment this would be the way to go, but that isn't most places I have interacted with.

[martinpitt]

Thanks @jmbit ! Could you live with the "show an error page with CA import instructions" last resort?

[allisonkarlitskaya]

Put a different way: how many people here are currently accessing their Cockpit server by entering a URL like https://myserver.mydomain.org/, https://myserver.local/, or https://129.168.0.100/ and clicking past the "yes, I know this is insecure!" button?

If this applies do you, do you feel guilty/worried about doing this? Would you be willing to consider an alternative? Would you appreciate it if we made efforts to provide an alternative?

[purpleidea]

On mastodon it was asked for comments:

https://mastodon.social/@Cockpit@fosstodon.org/114143725996610452

My opinion is you should always be able to have the simple, self-signed, "i know it's not a public cert" situation. Of course better options are better for when you have fancier infra, but not everyone has cycles to set all that up. Thanks for working on cockpit!

[martinpitt]

@purpleidea thanks for your reply! Yes, it is clear and undisputed that we need some self-signed fallback. The point of this survey is to collect situations where none of the better options (real cert, or Cockpit Client etc.) works, and if you could live with the additional CA import into the browser?

[purpleidea]

if you could live with the additional CA import into the browser

I would prefer not. My main reason is that (1) it's an extra step and (2) I don't know how to automate that step atm. (Think https://github.com/purpleidea/mgmt/ setting up a users machine automatically for them.) HTH

[martinpitt]

@purpleidea For (2) how do you automate it with the self-signed cert? The usual automation would be to start cockpit once (so that it generates its cert) or run cockpit-certificate-ensure, and then add /etc/cockpit/ws-certs.d/0-self-signed-ca.pem to /etc/pki/ca-trust/source/anchors/ (or /etc/ssl/certs/ on Debian-ish OSes) and running update-ca-trust. That wouldn't change.

[amessina]

I just recently updated my cockpit instances to use FreeIPA issued certificates as I already had FreeIPA in place for some time. It's a great option only since I already have the infrastructure setup. For the home lab or small business setup with a couple servers, FreeIPA would be overkill just to support cockpit. But a simple web-proxy handling TLS termination and the self-signed certificate might be a lightweight and less complicated. If it's easy enough to create that on install, but "upgrade" to FreeIPA or Let's Encrypt, maybe that's a sensisble solution.

[martinpitt]

Thanks to all responders! This is still such an intrinsically hard problem that we'll not pursue web service workers for the time being -- the cost to many users would IMHO be too high. 😢