Inconsistent generation of stunnel certificates on upgrade

[ydirson]

Upgrading to XCP-ng 8.3.0-rc1 (i.e. not far from current XS8, featuring XAPI 24.16.0) from various older versions show various behaviours, which result in missing stunnel certificates, which in turn cause at least

• inability to pool-join to a pool missing such certs
• inability of XO to administrate such a pool

Tested combinations:

• 8.2: OK, both /etc/stunnel/certs/ and /etc/stunnel/certs-pool/ are generated
• 7.5: KO, /etc/stunnel/certs/ is generated, but /etc/stunnel/certs-pool/ is not
• 8.2 upgraded from 7.5: KO, none is generated

[psafont]

If I understand correctly, the second case is a direct upgrade from 7.5 to 8.3. This was never tested, so I'm not surprised it's broken in strange ways.
The third case is an upgrade from 7.5 to 8.2, to 8.3, correct? This is worrying since it should be the same as the first case.

[ydirson]

The third case is an upgrade from 7.5 to 8.2, to 8.3, correct? This is worrying since it should be the same as the first case.

right

[ydirson]

It likely that in above tests the host that got /etc/stunnel/certs/ but not certs-pool got there through a XAPI call from the XO SDN plugin, which provided that cert.

OTOH I got several cases of a 8.2.1 install not getting any of those dirs created after update to 8.3. update-ca-bundle.sh is also obviously not launched (running it manually gets empty certs dirs created, as well as the resulting bundles)

What component should be responsible for getting this straight when upgrading from 8.2?

[psafont]

gencert.service should generate them, if they don't exist. This is done on host startup, through the systemd dependency chain xapi -> stunnel -> gencert

[ydirson]

gencert.service only includes calls to gencert for certs in /etc/xensource/, and restarting the service does not change the situation.

I have noticed that xe host-refresh-server-certificate does something, but it only generates a .new.pem, so it could be part of the picture, but I can't see who would be running it.

[17:40 host1 ~]# find /etc/stunnel/ -ls
721503    4 drwxr-xr-x   2 root     root         4096 Jul 29 11:58 /etc/stunnel/
722029    4 -rw-------   1 root     root          737 Jul 29 11:58 /etc/stunnel/xapi.conf
[17:41 host1 ~]# xe host-refresh-server-certificate host=host1 
[17:41 host1 ~]# find /etc/stunnel/ -ls
721503    4 drwxr-xr-x   4 root     root         4096 Jul 29 17:41 /etc/stunnel/
722072    0 -rw-r--r--   1 root     root            0 Jul 29 17:41 /etc/stunnel/xapi-stunnel-ca-bundle.pem
722029    4 -rw-------   1 root     root          737 Jul 29 11:58 /etc/stunnel/xapi.conf
722069    4 drwxr-xr-x   2 root     root         4096 Jul 29 17:41 /etc/stunnel/certs
722070    4 drwx------   2 root     root         4096 Jul 29 17:41 /etc/stunnel/certs-pool
722071    4 -rw-r--r--   1 root     root         1115 Jul 29 17:41 /etc/stunnel/certs-pool/17cadc57-e26c-41d9-8e2a-67fa37f3e3e6.new.pem
722073    0 -rw-r--r--   1 root     root            0 Jul 29 17:41 /etc/stunnel/xapi-pool-ca-bundle.pem

[psafont]

This looks like a bug in the distribution code, which should rename the certificate at the end:

ocaml/xapi/cert_distrib.ml

let distribute_new_host_cert ~__context ~host ~content =
  let hosts = Db.Host.get_all ~__context in
  let uuid = Db.Host.get_uuid ~__context ~self:host in
  let file =
    WireProtocol.{filename= Printf.sprintf "%s.new.pem" uuid; content}
  in
  let job rpc session_id host =
    Worker.remote_write_certs_fs HostPoolCertificate Merge [file] host rpc
      session_id
  in
  Helpers.call_api_functions ~__context @@ fun rpc session_id ->
  List.iter (fun host -> job rpc session_id host) hosts ;
  List.iter (fun host -> Worker.remote_regen_bundle host rpc session_id) hosts

[ydirson]

This looks like a bug in the distribution code, which should rename the certificate at the end:

Even if there is a bug there, that does not explain (even more so, in fact) how such certs are in fact generated without that .new part on a brand new install.

[ydirson]

It also seems the update-ca-bundle.sh, which would create empty dirs and bundles in /etc/stunnel/ should they not exist, does not get created. That one is a helper script for ocaml update_ca_bundle() - that it is not called may be linked to the pool certs not being generated.

As long as those bundles are not created, pool-join is still broken.