An ACME `ClusterIssuer` cannot be used with `spec.tls.issuerConf`

[yoav-katz]

Report

Setting spec.tls.issuerConf to an existing ACME ClusterIssuer (Let's Encrypt, or any ACME server) does not work. The Operator resolves this to external issuer mode and points all leaf Certificate resources at that issuer, but two of them request identifiers that ACME can never issue:

Certificate	commonName and dnsNames
<cluster>-replication-cert	_crunchyrepl
<cluster>-pgbackrest-client-cert	pgbackrest@<cluster-uid>

cert-manager submits commonName and every dnsNames entry to the ACME server as identifiers requiring an HTTP-01 or DNS-01 challenge. Neither of these strings is a resolvable DNS name, so the Order can never be fulfilled.

Both are client certificates whose identity is consumed by CN matching, not by hostname verification:
The other four certificates (cluster, instance, repo host, PgBouncer) are fine with an ACME issuer, given an ACME server that issues the clientAuth EKU and a spec.clusterServiceDNSSuffix set to a domain that server will validate.

More about the problem

There is also no usable workaround:

• spec.secrets.customReplicationClientTLSSecret does bypass cert-manager for the replication cert (internal/controller/postgrescluster/patroni.go:381), but PostgreSQL validates that client cert against ssl_ca_file = /pgconf/tls/ca.crt (internal/postgres/parameters.go:31) - the CA of the cert-manager-issued server cert. A hand-supplied cert would have to be signed by the ACME CA, which is not possible.
• The pgBackRest client cert has no equivalent field. The only opt-out is spec.tls.certManagementPolicy: userProvidedOnly, which is all-or-nothing and disables cert-manager for the whole cluster.

Ideally the internal mTLS client certificates would be issued independently of spec.tls.issuerConf - either from the Operator's own CA or from a separately configurable issuer - since their identifiers are not DNS names and were never meant to be publicly validated.

Steps to reproduce

1. Install cert-manager and create an ACME ClusterIssuer.
2. Set spec.tls.issuerConf to that issuer (kind: ClusterIssuer, group: cert-manager.io) and deploy a cluster.

Versions

not relevant

Anything else?

The in-flight cert-manager documentation for 3.1.0 (percona/k8spg-docs#414) lists ACME and Let's Encrypt as supported external issuers, and states that the configured issuer must sign every leaf certificate, including the replication and pgBackRest ones. That combination is not achievable, so the docs may need adjusting alongside the fix.

[yoav-katz]

I dont know if I'll call it a bug but it does mean acme isnt supported with cert-manager yet

[egegunes]

@yoav-katz I think this more a feature request than a bug, because I don't see any other way than:

Ideally the internal mTLS client certificates would be issued independently of spec.tls.issuerConf - either from the Operator's own CA or from a separately configurable issuer - since their identifiers are not DNS names and were never meant to be publicly validated.

[yoav-katz]

@egegunes totally agree, the bug is that in the next release notes there is a note about supporting acme - which is not true.

[yoav-katz]

On the implementation side, as always, I’d be happy to contribute.

What do you think about separating the Issuers for internal and external use cases?

Another approach I thought of is allowing the _crunchyrepl user to be configured with a DNS suffix (e.g. _crunchyrepl.example.com). However, I have some security concerns about this approach, so I’m not sure it would be a good direction.

[egegunes]

Rather than adding an arbitrary suffix to _crunchyrepl user, I'd separate Issuers for internal and external but I'm not sure how the API should look like.

[yoav-katz]

@egegunes What do you think about

spec:
  tls:
    issuerConf:
      name: letsencrypt
      kind: ClusterIssuer
      group: cert-manager.io

    internalIssuerConf:
      name: postgres-internal
      kind: Issuer
      group: cert-manager.io

internalIssuerConf would be optional. When it is not specified, issuerConf is used for all certificates, preserving the current behavior.

This would allow us to use a separate issuer for the internal client certificates (_crunchyrepl and pgBackRest) without making the API more complicated or changing their identity semantics.

[yoav-katz]

For this, we’ll need to implement CA bundle concatenation so PostgreSQL can trust certificates issued by both CAs. Currently, the CA/chain is provided through tls.crt in the cert-manager-managed Secret, which we cannot modify. Therefore, we’ll need to create a new -ca-bundle Secret containing the combined CA bundle and use it for the relevant trust stores.

[yoav-katz]

I confirmed this against the cert-manager source: an ACME-backed Certificate yields
only tls.crt and tls.key. It can't be recovered from tls.crt either - ACME
returns leaf + intermediate with no root, and PostgreSQL won't anchor on a bare
intermediate.

We treat ca.crt as mandatory: it's what becomes ssl_ca_file, the replication
sslrootcert, patroni.ca-roots and pgbackrest.ca-roots.

So ACME requires the user to supply the anchor themselves. We already have this for
pgBouncer via additionalTrustedCAs - since the anchor is cluster-wide rather than
proxy-specific, I'd add it as spec.tls.additionalTrustedCAs.

I will open 2 new PR's:

1. spec.tls.additionalTRustedCAs - for:

issuerConf	ca.crt	additionalTrustedCAs	bundle
CA-type (today)	present	empty	just ca.crt — unchanged
ACME	absent	set	just the user anchor
CA-type + extra anchors	present	set	both

2. spec.tls.internalIssuerConf - will follow the same pattern for injecting the issuerConf ca.crt to ca-bundle and will be exclusivly for _cruncyrepl and pgbackrest certificates - it is optional and can be set only when issuerConf is set.

[egegunes]

I think adding spec.tls.internalIssuerConf makes sense, yes. @hors wdyt of adding this into other operators as well?

[yoav-katz]

I got blocked on the exat same problems in psmdb-operator (for clustername-mongos, clustername-rs0,clustername-config, clustername-rs0.namespace etc etc). as for my understanding mongodb requires all certificate san's on both ssl and internal-ssl so it may be problematic - but not actually sure about that.
havent checked mysql

I have created:
#1784
for the additionalTrustedCAs

[yoav-katz]

There is still one remaining problem with this design: issuerConf is still used to create certificates containing names such as clustername-primary, clustername-primary.namespace, clustername-replicas, etc.

I validated the code and found that only one place actually requires these non-FQDN names: the PgBouncer connection.
To make issuerConf fully compatible with ACME, I think we should use fully qualified DNS names for the generated certificates and add a flag that allows removing the short/non-FQDN names from the certificate.

This would allow ACME-backed issuerConf configurations to issue the primary and replica certificates without requesting invalid identifiers, while preserving the existing behavior for users who still rely on the short Kubernetes service names.

I haven't settled on the API yet, but my initial thought was something like spec.tls.fullFQDNOnly

TL;DR, we need to add:

1. spec.tls.additionalTrustedCAs feat(spec.tls.additionalTrustedCAs) #1784
2. spec.tls.internalIssuerConf
3. spec.tls.fullFQDNOnly

This is starting to become a much larger feature than I initially expected, with several API and certificate-management changes involved. So I completely understand if this scope is considered too large and the feature gets pushed back.