CRDs create conflicts when the cert-manager is replaced

[Sleepful]

What happened?

The issue looks like this:

cert-manager/cert-manager#3483

and it is necessary to remove all the old CRDs so that Pulumi does not conflict upon installing the cert-manager.

To solve the conflict it becomes necessary to first delete all the CRDs manually.

Example

cert-manager/cert-manager#3483

(these are the error messages during pulumi up)

  pulumi:pulumi:Stack (JoseV/doks/do-prod)
    isprod
    error: update failed
    error: kubernetes-cert-manager:index:CertManager resource 'cert-manager' has a problem: grpc: the client connection is closing


  kubernetes-cert-manager:index:CertManager$kubernetes:helm.sh/v3:Release (cert-manager-helm)
    error: Unable to continue with install: CustomResourceDefinition "certificaterequests.cert-manager.io" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "cert-manager-helm-19cd9255": current value is "cert-manager-helm-39ddf332"

Output of pulumi about

Dependencies:
NAME                              VERSION
@pulumi/digitalocean              4.41.0
@pulumi/docker                    4.6.2
@pulumi/kubernetes-ingress-nginx  0.1.3
@pulumi/kubernetes                4.22.1
@pulumi/pulumi                    3.160.0
@pulumi/kubernetes-cert-manager   0.2.0
@types/node                       18.19.86
ts-node                           10.9.2
typescript                        5.8.2

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rquitales]

@Sleepful are you able to provide some code and steps to help us repro this issue? Thanks

[Tungsten78]

I too am encountering this. cc: @aureq

There appears to related to a change in https://cert-manager.io/docs/releases/release-notes/release-notes-1.15/

The indicated helm option crds.keep=false is not currently exposed by the resource. Perhaps it should default to true for this resource?

As indicated by @Sleepful, the workaround is to manually remove all the crds. Workaround listed below for clarity.

kubectl delete crd certificaterequests.cert-manager.io
kubectl delete crd certificates.cert-manager.io
kubectl delete crd challenges.acme.cert-manager.io
kubectl delete crd clusterissuers.cert-manager.io
kubectl delete crd issuers.cert-manager.io
kubectl delete crd orders.acme.cert-manager.io

[Tungsten78]

Steps to reproduce is to pulumi up/down/up using the pulumi example.

[aureq]

Here is a quick reproduction, provided that you have a k8s cluster running accessible via KUBECONFIG.

Steps

1. Create a k8s cluster
2. Run the following app targeting your k8s cluster.
3. Everything is deployed as expected ✅
4. Change the namespace name
5. Deploy the changes (PULUMI_DEBUG_GRPC=./grpc.log p up --yes --skip-preview --verbose=9 --logtostderr 2> p-up.log). Update fails with the message ❌

error: kubernetes-cert-manager:index:CertManager resource 'cert-manager-deployment' has a problem: grpc: the client connection is closing

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as certmanager from "@pulumi/kubernetes-cert-manager";

export = async () => {

    const ns = new k8s.core.v1.Namespace(`cert-manager-namespace`, {
        metadata: {
            name: "cert-system",
        }
    });

    const cm = new certmanager.CertManager(`cert-manager-deployment`, {
        installCRDs: true,
        helmOptions: {
            namespace: ns.metadata.name,
        },
    }, { parent: ns });

}

Attached to this comment are the log files.

Notes

When destroying the stack, the following warning is shown indicating that CRDs aren't deleted.

warning: Helm uninstall returned information: "These resources were kept due to the resource policy:\n[CustomResourceDefinition] certificaterequests.cert-manager.io\n[CustomResourceDefinition] certificates.cert-manager.io\n[CustomResourceDefinition] challenges.acme.cert-manager.io\n[CustomResourceDefinition] clusterissuers.cert-manager.io\n[CustomResourceDefinition] issuers.cert-manager.io\n[CustomResourceDefinition] orders.acme.cert-manager.io\n"

For the update to succeed, one needs to run the command kubectl delete crd certificaterequests.cert-manager.io certificates.cert-manager.io challenges.acme.cert-manager.io clusterissuers.cert-manager.io issuers.cert-manager.io orders.acme.cert-manager.io

Logs

grpc.log
p-up.log

[EronWright]

To recap the issue, the cert-manager chart v1.15+ (PR) applies the helm.sh/resource-policy: keep annotation (see documentation) by default to the CRDs, so that they're orphaned when the release is uninstalled. A subsequent re-install of the chart then fails because Helm doesn't like to inherit preexisting objects unless they're prepped with the correct meta.helm.sh/release-name annotation.

I observe that the linked documentation warns about this problem, and is basically what we are experiencing here:

This can lead to problems if using helm install --replace on a release that has already been uninstalled, but has kept resources.

Another observation is that the cert-manager chart does not follow the recommended practice of installing CRDs using the special crds/ folder, also known as "Method 1: Let helm Do It For You". I do believe that a re-install would succeed because:

If the CRD already exists, it will be skipped with a warning. If you wish to skip the CRD installation step, you can pass the --skip-crds flag.

As for a workaround, I believe you can control the release name within helmOptions. By setting it to a constant, Helm will be willing to re-import the CRDs. The key is the meta.helm.sh/release-name annotation, which surprisingly isn't cleared by Helm when it keeps the object. Another workaround is to set crds.keep=false via helmOptions.values.