CA gets re-generated every time in ArgoCD

[chernetskyi]

ArgoCD does not support Helm lookup template function which is used by default to re-use the CA of the existing certificate.

Description

By default, Helm chart generates the certificate for the admission webhooks via Helm. However, it does so using a lookup template function to re-use the CA of the existing certificate:

k8s-agents-operator/charts/k8s-agents-operator/templates/_tls.tpl

Line 11 in 9fe1a4a

{{- $prevSecret := (lookup "v1" "Secret" .Release.Namespace (include "k8s-agents-operator.certificateSecret.name" . )) }}

The problem is lookup function does not work in ArgoCD.
argoproj/argo-cd#5202
argoproj/argo-cd#21745

Apparently, at one point, part of the setup relies on the certificate issued by the old CA, and another - by the new one. Not sure when or why this happens, but the regular ArgoCD application sync (that among other things re-creates the CA) is not enough.

When this happens, admission webhooks stop working, and the following log message can be observed in the manager container's logs:

http: TLS handshake error from 172.24.76.230:33540: remote error: tls: bad certificate

To temporarily address the issue, we sync ArgoCD application with prune on.

Steps to Reproduce

Sorry, not exactly sure. Tried syncing ArgoCD application with prune off and it was not enough by itself to reproduce the issue.

Expected Behavior

Because I do not think ArgoCD is an exotic environment, I expect the default configuration of the chart to work in ArgoCD.

I know the chart actually provides two other options to manage the certificates that should both work in ArgoCD. Because of them, I would at least expect this issue mentioned anywhere in the documentation. Something like

k8s-agents-operator Helm chart in its default configuration will not work correctly in ArgoCD. Please configure one of the two alternative methods to manage the certificates for admission webhooks.

Since we also know syncing with prune on resolves the issue temporarily, maybe the recommendation can be to turn enable pruning with automatic syncs:
https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/#automatic-pruning
However, I am not sure about the consequences of such solution.

Relevant Logs / Console output

The following log message can be observed in the manager container's logs:

http: TLS handshake error from 172.24.76.230:33540: remote error: tls: bad certificate

Your Environment

• EKS 1.33
• ArgoCD v3.1.1+fa342d1
• nri-bundle 6.0.26

Additional context

When was browsing through GitHub issues, found out that k8s-metadata-injection chart, for example, defaults to using Jobs to generate certificates, instead of generating them via Helm (does not even offer it as an option). I do not know why the default approaches differ in Helm charts, but unlike relying on lookup, Jobs should work in ArgoCD.

[Newrelic-Boomi]

https://new-relic.atlassian.net/browse/NR-504589

[TmNguyen12]

Thanks for the detailed issue report and bringing this to our attention.

Have you tried using cert-manager to manage certificates?
Here's how you'd configure it:

1. Install cert-manager in your cluster:

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

2. Update your values.yaml for k8s-agents-operator:

  admissionWebhooks:
    autoGenerateCert:
      enabled: false
    certManager:
      enabled: true

I'll take this back to the team to discuss options for handling ArgoCd compatibility or updating our documentation with recommendations on how best to work with ArgoCd.

Thank you.

[chernetskyi]

@TmNguyen12 I think enough time has passed since we deployed cert-manager to say that it solves the problem. However, I want to note the cert-manager solution was always known and we opened this issue highlight the problem with the default configuration, not ask for the solution to it.