ArgoCD Terraform Helm Install fails due to race condition in argocd-secret

[FalcoSuessgott]

Describe the bug

Trying to upgrade my cluster to 9.4.3 using Terraform and the Helm provider.
The Provider crashes with:

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to helm_release.argocd[0], provider "provider[\"registry.terraform.io/hashicorp/helm\"]" produced an unexpected new value:
│ .resources["secret/v1/argocd/argocd-secret"]: was
│ cty.StringVal("{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"server\",\"app.kubernetes.io/instance\":\"argo-cd\",\"app.kubernetes.io/name\":\"argocd-secret\",\"app.kubernetes.io/part-of\":\"argocd\",\"app.kubernetes.io/version\":\"v3.3.1\",\"helm.sh/chart\":\"argo-cd-9.4.3\"},\"name\":\"argocd-secret\",\"namespace\":\"argocd\"},\"type\":\"Opaque\"}"),
│ but now
│ cty.StringVal("{\"apiVersion\":\"v1\",\"data\":{\"server.secretkey\":\"<redacred>\"},\"kind\":\"Secret\",\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"server\",\"app.kubernetes.io/instance\":\"argo-cd\",\"app.kubernetes.io/name\":\"argocd-secret\",\"app.kubernetes.io/part-of\":\"argocd\",\"app.kubernetes.io/version\":\"v3.3.1\",\"helm.sh/chart\":\"argo-cd-9.4.3\"},\"name\":\"argocd-secret\",\"namespace\":\"argocd\"},\"type\":\"Opaque\"}").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

I even tried using another secret than argocd-secret for any sensitive data in my config, based on https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#alternative. But the terraform apply fails regardless.

Is this a known issue? Any work arounds available?

Related helm chart

argo-cd

Helm chart version

9.4.3

To Reproduce

terraform apply those resources:

resource "kubernetes_secret_v1" "argocd_oidc_secret" {
  count = var.argocd.enabled ? 1 : 0

  metadata {
    name      = "argocd-oidc-secret"
    namespace = "argocd"

    labels = {
      # https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#alternative
      "app.kubernetes.io/part-of" = "argocd"
    }
  }

  data = {
    "oidc.azure.clientSecret" = base64encode(jsondecode(data.aws_secretsmanager_secret_version.argocd_sso.secret_string).secret)
  }

  type = "Opaque"
}

resource "helm_release" "argocd" {
  count = var.argocd.enabled ? 1 : 0

  name             = "argo-cd"
  chart            = "argo-cd"
  namespace        = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  version          = "9.4.3"
  timeout          = "1500"
  create_namespace = true
  wait             = true

  values = [
    <<-EOT
    configs:
      cm:
        # OIDC config for Azure AD
        url: https://argocd.${local.domain_suffix}/ 
        oidc.config: |
          name: Azure
          issuer: https://login.microsoftonline.com/${jsondecode(data.aws_secretsmanager_secret_version.argocd_sso.secret_string).tenant}/v2.0
          clientID: ${jsondecode(data.aws_secretsmanager_secret_version.argocd_sso.secret_string).id}
          clientSecret: $argocd-oidc-secret:oidc.azure.clientSecret
          azure:
            useWorkloadIdentity: false 
          requestedIDTokenClaims:
            groups:
              essential: true
          requestedScopes:
            - openid
            - profile
            - email
    EOT
  ]

  depends_on = [module.eks, kubernetes_secret_v1.argocd_oidc_secret]
}

Expected behavior

I expect the terraform apply to succeed and to install the helm chart version 9.4.3

Screenshots

No response

Additional context

No response

[FalcoSuessgott]

Okay, I narrowed it down. It's not the argocd-helm chart, it's something in the TF Helm provider. I am able to install argoc helm chart version 9.4.3 using TF Helm Provider version 3.0.2, but with 3.1.1 & 3.1.0 it's crashing. Trying to find the change to understand what's happening (See diff).

[FalcoSuessgott]

Alright I think I found the issue:

In hashicorp/terraform-provider-helm@c8c5571 the helm provider experiment manifest = true was added:

provider "helm" {
  experiments = {
    manifest = true
  }
}

Which adds the fully rendered helm chart in terraforms plan output. Meaning it now also tracks all helm chart resources in its runtime state.

Since in the case of ArgoCD the argocd-secret is first created and then later modified (ArgoCD generates a server.secretkey and writes that to the argocd-secret) the provider crashes since the argocd-secret has diverged from what was shown in the initial TF plan.

The workaround is to remove the manifest experiment. I will open an issue in TF Helm repo to track it over there.

rel. hashicorp/terraform-provider-helm#829 hashicorp/terraform-provider-helm#1581 hashicorp/terraform-provider-helm#805