Replace jx-git-operator with Argo CD

[roulettedares]

The release pipeline on the gitops cluster repo (bootjob) has a few issues

• it does not truly run in parallel. as the number of namespaces and installed charts grows, the pipeline duration gets exponentially longer
• a single kubectl apply failure will stop the entire pipeline before applying all valid k8s manifests
• failure notifications can be sent via slack, but nobody is pinged directly (at least i can't get the directmessage settings to work)
• the makefile that runs during the release pipeline is spaghetti and not very flexible

We run ArgoCD to sync the our non-jenkins-x config repo to our preprod environment and it is very intuitive and flexible. There is a proposal in the kubernetes slack #jenkins-x-dev channel to replace the above process with Argo CD

The most basic POC could look something like

1. install the ArgoCD helm chart via terraform with lifecycle configuration to ignore all future changes. normally, we would manage helm releases via helmfile, but because we need to bootstrap the cluster and run a first ArgoCD sync, we can use the terraform helm provider

   resource "helm_release" "argocd_bootstrap" {
     chart            = "argo-cd"
     create_namespace = true
     namespace        = var.namespace
     name             = "argocd"
     version          = "5.5.7"
     repository       = "https://argoproj.github.io/argo-helm"
     values = [
       jsonencode(
         {
           "controller" : {
             "serviceAccount" : {
               "annotations" : {
                 "iam.gke.io/gcp-service-account" : "argocd-${var.cluster_name}@${var.gcp_project}.iam.gserviceaccount.com"
               }
             },
           },
           "repoServer" : {
             "autoscaling" : {
               "enabled" : true,
               "minReplicas" : 2
             },
             "initContainers" : [
               {
                 "name" : "download-tools",
                 "image" : "ghcr.io/helmfile/helmfile:v0.147.0",
                 "command" : [
                   "sh",
                   "-c"
                 ],
                 "args" : [
                   "wget -qO /custom-tools/argo-cd-helmfile.sh https://raw.githubusercontent.com/travisghansen/argo-cd-helmfile/master/src/argo-cd-helmfile.sh && chmod +x /custom-tools/argo-cd-helmfile.sh && mv /usr/local/bin/helmfile /custom-tools/helmfile"
                 ],
                 "volumeMounts" : [
                   {
                     "mountPath" : "/custom-tools",
                     "name" : "custom-tools"
                   }
                 ]
               }
             ],
             "serviceAccount" : {
               "annotations" : {
                 "iam.gke.io/gcp-service-account" : "argocd-${var.cluster_name}@${var.gcp_project}.iam.gserviceaccount.com"
               }
             },
             "volumes" : [
               {
                 "name" : "custom-tools",
                 "emptyDir" : {}
               }
             ],
             "volumeMounts" : [
               {
                 "mountPath" : "/usr/local/bin/argo-cd-helmfile.sh",
                 "name" : "custom-tools",
                 "subPath" : "argo-cd-helmfile.sh"
               },
               {
                 "mountPath" : "/usr/local/bin/helmfile",
                 "name" : "custom-tools",
                 "subPath" : "helmfile"
               }
             ]
           },
           "server" : {
             "autoscaling" : {
               "enabled" : true,
               "minReplicas" : 2
             }
             "ingress" : {
               "enabled" : true,
               "annotations" : {
                 "nginx.ingress.kubernetes.io/backend-protocol" : "HTTPS",
                 "nginx.ingress.kubernetes.io/force-ssl-redirect" : "true",
                 "nginx.ingress.kubernetes.io/ssl-passthrough" : "true"
               },
               "hosts" : [
                 "argocd.${var.apex_domain}"
               ],
               "serviceAccount" : {
                 "annotations" : {
                   "iam.gke.io/gcp-service-account" : "argocd-${var.cluster_name}@${var.gcp_project}.iam.gserviceaccount.com"
                 }
               }
             }
           }
         }
       )
     ] 
   
     set {
       name  = "server.config.configManagementPlugins"
       value = <<-EOT
       - name: helmfile
         init:                          # Optional command to initialize application source directory
           command: ["argo-cd-helmfile.sh"]
           args: ["init"]
         generate:                      # Command to generate manifests YAML
           command: ["argo-cd-helmfile.sh"]
           args: ["generate"]
       EOT
     }
     set {
       name  = "configs.credentialTemplates.https-creds.url"
       value = regex("\\w+://\\w+\\.\\w+", var.jx_git_url)
     }
     set_sensitive {
       name  = "configs.credentialTemplates.https-creds.username"
       value = var.jx_bot_username
     }
     set_sensitive {
       name  = "configs.credentialTemplates.https-creds.password"
       value = var.jx_bot_token
     }
   
     dynamic "set" {
       for_each = var.helm_settings
       content {
         name  = set.key
         value = set.value
       }
   
     lifecycle {
       ignore_changes = all
     }
   }

2. use terraform to configure ArgoCD to sync the config-root folder of the dev gitops repo to the dev gke cluster. maybe we can package this as a separate helm chart called argo-cd-apps or something

     - apiVersion: argoproj.io/v1alpha1
       kind: ApplicationSet
       metadata:
         name: dev
       spec:
         generators:
         - git:
             repoURL: https://github.com/{{.Values.jxRequirements.cluster.environmentGitOwner}}/{{.Values.jxRequirements.environments.0.repository}}
             revision: HEAD
             directories:
             - path: helmfiles/*
             # - path: config-root/customresourcedefinitions
             # - path: config-root/namespaces/*
         template:
           metadata:
             name: '{{path.basename}}'
           spec:
             project: default
             source:
               repoURL: https://github.com/{{.Values.jxRequirements.cluster.environmentGitOwner}}/{{.Values.jxRequirements.environments.0.repository}}
               targetRevision: HEAD
               path: '{{path}}'
               plugin:
                 env:
                 - name: HELMFILE_USE_CONTEXT_NAMESPACE
                   value: "true"
                 - name: HELM_TEMPLATE_OPTIONS
                   value: --skip-tests
             destination:
               server: https://kubernetes.default.svc
               namespace: '{{path.basename}}'
             syncPolicy:
               automated:
                 prune: true
                 selfHeal: true
               syncOptions:
               - CreateNamespace=true

   • We should probably figure out whether we want to continue rendering kubernetes templates and committing them back to the cluster repo at PR time or if we should just use argo to sync directly from the helmfile. There's an example bot and a github action that post the output of helmfile diff or argo diff as a PR comment
3. After the first sync, ArgoCD manages its own helm chart installation

i have some rough ideas here:
jenkins-x/terraform-google-jx#228
https://github.com/joshuasimon-taulia/helm-argo-cd-apps/commit/4572c611887190bc4f9640e177a8e902ff7b6558

this is what the demo appset generates in my atlantis project

when you click on the "namespace" application, the ui drills down into your actual k8s objects

[keskad]

Hi, thanks for taking my idea to the table, I'm so happy because of that :) I hope I will be able to help you somehow with it soon. Atm I can jump into the discussion at least.

1. In my opinion the target behavior could be to not render the YAML's and to use helmfiles in ArgoCD directly. That would allow us to remove a lot of the jx gitops commands code, have less to maintain - the jx cli already has too many commands.

2. Secondly I think we should go into a direction to separate two groups of resources from each other and treat them separately.

a) First group is for stateful resources and resources that have recursive dependencies. Eamples: StatefulSet, Namespace, PVC. Those kinds of resources should be protected from accidential pruning using ArgoCD's built-in mechanisms

b) Second group could be the rest of resources that should be pruned automatically and synced automatically.

3. To protect against mistakes I suggest to configure few kind: AppProject objects with restrictions like following examples:

• customresourcedefinitions: should contain only objects of type CRD
• secrets: Should contain only objects of type ExternalSecret

That allows to raise an alert if we accidently put an object to a place where it should not be.

[keskad]

Going further;

I think that we could try to remove project's configuration in ConfigMap form from the bootrepo and replace them with CRD's eg. JXProject. When ArgoCD will sync those CRD's to the cluster then:

1. New dedicated controller will discover newly applied CRD's JXProject
2. Glue all CRD's JXPRoject from the cluster into a configmap that the Lighthouse will understand and apply to the cluster
3. Create/update the webhooks to the Github/Gitlab/Gitea/Bitbucket using go-scm

That would result in:

• Less merge conflicts in bootrepo
• Project configuration in bootrepo would be easier to understand
• Easier development projects removal from the JX
• Possibility to list projects within kubectl on the cluster and playing with the configuration as part of testing something
• Much easier procedure of importing a new project

[keskad]

Regarding the ArgoCD installation I would leave a possibility to have a custom way of installation, because there are multiple ways of installing ArgoCD, including by kustomize, by helm, using operator.

Probably best would be to split it in docs into two steps:

1. How to install ArgoCD (eg. single patch with a Helm Chart)
2. How to install JX on top of ArgoCD

The second point should be universal and independent of the type of ArgoCD installation.
I have worked in a project, where we were using an ArgoCD operator and installing dozens of instances with its specific settings.

[keskad]

I just had loose a thought about the JXProject CRD - introducing it we could give users the possibility to extend Jenkins X with their own operators.

For example people will be able to create operators reacting on project creation/deletion to e.g. register it in monitoring, or even in tasks management software (like pipeline status integration in some external systems).

And more simpler form - ArgoCD has a post-sync hooks, a post-sync hook can be used to run e.g. Python script evertime some project in ArgoCD is synced.

I see a lot of benefits of moving from bootjob to ArgoCD 😄

[roulettedares]

@ankitm123 @msvticket @tomhobson how do you feel about

• replacing chore: regenerate commit backs to the gitops repo with pr comments that contain the output helmfile diff or argo diff. ex: bot and a github action that post the output of as a PR comment
• my implementation with https://github.com/travisghansen/argo-cd-helmfile generates 1 top-level ArgoCD Application object per k8s namespace which contains all k8s objects rendered by the respective helmfile. there is currently no good way to loop through the releases array in helmfile.yaml in the an argo ApplicationSets or split the output of the ConfigManagementPlugin into multiple argo applications. ideally for organizational purposes, we would create 1 argo Application per helm chart release in helmfile.yaml. you can see what i mean in the above screenshot