Namespaces not taken into account

[maximemf]

Hello,

We're just starting to use your plugin, and we're facing an issue. Maybe we're missing something.

We have a Helmfile configuration with many Helm releases to be installed in different namespaces. The namespaces are defined in the Helmfile configuration for each release.

But when we want to synchronize the Argo Application related to this Helmfile (with no namespace defined in the Application), we have errors like this for each resource:

After some investigations, these errors only happen for Helm charts that don't define the namespace metadata field in the charts templates.
There is no error when the field that corresponds to the namespace in the metadata section is defined in the templates, in this case the right namespace defined in the Helmfile configuration is taken into account. But most of the Helm charts are not defining this value by default in the templates...

It seems to not be able to retrieve or to apply the namespace from the context, that we define in the Helmfile configuration, if the namespace field is not already present in the chart.

Do you have any idea about what could be the issue?

[travisghansen]

Try setting the env HELMFILE_USE_CONTEXT_NAMESPACE=true and see what you get. The readme has a few details and/or look at the src to see how it changes the output.

[maximemf]

From what I understand, it's only useful if the namespace is set in the Application, to kinda ignore it and use the context instead, but that's not even our case. We tried setting this variable nonetheless, but it didn't change anything.
I guess we'll have to debug the script manually to find where the issue could come from 😕

[travisghansen]

Ok, given that we’re templating everything entirely and then argo applies the values in a single operation it posses a challenge (to do something like post-processing) because the context is lost at that point.

I think helmfile has an ability to do a post render hook on a per release basis where hypothetically you could find all namespaced assets and try to inject the ns if not present.

[maximemf]

I've done some research and I understand better the origin of these errors, with helm template not using the namespace from the context for some obscure reasons that seem to be a great source of debate in the community (for instance helm/helm#3553).

But after some additional investigations, I found out that Helmfile already implements a "secret feature" to get around this issue, which is the forceNamespace parameter, to be defined for each release. This parameter makes Helmfile complete the manifests generated by helmfile template with the namespace from the configuration (see roboll/helmfile#326 (comment) 🙏).

So our errors have been fixed, thank you for your reactivity, we'll keep using your great plugin, and come back to you if we face any further issue 😄

[travisghansen]

Wow great research and references! Thanks for sharing your findings.

[SirSilly]

Hello,

Thank you for sharing issue with namespaces - really helped me solving the same issue this issue creator stumbled on.
But after that I do have another issue:

When I want to create / sync for the first time, namespaces which are provided in helmfile

namespace: test
forceNamespace: test

are not being created and argo-cd throws an error that "Sync Failed", because "Nnamespaces "test" not found"

Am I missing some additional flag or env added to deployment? I am currently using only HELMFILE_GLOBAL_OPTIONS="-e test" as it is environment based deployment.

Thank you.

[travisghansen]

Does this help that? https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#create-namespace

[SirSilly]

Yes, that helps.
Thank you!

So just to be clear.. It is not enough to simply specify namespaces in helmfile and additional stuff (as mentioned in your answer) needs to be deployed. Can this be added to helmfile deployment or has to be done separately, I mean before deployment?

I should note that my helmfile contains multiple namespaces that needs to be created.

[maximemf]

That's another issue we also faced after using the forceNamespace parameter of Helmfile, and we were not able to fix it with the CreateNamespace option of Argo.

https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#create-namespace

Note that the namespace to be created must be informed in the spec.destination.namespace field of the Application resource.

As we have a multi-namespaces Application, we shouldn't set the spec.destination.namespace field, and so the CreateNamespace option is not working.

We didn't work more on this yet, but I guess we'll have to craft something as a workaround, like preInstall Helmfile hooks to create the namespaces, but if anyone have a better idea I'll take it 🙂

[SirSilly]

What have I did to go as a workaround:

{{- if .Values.namespaces.test.enabled }}
apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Values.namespaces.test.name }}
{{- end }}

and then added in values files values/keys respectively. Not the cleanest workaround, but it works.
Although, another issue appears. For example my full helmfile deployment contains prometheus-stack deployment and for some reason argo-CD does not install CRDs, even if I set in prometheus values file installCRDs: true

[j-wozniack]

If any one was looking at this and still needed ideas. I tried using the HELMFILE_INIT_SCRIPT_FILE, just created a basic bash script. It can create namespaces on remote clusters or just locally in the cluster. I use environment variables so this determines which releases you want to target and grabs the namespace to create and the hooks to run on each release.

It is a little ugly, but it runs quickly, and efficiently.

create-ns.sh

#!/bin/bash
function KUBE_CONF {
    KUBE_CONFIG_PATH="/tmp/kube.yaml"
    # Get config
    SECRETCONFIGJSON=$(kubectl get secret "server-${ENVIRONMENT}" -n argocd -o jsonpath='{.data.config}' | base64 --decode | jq .)
    CLIENTCERT=$(echo ${SECRETCONFIGJSON} | jq '.tlsClientConfig.certData')
    CLIENTKEY=$(echo ${SECRETCONFIGJSON} | jq '.tlsClientConfig.keyData')
    CA=$(echo ${SECRETCONFIGJSON} | jq '.tlsClientConfig.caData')
    # Get Server
    SECRETSERVER=$(kubectl get secret "server-${ENVIRONMENT}" -n argocd -o jsonpath='{.data.server}' | base64 --decode) 

    CONFIG="---
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: ${CA}
        server: ${SECRETSERVER}
    name: default
    contexts:
    - context:
        cluster: default
        user: default
    name: default
    current-context: default
    kind: Config
    preferences: {}
    users:
    - name: default
    user:
        client-certificate-data: ${CLIENTCERT}
        client-key-data: ${CLIENTKEY}
    "
    # Create temporary kubeconfig
    echo "$CONFIG" | yq eval-all '.' > /tmp/kube.yaml
    export KUBECONFIG="${KUBE_CONFIG_PATH}"
}

function RUN_HELM_HOOKS {
    HELMFILE_LOCATION="${PWD}/helmfile.yaml"
    # Start helmfile hooks
    HELMFILE_RELEASES=$(helmfile -f ${HELMFILE_LOCATION} list ${HELMFILE_GLOBAL_OPTIONS} --output json | jq -r '.[] | select(.enabled) | "\(.name)=\(.namespace)"')
    for r in $HELMFILE_RELEASES; do
        NAME=$(echo "$r" | grep -oE "^[^=]+")
        NS=$(echo "$r" | grep -oE "[^=]+$")
        # Check NS
        kubectl get ns "$NS" &>/dev/null
        if [[ $? -ne 0 ]]; then
            kubectl create ns "$NS"
        else
            continue
        fi

        # Check Hooks
        HELMFILE_HOOKS=$(rname="$NAME" yq eval '.releases[] | select(.name == env(rname)) | .hooks.[].command' ${HELMFILE_LOCATION})
        
        if [ -z "$HELMFILE_HOOKS" ]; then
            continue
        fi

        for h in $HELMFILE_HOOKS; do
            h=$(basename "$h")
            if [ $h == "Your hook name 1" ]; then
                # Run your hook
            fi
            if [ $h == "Your hook name 2" ]; then
               # Run the second hook
            fi
        done
    done
}

function CLEAN_UP {
    unset KUBECONFIG
    rm -f $KUBE_CONFIG_PATH
}

# Get the environment based on what you set "-e" to
ENVIRONMENT=$(echo "$HELMFILE_GLOBAL_OPTIONS" | grep -oP '\-e \K.*')

# IF it is a local deployment in the cluster, only run helm hooks. The pod knows the cluster config
if [[ "${ENVIRONMENT}" == *"Your local environment variable"* ]]; then
    RUN_HELM_HOOKS
else
    KUBE_CONF
    RUN_HELM_HOOKS
fi

CLEAN_UP

Just mount this in the helmfile plugin pod, and set the parameters in the app:

      - name: HELMFILE_INIT_SCRIPT_FILE
        value: '/tmp/scripts/create-ns.sh'