Skip to content

[kube-prometheus-stack] Admission webhook cross-namespace permission error when deployed via ArgoCD ApplicationSet with Kustomize namespace forcing #6304

@diomedesadmin

Description

@diomedesadmin

Describe the bug a clear and concise description of what the bug is.

The admission webhook job is created in the target namespace (e.g., monitoring-prod) but attempts to access secrets from the ArgoCD Application's namespace (argocd), resulting in a "forbidden" error and deployment failure.

What's your helm version?

version.BuildInfo{Version:"v3.17.3", GitCommit:"v3.17.3", GitTreeState:"", GoVersion:"go1.24.6"}

What's your kubectl version?

Client Version: v1.33.3 Kustomize Version: v5.6.0 Server Version: v1.33.5-eks-3cfe0ce

Which chart?

kube-prometheus-stack

What's the chart version?

79.0.1

What happened?

When deploying kube-prometheus-stack via ArgoCD ApplicationSet with Kustomize namespace forcing,
the admission webhook job fails with a cross-namespace permission error:

{"err":"secrets kube-prometheus-stack-admission is forbidden: User
system:serviceaccount:monitoring-prod:kube-prometheus-stack-admission cannot get resource
secrets in API group in the namespace argocd","level":"fatal","msg":"error getting secret"}

The admission webhook job runs in the target namespace (monitoring-prod) but tries to access
secrets from the ArgoCD Application's namespace (argocd), causing a permission denied error. The
job reaches backoff limit and fails repeatedly.

What you expected to happen?

What you expected to happen?
The admission webhook should either:

  1. Access secrets from the correct namespace (same as where the job runs)
  2. Have proper cross-namespace RBAC permissions
  3. Work seamlessly with ArgoCD's namespace isolation patterns

How to reproduce it?

How to reproduce it?
1.

Deploy ArgoCD ApplicationSet with Kustomize namespace forcing:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
  template:
    spec:
      source:
        kustomize:
          namespace: monitoring-{{path.basename}}
      destination:
        namespace: monitoring-{{path.basename}}
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
  1. Target kube-prometheus-stack chart to a namespace different from ArgoCD's namespace
  2. Observe admission webhook job failure with cross-namespace permission error

Enter the changed values of values.yaml?

NONE, empty file.

Enter the command that you execute and failing/misfunctioning.

ArgoCD ApplicationSet deployment (not direct helm command)

Anything else we need to know?

This appears related to ArgoCD's design where Applications live in the argocd namespace but
deploy to target namespaces. The admission webhook job expects to find secrets in the same
namespace as the ArgoCD Application object, not where the workload actually runs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions