[BUG] cluster-agent metricsProvider RoleBinding for extension-apiserver-authentication-reader created in wrong namespace

[noizo]

Pre-submission Checklist

• I have searched existing issues and this is not a duplicate
• This is a Helm chart issue, not a Datadog product/service problem

Helm chart name

datadog

Helm chart version

3.88.3 - 3.160.2, current main as well

Bug Report

What happened:
When clusterAgent.metricsProvider.enabled: true and the chart is installed outside kube-system (e.g. datadog-system), the RoleBinding <release>-cluster-agent-apiserver is created in {{ .Release.Namespace }} but references the Role extension-apiserver-authentication-reader, which only exists in kube-system.

RoleBindings are namespace-scoped and can only reference Roles in the same namespace, so the binding silently does nothing. The Cluster Agent's external metrics API server (port 8443) cannot read the extension-apiserver-authentication ConfigMap and rejects all requests with 403 Forbidden:

RBAC: role.rbac.authorization.k8s.io "extension-apiserver-authentication-reader" not found

This breaks the DatadogMetric CRD flow — all DatadogMetric CRs remain Active: False, Valid: False indefinitely.

The bug is in charts/datadog/templates/cluster-agent-rbac.yaml:

apiVersion: {{ template "rbac.apiVersion" . }}
kind: RoleBinding
metadata:
  name: "{{ template "datadog.fullname" . }}-cluster-agent-apiserver"
  namespace: {{ .Release.Namespace }}          # <-- should be kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
  - kind: ServiceAccount
    name: {{ template "datadog.fullname" . }}-cluster-agent
    namespace: {{ .Release.Namespace }}

What I expected:
The RoleBinding should be created in kube-system where the extension-apiserver-authentication-reader Role exists. This is the standard pattern used by other extension API servers (metrics-server, KEDA).

Steps to Reproduce

1. Install the chart in a namespace other than kube-system:

   helm install datadog datadog/datadog -n datadog-system --set clusterAgent.metricsProvider.enabled=true --set clusterAgent.metricsProvider.useDatadogMetrics=true
   

2. Create a DatadogMetric CR
3. Observe it stays Active: False, Valid: False indefinitely
4. Verify the role is missing in the release namespace:

   kubectl get role extension-apiserver-authentication-reader -n kube-system      # exists
   kubectl get role extension-apiserver-authentication-reader -n datadog-system   # NotFound

Environment

Kubernetes version: v1.32 (EKS)
Helm version: v4.1.3
Cloud provider: AWS

Additional Context

This likely hasn't been widely reported because most deployments install Datadog in kube-system (where the Role exists) or use the Datadog Operator (which manages its own RBAC).

Suggested fix — change the namespace to kube-system:

apiVersion: {{ template "rbac.apiVersion" . }}
kind: RoleBinding
metadata:
  name: "{{ template "datadog.fullname" . }}-cluster-agent-apiserver"
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
  - kind: ServiceAccount
    name: {{ template "datadog.fullname" . }}-cluster-agent
    namespace: {{ .Release.Namespace }}

[gabedos]

Hi @noizo thanks for reaching out about this flaw in the helm-chart.

The extension-apiserver-authentication-reader role does exist in the kube-system namespace and thus requires the rolebinding to also be created in that same namespace to properly work. In practice with modern charts, however, this should not be a problem because we now add the permission in the cluster agent's role directly. We add a permission that is scoped down to just resourceName: extension-apiserver-authentication here.

I was able to verify the cluster-agent service account is able to reach the configmap via:

% k auth can-i get configmap/extension-apiserver-authentication --as=system:serviceaccount:datadog-agent:datadog-agent-linux-cluster-agent -n kube-system
yes
% k auth can-i list configmap/extension-apiserver-authentication --as=system:serviceaccount:datadog-agent:datadog-agent-linux-cluster-agent -n kube-system
yes
% k auth can-i watch configmap/extension-apiserver-authentication --as=system:serviceaccount:datadog-agent:datadog-agent-linux-cluster-agent -n kube-system
yes
% k auth can-i delete configmap/extension-apiserver-authentication --as=system:serviceaccount:datadog-agent:datadog-agent-linux-cluster-agent -n kube-system
no

Can you please share any other specific bugs you have be having for setting up the Agent with the metric provider? Thanks

[noizo]

Confirmed: ClusterRole fix is sufficient, RoleBinding is unnecessary

I hit this issue while enabling the Datadog Cluster Agent metricsProvider for KEDA integration on EKS clusters (Datadog installed in datadog-system, not kube-system).

My experience

I initially misdiagnosed the problem. When testing the metrics API on port 8443, I saw 403 errors and found the buggy datadog-cluster-agent-apiserver RoleBinding that this issue describes — created in the release namespace (datadog-system) but referencing the extension-apiserver-authentication-reader Role which only exists in kube-system. I incorrectly attributed the 403 to this broken RoleBinding and added a manual RoleBinding in kube-system as a workaround.

After further investigation, I confirmed that the 403 was from test probes running as system:anonymous, not from the cluster agent's own auth chain. The ClusterRole already has the fix — lines 93-104 in cluster-agent-rbac.yaml:

- apiGroups: [""]
  resourceNames: [extension-apiserver-authentication]
  resources: [configmaps]
  verbs: [get, list, watch]

This ClusterRole rule, combined with the existing ClusterRoleBinding, provides the necessary access. The namespace-scoped RoleBinding is redundant.

Validation

On my cluster (upstream chart datadog-3.160.2, internal chart datadog-1.35.0, agent 7.74.1):

1. Deleted the manual kube-system RoleBinding workaround
2. Restarted the cluster agent pod
3. Auth controllers (RequestHeaderAuthRequestController, client-ca) synced successfully
4. Created a test HPA referencing a DatadogMetric CR
5. Result: Active=True, Valid=True, currentValue=112826297.88 — real data from DD API
6. No auth errors in cluster agent logs

The buggy datadog-cluster-agent-apiserver RoleBinding in the release namespace is still present and harmless (binds to a non-existent Role, so it's a no-op). The ClusterRole covers everything.

Suggestion

This issue can likely be closed as resolved. The ClusterRole-based fix in modern chart versions fully addresses the access requirement. The broken namespace-scoped RoleBinding is a cosmetic issue at most — it does nothing, but it also breaks nothing. Cleaning it up in the chart template would be nice for clarity, but it's not functionally required.

As @gabedos noted in the earlier comment, the resourceName-scoped ClusterRole rule is the intended fix.