Pre-submission Checklist
Helm chart name
datadog
Helm chart version
3.145.1 (also confirmed on the latest, 3.228.0)
Bug Report
What happened:
The datadog chart lets you deliver check configurations to the node agents (datadog.confd) and to the cluster agent (clusterAgent.confd / clusterAgent.advancedConfd), but there is no equivalent for the cluster check runners. clusterChecksRunner has many options, but none of them provides check configuration (there is no confd). The only way to place a check config into the runner's conf.d is to build a ConfigMap yourself and mount it via clusterChecksRunner.volumes / volumeMounts.
Our concrete need comes from a production incident. Cluster check runners run their own forwarder, and it can drop metric payloads under load (buffer overflow / backpressure), the same way node agents can. On large clusters a runner handling a heavy check like kube_state_core (pod metrics) is especially exposed. We hit this: kubernetes_state.pod.* metrics were undercounting on a large cluster because the runner's forwarder was dropping over half of its payloads at peak. It was silent, because the forwarder metrics (datadog.forwarder.transactions.dropped, .high_priority_queue_full) are only collected on the node agents by default (we add the go_expvar config through datadog.confd), not on the cluster check runners.
What I expected:
A clusterChecksRunner.confd (and/or advancedConfd) option, mirroring datadog.confd and clusterAgent.confd, so check configs can be delivered to the cluster check runner conf.d through Helm values, without hand-building and mounting a custom ConfigMap. This gives alerting parity between node agents and cluster check runners.
Steps to Reproduce
- Deploy the datadog chart with
clusterChecksRunner.enabled: true.
- Try to add a check config to the runners (for example a
go_expvar config at conf.d/go_expvar.d/conf.yaml) through Helm values.
- There is no
clusterChecksRunner.confd. The only option is to create your own ConfigMap and mount it via clusterChecksRunner.volumes / volumeMounts.
Environment
Kubernetes version: v1.35.5-eks-0247562 (EKS)
Helm version: v3.18.4
Cloud provider: AWS (EKS)
Additional Context
Current workaround: a custom ConfigMap with the go_expvar config (the forwarder/Transactions/* paths, namespace datadog) mounted into the runner pods via clusterChecksRunner.volumes / volumeMounts into conf.d/go_expvar.d. It works, but it is fragile: Helm replaces list values across values files, so it collides with any other clusterChecksRunner.volumes override (we have a cluster that mounts a secret for another cluster check), which forces us to duplicate the go_expvar volume in those files to avoid losing it.
Impact: dropped payloads on cluster check runners go undetected by default. In our case pod metrics were undercounting before anyone noticed, and we only found the cause through manual investigation (flares, agent status). We also built a custom script to poll every cluster's runners to find which ones were dropping. Anyone running cluster check runners at scale has the same blind spot out of the box, plus the maintenance and fragility of the ConfigMap workaround.
This is also being tracked with Datadog Support (case 2915840).
Pre-submission Checklist
Helm chart name
datadog
Helm chart version
3.145.1 (also confirmed on the latest, 3.228.0)
Bug Report
What happened:
The datadog chart lets you deliver check configurations to the node agents (
datadog.confd) and to the cluster agent (clusterAgent.confd/clusterAgent.advancedConfd), but there is no equivalent for the cluster check runners.clusterChecksRunnerhas many options, but none of them provides check configuration (there is noconfd). The only way to place a check config into the runner'sconf.dis to build a ConfigMap yourself and mount it viaclusterChecksRunner.volumes/volumeMounts.Our concrete need comes from a production incident. Cluster check runners run their own forwarder, and it can drop metric payloads under load (buffer overflow / backpressure), the same way node agents can. On large clusters a runner handling a heavy check like
kube_state_core(pod metrics) is especially exposed. We hit this:kubernetes_state.pod.*metrics were undercounting on a large cluster because the runner's forwarder was dropping over half of its payloads at peak. It was silent, because the forwarder metrics (datadog.forwarder.transactions.dropped,.high_priority_queue_full) are only collected on the node agents by default (we add thego_expvarconfig throughdatadog.confd), not on the cluster check runners.What I expected:
A
clusterChecksRunner.confd(and/oradvancedConfd) option, mirroringdatadog.confdandclusterAgent.confd, so check configs can be delivered to the cluster check runnerconf.dthrough Helm values, without hand-building and mounting a custom ConfigMap. This gives alerting parity between node agents and cluster check runners.Steps to Reproduce
clusterChecksRunner.enabled: true.go_expvarconfig atconf.d/go_expvar.d/conf.yaml) through Helm values.clusterChecksRunner.confd. The only option is to create your own ConfigMap and mount it viaclusterChecksRunner.volumes/volumeMounts.Environment
Kubernetes version: v1.35.5-eks-0247562 (EKS)
Helm version: v3.18.4
Cloud provider: AWS (EKS)
Additional Context
Current workaround: a custom ConfigMap with the go_expvar config (the
forwarder/Transactions/*paths, namespacedatadog) mounted into the runner pods viaclusterChecksRunner.volumes/volumeMountsintoconf.d/go_expvar.d. It works, but it is fragile: Helm replaces list values across values files, so it collides with any otherclusterChecksRunner.volumesoverride (we have a cluster that mounts a secret for another cluster check), which forces us to duplicate the go_expvar volume in those files to avoid losing it.Impact: dropped payloads on cluster check runners go undetected by default. In our case pod metrics were undercounting before anyone noticed, and we only found the cause through manual investigation (flares,
agent status). We also built a custom script to poll every cluster's runners to find which ones were dropping. Anyone running cluster check runners at scale has the same blind spot out of the box, plus the maintenance and fragility of the ConfigMap workaround.This is also being tracked with Datadog Support (case 2915840).