Split Kubernetes into node-local and global agents

[therc]

Given that:

• Right now, the recommended installation is to have a dd-agent pod on each node, through a DaemonSet. That works reasonably well, save minor issues about regular apps discovering where the node-local statsd really is (I'm working to make this easier upstream).

• But also, if you collect events, kubernetes.yaml.example tells you to only do that from a single agent in the cluster. How would one do that? It's not trivial to do with just a DaemonSet, unless you resort to a StatefulSet or some label-based hack, which is brittle and prone to fail in unexpected ways.

• Last but not least, kubernetes_state.yaml.example does NOT tell you to only run the check from a single agent. Based on my experience, that causes only headaches. You get alerts from all the nodes in the cluster. I'm sure you folks aren't too thrilled about getting a lot of duplicate data, either.

I have a proposal to combine the above:

1. Keep running regular statsd/agents as a DaemonSet.
2. Recommend using Docker-based (i.e. based on image name) service discovery.The agent that lives on the same node where the kube-state-metrics pod is running will the lucky one to send kubernetes_state data. Make clear in the example that the check should only be run from one place in the cluster and that service discovery is the best way to do it. In other words, tell users that in most case configuring the check manually is not a good idea.
3. Have a separate Deployment, with one replica, which only runs checks that are global in scope:
   • event collection
   • control plane checks (conveniently, I just created More checks for the Kubernetes control plane #3112 for that)
   • anything else that needs to talk to the API servers

That's all very easy to implement, except for 3). The current check logic is:

• Get the node's list of pods from the Kubelet
• Perform Kubelet health checks
• Fetch cAdvisor/Docker metrics about every pod
• If event collection is enabled, call the API server and fetch them. Curiously enough, _process_events gets passed the list of pods, but it doesn't do anything with that.

The simplest fix is to have a new setting ("global checks"?), which prevents the check from talking to the Kubelet, Docker and cAdvisor. The new, alternate mode would only talk to the API servers and the rest of the control plane in order to collect events and control plane data.

Does it sound reasonable? Anything missing?

[cknowles]

I was looking at how we could utilise the new agent feature to collect events from multiple namespaces. What is the current recommendation if we want to collect events alongside the standard Daemon Set? Do you have any example YAMLs?

As a later follow up, once this is working it would be great to have it integrated into the Helm Chart.

[hkaj]

Hey @therc, @c-knowles

Thanks for the feedback. We are in the process of addressing the issue of cluster-level metrics/metadata/events, so stay tuned about this. Meanwhile, the current recommended way is to have two deployments for the agent:

• a daemon set that will collect node level data (so it should not have k8s event collection enabled). Note that this daemon set will take care of kube-state-metrics since this check is automatically enabled by Auto Discovery on the node where the kube-state-metrics pod is running. You don't need to write stuff to conf.d for it.
• a separate deployment with a single agent that collects events (and later other cluster-level stuff) that has a toleration for the NoSchedule taint (or any other taint that the daemon set does not tolerate).

We know it's not ideal since it requires planning ahead, and non-trivial configuration to deploy the agent. We are aiming to reduce the friction here, either by having a lightweight agent as you suggested, or by having a leader agent that would collect this cluster-level data on top of the rest.
Both solutions come with tradeoffs, we'll keep this issue updated with the state of things.

[cknowles]

@hkaj Thanks, I think we could add that to the Helm chart then so it's easier to setup. Is there any downside to running a Daemon Set plus a single replica Deployment without the taints? i.e. one node would have two agents. Or do the two agents interact in a way that causes issues?

[hkaj]

@c-knowles Running two agents on the same node can lead to wrong values for some metrics because it's not always clear in the backend what to do with two points that have the exact same metadata but a slightly different timestamp (should one replace the other, should they be averaged, summed, etc).
Also if a host metadata query fails for some reason (network blip, endpoint unavailable), you can have the agents report 2 hosts in datadog for the same node, messing up its metrics and increasing your bill.
Therefore we recommend against it.

[therc]

@hkaj I didn't update this issue since, but yes, the DaemonSet and Auto Discovery take care of the kubernetes and kube-state checks (and kube-dns, which was added recently). I was hoping to avoid taints for the event collection. Looking forward to the next set of improvements.

[a-chernykh]

Just wanted to confirm what @hkaj said - we're getting doubled system.disk.used periodically from the node which runs 2 DataDog agents - one in normal mode and another one in Kubernetes Events collection mode.