Feature proposal: kubernetes secret push

[egeland]

I have a little app that listens for the Confidant webhook and on updates, pulls secrets from Confidant and converts to and places them into Kubernetes Secrets format on a cluster.
The naming of the Confidant Service and Credentials map by convention to k8s clusters and namespaces.

I'm wondering if this feature would be acceptable to be incorporated into Confidant itself, with an option to enable the functionality?

If this sounds interesting, I'd be happy to provide a PR to make it happen.

[ryan-lane]

So this would be a different return format for get_service and get_credential?

[egeland]

More like another call out - like the webhook or graphite events do - directly to the k8s cluster (as specified by the name mapping), inserting the Secrets as needed.

The advantage with doing it inside Confidant itself would be that the user could get feedback on the status in the Confidant UI.
My current external app gives poor visibility into whether the update succeeded or not.
We'd need to tweak the UI a little to flash a "kubernetes update status", I guess.

[ryan-lane]

Ah. I thought you had k8s fetching from confidant, but it looks like you're pushing into k8s? A downside of k8s's secrets is that they aren't really encrypted at all. Have you considered adding a k8s plugin for accessing secrets from confidant directly?

[egeland]

Yes, I have an app that pulls credentials on changes (webhook as trigger) and pushes into k8s secret format into k8s.

Adding it as a plugin to k8s itself might be worthwhile, but seems like a bigger task. :)

[ryan-lane]

In the long-run having direct support in k8s is probably the best option. Otherwise, having confidant push directly into k8s will likely be less reliable than a daemon that can occasionally poll confidant while getting push notifications to make that polling loop quicker.

We're pretty hesitant to add a webhook to push the secrets across the wire, because it changes the threat model, and it means we'd probably want to make it reliable by doing the work async (using SQS or something along those lines to retry until successful), which adds a lot of complexity.

We've found polling to be really effective, especially since confidant is able to handle the load well (we poll every minute from every node).

Do you have a metrics system of some variety? What we do to track updates is send an event into the metrics system from confidant, so that we can know when an update was triggered, then send a metric from the nodes that tell us the metric was fetched, so that we can have them visible on the service dashboards, which lets us view confidant updates like service deploys.

A difficulty with adding something that tracks update status is that it adds a write on every service read, which wouldn't be very scalable.