Agents need a way to store keys externally in Azure Key vault

[0xRBell]

We have a scenario where agents are using azure_imds node attestor but the nodes are AKS k8s nodes. These nodes have ephemeral OS disks that can be wiped/replaced on upgrades.
I have tested different locations on the disks and even in /var/lib the data gets wiped

Proposed solution:

• Add a new key_manager plugin for azure_key_vault
• All agents can share a key vault if needed
• On agent start up kick off a background job that will try to delete any keys that have an expiration older than max(customTTL, minimumTTLAllowed)
• Either move as much of the server azure key vault key_manager plugin code to common or use it to create an equivalent agent package

currently using this implementation to test with
https://github.com/0xRBell/spire/pull/2/files

[amartinezfayo]

Thank you, @0xRBell, for bringing this up.
One concern with a cloud-based agent KeyManager plugin is that a large number of agents may end up creating keys and performing transactions over those keys, which could result in significant costs.
Have you considered what the associated costs would look like at your scale if this plugin were used?

[0xRBell]

That is definitely a very valid concern. For our specific use case we have an agent running on a subset of nodes that only our spire server runs on. For us its essentially just the one workload for our k8s cluster level spire server to get its downstream workload from the parent. All other workloads use a cluster agent pointing to our cluster server.

[amartinezfayo]

Thank you @0xRBell.
We think that the project will benefit from supporting this kind of plugin. If we provide some warning in the documentation about potential costs, then we should be ok.
I've added this to the backlog, and we will welcome a contribution :)

[0xRBell]

Thank you. For our use case we will likely need both azure_key_vault as well as aws_kms plugins. Is this one issue sufficient for both or would you prefer I create two separate issues? I will get to the initial PRs likely after the new year.

[amartinezfayo]

That sounds great, thank you @0xRBell!

I think that it would be good to have separate issues.
If you have a branch where you are working on the implementations, please share that in the issues created, so we can provide some initial / early feedback about the implementation.
We have some learnings related with the aws_kms plugin in the server, where we should leverage the GetResources API to avoid scanning all the keys (#6269). Please consider that when working on the aws_kms plugin for the agent.