Agents need a way to store keys externally in AWS KMS #6508
Description
There is a need for agents running on EKS clusters to be able to store their keys in kms. Since cluster nodes can wipe the OS disk on upgrades or restarts its not safe to use hostPath mounts to store keys using disk keymanager plugin.
Ideally the agent can leverage the same kms behavior the server uses for storing its keys.
note:
This should not be used with large workload counts as cost could quickly grow out of control.
example use case:
There is a spire server running on a k8s cluster that is a downstream of another spire server. This cluster level server needs to get a workload from an agent that is configured to attest to the upstream spire server. When using the aws_iid node attestor there are scenarios when the nodes can wipe their OS disks and therefor delete the data and key dirs resulting in orphaned agents that can't re-attest unless they are evicted from the upstream.
Allowing the agent to store its keys external from the node mitigates this issue.
During design phase reference learnings from server plugin list here #6269