Make the medium for the `vault-env` volume configurable

[carsonoid]

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I agree to follow the Code of Conduct.

Problem Description

The medium for the vault-env volume is currently hard-coded to be Memory. This seems to have always been the case for the webhook.

Using Memory as the medium means that the ~26mb for the vault-env binary counts against the container memory limit. I can't see a reason for this as the default over a normal emptyDir.

This may not seem like much but if you have a node maxed out at 110 pods and each of them is using a tempfs backed volume for vault-env then that is 2,860 mb of ram being used that could have easily been on disk with no real performance or stability impact that I am aware of.

Proposed Solution

1. A top-level configuration that tells the webhook to not set medium: Memory for a by default.
2. A per-pod annotation that tells the webhook to not set medium: Memory for a specific pod.

Alternatives Considered

No response

Additional Information

No response

[csatib02]

Hey @carsonoid,

This looks like a straightforward and beneficial enhancement. If someone is planning to work on it, please ensure the changes are added to our new webhook repository: https://github.com/bank-vaults/secrets-webhook.