Application controller can run out of disk space on '/dev/shm', when deploying concurrently to a large number of clusters, or with many large manifests #19201
Description
Checklist:
- I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- I've included steps to reproduce the bug.
- I've pasted the output of
argocd version.
Describe the bug
At present, Argo CD (via gitops-engine) uses /dev/shm to store temporary files that are passed to kubectl code, such as manifests and cluster credentials. The temp directory logic can be found within pkg/utils/io/io.go of gitops-engine.
However, since /dev/shm is backed by RAM, it has limited size (but has the security advantage of avoiding writes of sensitive info to disk). For many use cases this is not an issue, but, when deploying to a large number of kubernetes clusters, or deploying large k8s manifests, /dev/shm can run out of disk space.
We can/should allow Argo CD users to customize the target output path for application-controller/gitops-engine generated manifests, to allow them to avoid this issue.
I propose we add a new configuration parameter to argocd-cmd-params-cm (and connected to Application Controller Deployment/StatefulSet YAML), that allow users to specify a custom application controller manifest destination path, which will allow application controller to writing to a different path with the container, such as '/tmp'.
See the linked issue in gitops-engine for further details.
This issue has a corresponding issue on gitops-engine: argoproj/gitops-engine#614
And PR: argoproj/gitops-engine#613
To Reproduce
Concurrently deploy to a large number of clusters, or deploy a large number of large manifests: if the concurrent KiB of generated resources exceed /dev/shm size, /dev/shm runs out of space, and an error occurs.
Expected behavior
Users should be able to specify custom Argo CD path/volume to store generated manifests, allowing them to avoid running out of disk space in /dev/shm.
Version
All