[helm] Add State Persistence Parameter for Agent Presets in Helm Chart

[pkoutsovasilis]

Describe the enhancement:
Introduce a statePersistence parameter for each agent preset in the Helm chart, with default configurations based on the preset mode. Managing agent state in conjunction with the agent deployment mode can be complex and non-intuitive for users. Therefore, the Helm chart should simplify this process by providing sensible defaults while allowing flexibility for custom configurations. The statePersistence parameter accepts the following values:

• HostPath: Suitable for agents running integrations that require data persistence. This will be the default for the daemonset mode of the agent preset if the parameter is not set.
• EmptyDir: Suitable for agents running integrations that do not require data persistence. This will be the default for statefulset and deployment modes of the agent preset if the parameter is not set.
• None: Disables automated state persistence, allowing users to configure custom volumes.

Users can override the defaults by setting statePersistence to None and using the existing extraVolumes and extraVolumeMounts parameters to define custom volume configurations.

Describe a specific use case for the enhancement or feature:
This feature provides a seamless way to handle agent state persistence. For example, DaemonSet agents can use HostPath for persistent data, while StatefulSet and Deployment agents use EmptyDir for ephemeral storage. By automating state management based on deployment mode, the Helm chart makes it easier for users to deploy agents without worrying about intricate volume configurations. Advanced users can disable the default behavior and configure custom volumes as needed.

What is the definition of done?

• The Helm chart introduces a statePersistence enum parameter with values: HostPath, EmptyDir, and None.
• Safe defaults: HostPath for DaemonSet, EmptyDir for StatefulSet and Deployment.
• Users can specify None to disable defaults and define custom volumes with extraVolumes and extraVolumeMounts.
• Documentation is updated to explain the statePersistence parameter, defaults, and customization options.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[swiatekm]

Do we also want a mode where we use persistent volumes in StatefulSet mode, using the cluster's default storage class? From experience with Otel at least, people often want this.

[pkoutsovasilis]

Do we also want a mode where we use persistent volumes in StatefulSet mode, using the cluster's default storage class? From experience with Otel at least, people often want this.

@swiatekm that’s actually a great point; in my initial thinking, I considered PVC as part of the “advanced” configuration where the user would select None for statePersistence and manually set up their persistent volumes. However, having another type that leverages the cluster’s default storage class as a built-in option could indeed simplify things for users who frequently work with persistent storage in StatefulSet mode. It sounds like a sensible addition worth exploring!