Agent configuration path should be writeable and persisted

[barkbay]

Issue extracted from elastic/elastic-agent#5185 (comment)

In managed mode, Agent receives configuration from Fleet, and this configuration needs to reside in the state path so that it persists across restarts.

The recommendations from the Agent team are the following:

• Keep CONFIG_PATH pointing to the state path.
• If a user wants to mount additional static config into the Pod, they can do that and they can point to it using the -c argument to elastic-agent container ... cmd, and this can safely live outside the state path.

As a side note not having the configuration directory writeable (because of some dropped capabilities, read only fs ...) can also lead to the following error at startup:

agent container initialisation - effective capabilities
agent container initialisation - ambient capabilities
agent container initialisation - chown paths
Warning: cannot chown agent paths without CAP_CHOWN or CAP_DAC_OVERRIDE capabilities
Policy selected for enrollment:  eck-fleet-server
Error: could not create new vault: failed to create vault path: /etc/agent/vault, err: mkdir /etc/agent/vault: permission denied
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.19/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.19/fleet-troubleshooting.html

A workaround is then to apply the changes suggested above:

        containers:
        - name: agent
          env:
            - name: STATE_PATH
              value: /usr/share/elastic-agent/state
            - name: CONFIG_PATH
              value: /usr/share/elastic-agent/state
          args:
            - -e
            - -c
            - /etc/agent/elastic-agent.yml