[Bug] GF_PATHS_PLUGINS content removed by operator

[dadez]

Describe the bug
I have a container, based on upstream grafana 11.2.0 where I install the plugin yesoreyeram-infinity-datasource
When I run this container locally docker run, the plugin is available in grafana.

docker run log output: we can see that the plugin is registered and that 55 plugins are loaded

logger=plugin.store t=2024-08-30T10:53:54.024854111Z level=info msg="Loading plugins..."
logger=plugins.registration t=2024-08-30T10:53:54.048655368Z level=error msg="Could not register plugin" pluginId=xychart error="plugin xychart is already registered"
logger=plugins.initialization t=2024-08-30T10:53:54.048695303Z level=error msg="Could not initialize plugin" pluginId=xychart error="plugin xychart is already registered"
logger=local.finder t=2024-08-30T10:53:54.048732386Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
logger=plugins.registration t=2024-08-30T10:53:54.135233713Z level=info msg="Plugin registered" pluginId=yesoreyeram-infinity-datasource
logger=plugin.store t=2024-08-30T10:53:54.135264186Z level=info msg="Plugins loaded" count=55 duration=110.410619ms

When I run this container on OpenShift into a Grafana instance managed by the grafana-operator, the plugin is missing
cotainer start log: no information about the plugin registration, only 54 plugins are loaded.

logger=plugin.store t=2024-08-30T11:06:37.020928461Z level=info msg="Loading plugins..."
logger=plugins.registration t=2024-08-30T11:06:37.054597216Z level=error msg="Could not register plugin" pluginId=xychart error="plugin xychart is already registered"
logger=plugins.initialization t=2024-08-30T11:06:37.054621428Z level=error msg="Could not initialize plugin" pluginId=xychart error="plugin xychart is already registered"
logger=local.finder t=2024-08-30T11:06:37.054684443Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
logger=plugin.store t=2024-08-30T11:06:37.054696715Z level=info msg="Plugins loaded" count=54 duration=33.768951ms

The folder /var/lib/grafana/plugins is empty in the container.

Version
v5.8.0

To Reproduce
Steps to reproduce the behavior:

1. build a container based on grafana/grafana:11.2.0 and install the plugin with grafana cli plugins install yesoreyeram-infinity-datasource 2.9.5
2. Setup / upgrade a Grafana instance using this image
3. Search for the pluing in grafana
4. List the content of folder $GF_PATHS_PLUGINS (/var/lib/grafana/plugins)

Expected behavior
The content of the $GF_PATHS_PLUGINS should not be empty and the plugin should be available for use in grafana.
It should be possible to manage plugins without any internet access.

Suspect component/Location where the bug might be occurring
unknown

Screenshots
If applicable, add screenshots to help explain your problem.

Runtime (please complete the following information):

• OS: [linux manjaro (for docker run only)]
• Grafana Operator Version [v5.8.0]
• Environment: [Openshift 4.14.25]
• Deployment type: [olm]

Additional context

• /var/lib/grafana is mounted as pvc
• our grafana instance has no internet connection, therefore the installation of plugins described in https://grafana.github.io/grafana-operator/docs/datasources/#plugins is not an option
• It works using the same container ,on the same platform installing a bitnami based helm chart
• Setting public_key_retrieval_disabled to true and allow_loading_unsigned_plugins to yesoreyeram-infinity-datasource did not help