App Framework: Applications not being reinstalled if node with ebs was deleted

[yaroslav-nakonechnikov]

Please select the type of request

Enhancement

Tell us more

Describe the request
I see, that after terminating pod (by whatever reason) doesn't invoke app reinstalls till new version is published.

Expected behavior
When any pod starts - it checks for apps, and if needed - install them.

Splunk setup on K8S
k8s on eks

Reproduction/Testing steps
install some application thru appframework
delete pod with storage
wait till it starts

Proposed changes(optional)
would be cool if it will start to work as creating list on defaults.yml, where it is possible to define what should be installed.
maybe some flag: reinstall_apps_on_creation: <bool>, which will force installing apps on true, and leave as is on false.

[yaroslav-nakonechnikov]

also,

kubectl patch cm/splunk-splunk-operator-manual-app-update --type merge -p '{"data":{"ClusterManager":"status: on\nrefCount: 1"}}'

doesn't do the magic, apps are not being installed.

[yaroslav-nakonechnikov]

is there any command which can be triggered to force install apps?

[sgontla]

@iaroslav-nakonechnikov , in the case of Search Head Cluster OR Indexer Cluster, on a pod termination, when a new pod is started, it will get the app packages thorough the bundles. So, the App Framework needs to have persistent storage for both the deployer and the Cluster Manager, considering the reset scenarios.

So, same is the case in the case of standalone(assuming, this is the case you are exploring?). App install status is tracked per pod in the case of standalone, so, once a particular app is marked as installed, the next trigger will be only on the app package update. So, persistent volume is a pre-requisite in this case as well.

[yaroslav-nakonechnikov]

@sgontla i understand what you are saying. It will work, till there are no issues with persistent storage.
But this persistent storage might broke. For example, because of human error.

So, there need a way to force reinstalling all applications.

[marcispauls]

I would agree with @iaroslav-nakonechnikov infra breaks time to time and there is no way to force re-sync of apps.

[yaroslav-nakonechnikov]

how is it going? we need a way to reinstall apps

[DrMeosch]

Hi guys,

I have the following use-case. I run Splunk as a Distributed Clustered Deployment (Single Site) on AKS using Azure Blob as SmartStore. When the node is gone and the pods are recreated, then an app with the SmartStore configuration is not installed. That way the indexes are also not there at all and the indexers throwing "..Received event for unconfigured/disabled/deleted index.." errors. That happens despite having Persistent Storage for CM.

Probably, being able to setup SmartStore config directly in the helm chart as well as having a way to reinstall the apps would avoid this issue.

I hope to hear from you soon.
Have a nice day!