Description
Please select the type of request
Bug
Tell us more
Describe the request
When an existing Splunk indexer pod starts, the container starts as expected and the Splunk prints the "My GUID" message as expected.
However, the ansible playbook in Splunk always triggers a 2nd restart, in some cases I've seen 3 restarts. I find this consistently in my environment including the test environment.
Note I've also confirmed the same behaviour occurs on Splunk search heads, so I assume it's a generic issue with the ansible re-setting a setting and then deciding a restart is "required". I've done a difference of the configuration during these restarts and I usually find that encrypted pass4SymmKeys/other values were re-encrypted to a new value but have the same decrypted value (i.e. it was a pointless change)
Expected behavior
When a pod restarts I expect Splunk to startup, and not to restart when no changes are required. The ansible playbook from Splunk triggers the restart for no obvious reason.
Splunk setup on K8S
Splunk indexer cluster, 4 nodes, multi-site (2 nodes per site) in development.
Reproduction/Testing steps
Build indexer cluster, once all nodes are online restart 1 pod.
Observe the pod, and check how many times "My GUID" appears in the splunkd log.
K8s environment
K8s 1.28, same issue was on K8s 1.24.
Proposed changes(optional)
Update the splunk ansible playbooks embedded inside the container to reduce restarts to only when a "real" change has occurred and not on every time the pod comes online (when the pod has previously existed and has persistent storage)
Additional context(optional)
I have at least one support case which is likely related to this issue as the cluster/bucket level issues start only after the indexer is restarted shortly after coming online.