App Framework: splunk-operator ignores installing applications on second search-head cluster

[yaroslav-nakonechnikov]

Please select the type of request

Bug

Tell us more

Describe the request
we have installed 2 search-head clusters:

• one for internal needs of splunk team for tricky queries and findings with ESS.
• second one for generic access for everyone with simple setup of apps.

Today i found, that splunk-operator ignores checking applications, which should be installed on second search-head cluster.

[yn@ip-10-216-35-53 bin]$ kubectl logs pod/splunk-operator-controller-manager-ddccd8b97-nhxt5 -n splunk-operator -c manager | grep "Apps List retrieved from remote storage" | grep -v "splunk-apps/local_manager_apps/" | grep -v "splunk-apps/manager_apps" | grep -v "splunk-apps/e_local_deployer_apps" | grep -v "splunk-apps/e_deployer_apps"
[yn@ip-10-216-35-53 bin]$

there should be some logs with:
splunk-apps/c_deployer_apps and splunk-apps/c_local_deployer_apps strings, but nothing is found.

Expected behavior
All apps are being installed correct.

Splunk setup on K8S
EKS

Reproduction/Testing steps

[yn@ip-10-216-35-53 bin]$ kubectl get cm  -n splunk-operator | grep search
splunk-c-dev-search-head-defaults          1      6d
splunk-e-dev-search-head-defaults          1      6d

[yn@ip-10-216-35-53 bin]$ kubectl get statefulset  -n splunk-operator | grep search
splunk-c-dev-search-head        3/3     6d
splunk-e-dev-search-head        3/3     6d

[akondur]

Hi @iaroslav-nakonechnikov , could you please provide

1. YAML files for both the search head cluster CRs(Whatever part is shareable, at the least the appFramework config)?
2. Share the outputs of kubectl get shc/<name_of_shc> -o yaml for both your SHC CRs? I am looking to debug the status fields of the CR in the context of appFramework.

[yaroslav-nakonechnikov]

@akondur we decided to proceed with single search head cluster for the time being. So for some time this topic is not critical for us anymore.

[akondur]

Hey @iaroslav-nakonechnikov , can we go ahead and close the issue for now?

[yaroslav-nakonechnikov]

@akondur if you fixed an issue - you can close.
if you want to close just to make it closed - no, it is not solved.

lowering priority is not something that auto heals. It just will appear in future.

[akondur]

@iaroslav-nakonechnikov As requested here, could you please provide the requested information? We would require specific data from your setup to debug further.

[yaroslav-nakonechnikov]

@akondur root cause may be discovered in #1178.
atm i can't provide more info, as we moved away from using 2 search head clusters for some time.