Monitoring Console: Keep restarting

[Sinamcp]

Please select the type of request

Bug

Tell us more

Describe the request

• Ansible for monitoring console failed at dmc_asset_build_full - POST task

Splunk setup on K8S

• openshift 4.10
  1 clustermanager
  3 index node
  3 serach head nodes
  1 license manager
  1 console manager
  I deployed monitoring console with the following manifest(like doc)

apiVersion: enterprise.splunk.com/v4
kind: MonitoringConsole
metadata:
  name: mc-test
  namespace: default
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  clusterManagerRef:
    name: cm-test
  etcVolumeStorageConfig:
    storageCapacity: 2Gi
  varVolumeStorageConfig:
    storageCapacity: 5Gi

after applying the manifest, monitoring console pod start ansible, but it fails with the below error

TASK [splunk_monitor : dmc_asset_build_full - POST] ****************************
fatal: [localhost]: FAILED! => {
    "cache_control": "no-store, no-cache, must-revalidate, max-age=0",
    "changed": false,
    "connection": "Close",
    "content_length": "360",
    "content_type": "text/xml; charset=UTF-8",
    "date": "Sat, 18 Feb 2023 12:26:24 GMT",
    "elapsed": 0,
    "expires": "Thu, 26 Oct 1978 00:00:00 GMT",
    "redirected": false,
    "server": "Splunkd",
    "status": 503,
    "url": "https://127.0.0.1:8089/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC%20Asset%20-%20Build%20Full/dispatch",
    "vary": "Cookie, Authorization",
    "x_content_type_options": "nosniff",
    "x_frame_options": "SAMEORIGIN"
}

MSG:

Status code was 503 and not [200, 201, 409]: HTTP Error 503: Service Unavailable

[kashok-splunk]

Hi @Sinamcp
We don't need to provide cluster-manager info in the MC yaml, we can remove this information. Regarding the 503 error, are all the services up? as 503 error indicates service is not up. Additionally, can you try running this GET call from MC pod? curl -X GET -k -u admin:<PASSWORD> https://127.0.0.1:8089/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC%20Asset%20-%20Build%20Full?output_mode=json, this query just builds the MC asset table. Let me know what response you get here

[akondur]

Hi @Sinamcp , please let us know if there are any updates here.

[Sinamcp]

Hi @akondur

(okd4venv) (okd4venv) [centos@okd4-ts-2-utility ~]$ oc rsh splunk-mc-test-monitoring-console-0
sh-4.4$ curl -X GET -k -u admin:D7YSwvrMWXdxpzb7MOB7nIM4 https://127.0.0.1:8089/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC%20Asset%20-%20Build%20Full?output_mode=json
curl: (7) Failed to connect to 127.0.0.1 port 8089: Connection refused

[Sinamcp]

I don't know what should i do, please help me to fix