Infinite loop when running indexer-security-init.sh

[JcabreraC]

Wazuh version	Install type	Platform
4.3	Wazuh-Indexer	Linux

Description

When trying to execute the indexer-security-init.sh script and getting an error (probably due to configuration), if the configuration is modified and the wazuh-indexer service is restarted again, it remains in an infinite loop executing indexer-security-init.sh all the time, leaving the terminal unusable.

Steps to reproduce

• Edit the /etc/wazuh-indexer/opensearch.yml file with an incorrect configuration (e.g. not setting the network.host correctly)
• Execute the script indexer-security-init.sh and receive the following error:

  Security Admin v7
  Will connect to 0.0.0.0:9300 ... done
  Connected as CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US
  OpenSearch Version: 1.2.4
  OpenSearch Security Version: 1.2.4.0
  Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
  Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
  * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
  * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
  * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
  Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
  * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
  * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
  * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
  Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
  * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
  * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
  * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
  Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
  * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
  * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
  * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
  

• Edit the /etc/wazuh-indexer/opensearch.yml file with an correct configuration (e.g. setting the network.host correctly)
• Restard the service: systemctl restart wazuh-indexer

Result

The following message is displayed in an infinite loop, making it impossible to use the terminal:

   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{sv4jb2n9R86jOA7_GRKlzw}{0.0.0.0}{0.0.0.0:9300}]. This is not an error, will keep on trying ...
  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{sv4jb2n9R86jOA7_GRKlzw}{0.0.0.0}{0.0.0.0:9300}]] (org.opensearch.client.transport.NoNodeAvailableException/org.opensearch.client.transport.NoNodeAvailableException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)

[okynos]

After some tests over wazuh-indexer package in CentOS 7 and Ubuntu 20 systems. It seems impossible to reproduce and such problem is related to Security admin and indexer deep code. We can't fix this problem at this development stage.
I will close the issue until we can perform fixes in the indexer code.

[zbalkan]

This occurs on RHEL9 too. Just follow the assistant according to the docs, and create an Indexer cluster with at least 2 nodes. It is easy to reproduce.

But when you use just one node, the cluster start succeeds. The issue seems like somewhere in Indexer cluster discovery.

[c-bordon]

At the moment I was trying to reproduce the problem without success in Centos 7 and in Redhat 9, both in single node and in cluster, I will continue testing

[zbalkan]

Hi. In my case, it was my problem. I did skip firewall configuration. And that blocked cluster discovery. But I do not know for the other use cases.