[BUG] Opensearch security not initializing correctly

[DrissiReda]

What is the bug?

Cluster isn't able to go online. I'm trying the workaround to deploy a single node with security on

How can one reproduce the bug?

You can use my own cluster.yaml

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch-cluster
spec:
  security:
    config: 
      adminSecret:
        name: opensearch-admin-certs
      adminCredentialsSecret:
        name: opensearch-security-secrets
      securityConfigSecret:
        name: opensearch-security-secrets
    tls:
        http:
          generate: false
          secret:
            name: opensearch-tls-certs
        transport:
          generate: false
          secret:
            name: opensearch-tls-certs
          nodesDn: ["CN=opensearch-cluster-managers.doc"]
          adminDn: ["CN = opensearch-cluster-admin"]
  
  general:
    httpPort: 9200
    serviceName: opensearch
    version: 2.18.0
    monitoring:
     enable: false
    drainDataNodes: true
    setVMMaxMapCount: false
    additionalVolumes:
    - name: log-empty-dir
      path: /usr/share/opensearch/logs
      emptyDir: {}
    - name: tmp-empty-dir
      path: /tmp
      emptyDir: {}
    - name: secconfig
      emptyDir: {}
      path: /usr/share/opensearch/plugins/opensearch-security/securityconfig
    - name: admincerts
      secret:
        secretName: opensearch-admin-certs
      path: /tmp/admintls
    - name: keystore
      path: /usr/share/opensearch/config/opensearch.keystore
      subPath: opensearch.keystore
      secret:
        secretName: opensearch-keystore-secret
    additionalConfig:
      logger.level: "DEBUG"
      plugins.security.ssl.transport.enforce_hostname_verification: "false"
      plugins.security.ssl.http.enabled: "true"
      plugins.security.allow_unsafe_democertificates: "true"
      plugins.security.allow_default_init_securityindex: "true"
      plugins.security.enable_snapshot_restore_privilege: "true"
      plugins.security.check_snapshot_restore_write_privileges: "true"
      plugins.security.restapi.roles_enabled: "[all_access, security_rest_api_access]"

    podSecurityContext:
      runAsUser: 1000
      runAsGroup: 1000
      runAsNonRoot: true
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      privileged: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
  dashboards:
    opensearchCredentialsSecret:
      name: opensearch-security-secrets
    additionalVolumes:
    - name: dashboards-data
      path: /usr/share/opensearch-dashboards/data
      emptyDir: {}
    podSecurityContext:
      runAsUser: 1000
      runAsGroup: 1000
      runAsNonRoot: true
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      privileged: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
    tls:
      enable: false
      generate: false
    version: 2.18.0
    enable: true
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: managers
      replicas: 1
      additionalConfig:
        discovery.seed_hosts: opensearch-cluster-managers-0
        cluster.initial_master_nodes: opensearch-cluster-managers-0
      diskSize: "3Gi"
      jvm: -Xmx512M -Xms512M
      resources:
         requests:
            memory: "1024Mi"
            cpu: "1000m"
         limits:
            memory: "1024Mi"
            cpu: "1000m"
      roles:
        - "cluster_manager"
      persistence:
        pvc:
          storageClass: lvm-provisioner
          accessModes:
            - ReadWriteOnce

What is the expected behavior?

Security should be initialized automatically

What is your host/environment?

Kubernetes

Do you have any additional context?

The securityconfig update pod shows this output:

Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-88 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-88 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cn) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-89 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-89 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cn) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

When I try to run securityadmin.sh with --accept-red-cluster manually inside the cluster-managers I get timeouts on applying every configuration.

Running a curl -sk https://localhost:9200 inside the pod returns a simple: OpenSearch Security not initialized

[swoehrl-mw]

Hi @DrissiReda. The operator explicitly does not support single-node clusters. As such, any failure to get it working is not a bug in the operator. Single-node clusters will also very likely never be suported by the operator.