Cassandra pods started with `cass-management-api` image stay in a zombie state when Cassandra rejects invalid configuration

[kos-team]

Describe the bug
When applied invalid configuration, Cassandra pod started with cass-management-api image reports misconfiguration, but keeps running in a zombie state. When checking the running processes in the Cassandra pod, we see that the actual Cassandra server process has already exited, but the mgmt-api process is still running, serving the /health and /ready requests.

To Reproduce
We encounter this bug in the cass-management-api image when deploying the CassandraDataCenter using the cass-operator. This bug can be reproduced by first deploying the cass-operator and

1. Apply this CR:

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: dc1
spec:
  clusterName: dev
  serverType: cassandra
  serverVersion: "4.1.2"
  managementApiAuth:
    insecure: {}
  size: 4
  storageConfig:
      cassandraDataVolumeClaimSpec:
        storageClassName: standard
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  podTemplateSpec:
    spec:
      containers:
        - name: "cassandra"
  resources:
    requests:
      memory: 2Gi
      cpu: 1000m
  racks:
    - name: rack1
    - name: rack2
    - name: rack3
    - name: rack4
  config:
    jvm-server-options:
      initial_heap_size: "1G"
      max_heap_size: "1G"
    cassandra-yaml:
      num_tokens: 16
      authenticator: AllowAllAuthenticator
      authorizer: CassandraAuthorizer
      role_manager: CassandraRoleManager

Expected behavior
We expect the Cassandra container to crash after the Cassandra rejects the invalid configuration. Based on the internal error Cassandra reports, we can't configure authenticator: AllowAllAuthenticator if authorizer: CassandraAuthorizer. However, this is not mentioned in the config documentation.

The same behavior can be reproduced by removing authenticator: AllowAllAuthenticator in the CR since Cassandra defaults authenticator to be AllowAllAuthenticator.

Current behavior
The Cassandra containers are left in an zombie state where the container can be ssh-accessed but the Cassandra has already crashed. The liveness probe still returns true, the readiness probe is returning false.

Root Cause
In Cassandra's source code, having AllowAllAuthenticator and CassandraAuthenticator together causing this branch to be taken, returning the error.

However, cass-management-api does not check if the Cassandra process has existed or not, and continue to run even the Cassandra process has crashed.

Desktop (please complete the following information):

OS: Ubuntu 22.04
Browser [e.g. chrome, safari]
Reproduced on latest release v0.8.3

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: MAPI-62