-
Notifications
You must be signed in to change notification settings - Fork 40
Description
Description:
1- Does the problem persist?
Yes
2- Describe how to reproduce the issue:
- Deploy ecChronos with 2 datacenters with 3 Cassandra replicas each and 2 ecChronos, one for each datacenter
- Try to check the ecctool status; it shows the message "ecChronos is not running" for one of the data centers, the other one is fine.
- Try to get the schedules or run repairs, the result will be the message "ecChronos is not running" for one of the datacenters.
- The error is intermittent; it is not happening in every installation, only a few times.
Detailed description:
• What is happening?
ecChronos fails to start and remains in not running status.
• What did you expect to happen?
ecChronos should start successfully and report as running when checked with ecctool status.
• What have you tried?
Restarted the Pod to restart ecChronos, and after the restart, it worked again.
• What version of ecChronos are you using?
Latest version of agent/master branch
• Was the problem detected during an upgrade or downgrade procedure?
No
• Have you checked the ecChronos documents?
Yes.
• What do YOU think is the issue?
The logs show that mySession is null.
• Anything else that is worth being noted?
Full logs will be shared with Victor.
NullPointers in the logs that might be related to the error:
{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.MultiplexingNodeStateListener] Unexpected error while notifying node state listener com.ericsson.bss.cassandra.ecchronos.data.iptranslator.IpTranslator@34a6d9db of an onAdd event. (NullPointerException: Cannot invoke \"com.datastax.oss.driver.api.core.CqlSession.execute(com.datastax.oss.driver.api.core.cql.Statement)\" because \"this.mySession\" is null)"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Node(endPoint=/192.168.148.79:9042, hostId=dab1567d-c880-4390-aad6-8ea8e4d0dfa2, hashCode=4647686d) switched state to UP."
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Node added dab1567d-c880-4390-aad6-8ea8e4d0dfa2"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] myScheduleManager not ready when Node added dab1567d-c880-4390-aad6-8ea8e4d0dfa2"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Session during setupConfiguration call was null."}
{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.MultiplexingNodeStateListener] Unexpected error while notifying node state listener com.ericsson.bss.cassandra.ecchronos.data.iptranslator.IpTranslator@34a6d9db of an onUp event. (NullPointerException: Cannot invoke \"com.datastax.oss.driver.api.core.CqlSession.execute(com.datastax.oss.driver.api.core.cql.Statement)\" because \"this.mySession\" is null)"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Node added 4ca76dc1-c947-4570-b137-6ddac320d238"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] myScheduleManager not ready when Node added 4ca76dc1-c947-4570-b137-6ddac320d238"}
{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.MultiplexingNodeStateListener] Unexpected error while notifying node state listener com.ericsson.bss.cassandra.ecchronos.data.iptranslator.IpTranslator@34a6d9db of an onAdd event. (NullPointerException: Cannot invoke \"com.datastax.oss.driver.api.core.CqlSession.execute(com.datastax.oss.driver.api.core.cql.Statement)\" because \"this.mySession\" is null)"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Node(endPoint=/192.168.137.15:9042, hostId=4ca76dc1-c947-4570-b137-6ddac320d238, hashCode=91bfbc6) switched state to UP."}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Node added 4ca76dc1-c947-4570-b137-6ddac320d238"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] myScheduleManager not ready when Node added 4ca76dc1-c947-4570-b137-6ddac320d238"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Session during setupConfiguration call was null."}
{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.MultiplexingNodeStateListener] Unexpected error while notifying node state listener com.ericsson.bss.cassandra.ecchronos.data.iptranslator.IpTranslator@34a6d9db of an onUp event. (NullPointerException: Cannot invoke \"com.datastax.oss.driver.api.core.CqlSession.execute(com.datastax.oss.driver.api.core.cql.Statement)\" because \"this.mySession\" is null)"}
{"message":"[s0-io-0] [com.datastax.oss.driver.internal.core.metadata.SchemaAgreementChecker] [s0] Unknown peer d532fea4-c69b-4c01-b3d7-cbeb02cda896, excluding from schema agreement check"}
{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.token.NetworkTopologyReplicationStrategy] [s0] Error while computing token map for replication settings {class=org.apache.cassandra.locator.NetworkTopologyStrategy, datacenter1=3, datacenter2=1}: could not achieve replication factor 1 for datacenter datacenter2 (found only 0 replicas)."}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Node added d532fea4-c69b-4c01-b3d7-cbeb02cda896"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] myScheduleManager not ready when Node added d532fea4-c69b-4c01-b3d7-cbeb02cda896"}
{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.MultiplexingNodeStateListener] Unexpected error while notifying node state listener com.ericsson.bss.cassandra.ecchronos.data.iptranslator.IpTranslator@34a6d9db of an onAdd event. (NullPointerException: Cannot invoke \"com.datastax.oss.driver.api.core.CqlSession.execute(com.datastax.oss.driver.api.core.cql.Statement)\" because \"this.mySession\" is null)"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Node added 79176fa6-aef3-4d68-ba78-12e6c91df45c","metadata":{"container_name":"ecchronos","namespace":"epsdaua","pod_name":"wcdbcd-datacenter1-ecchronos-agent-7f8fbff99-mmh8t"},"service_id":"wcdbcd","severity":"info","timestamp":"2025-12-18T17:58:32.629Z","version":"1.0.0"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] myScheduleManager not ready when Node added 79176fa6-aef3-4d68-ba78-12e6c91df45c"}
{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.MultiplexingNodeStateListener] Unexpected error while notifying node state listener com.ericsson.bss.cassandra.ecchronos.data.iptranslator.IpTranslator@34a6d9db of an onAdd event. (NullPointerException: Cannot invoke \"com.datastax.oss.driver.api.core.CqlSession.execute(com.datastax.oss.driver.api.core.cql.Statement)\" because \"this.mySession\" is null)"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] Node added d9aaef89-dc35-45e5-835a-822ad4f258b3"}
{"message":"[s0-admin-0] [com.ericsson.bss.cassandra.ecchronos.core.impl.repair.DefaultRepairConfigurationProvider] myScheduleManager not ready when Node added d9aaef89-dc35-45e5-835a-822ad4f258b3"}
{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.MultiplexingNodeStateListener] Unexpected error while notifying node state listener com.ericsson.bss.cassandra.ecchronos.data.iptranslator.IpTranslator@34a6d9db of an onAdd event. (NullPointerException: Cannot invoke \"com.datastax.oss.driver.api.core.CqlSession.execute(com.datastax.oss.driver.api.core.cql.Statement)\" because \"this.mySession\" is null)"}