ecctool commands not working sometimes

[cezarpaulo16]

Description:

1- Does the problem persist?
Yes

2- Describe how to reproduce the issue:

1. Deploy ecChronos with 2 datacenters with 3 Cassandra replicas each and 2 ecChronos, one for each datacenter.
2. After deployment, check the ecctool commands "ecctool status", "ecctool schedules". The commands get stuck and do not provide any responses,
3. The issue is intermittent; it is only happening a few times.

Detailed description:

• What is happening?
ecctool commands get stuck and do not provide any answer.

• What did you expect to happen?
ecctool commands to work and return the message.

• What have you tried?
Restart the Pod. After the restart of ecChronos, it worked.

• What version of ecChronos are you using?
Latest version of agent/master branch

• Was the problem detected during an upgrade or downgrade procedure?
No

• Have you checked the ecChronos documents?
Yes.

• What do YOU think is the issue?

• Anything else that is worth being noted?

Full logs will be shared with Victor.

[VictorCavichioli]

The only reason I see of not having response from ecctool is that http request is taking too much time to return a answer, this could happen because of network interference or because of the TLS handshake, anyway I don't see failed HTTP requests in the logs, so I need more information to work on this

[cezarpaulo16]

Well, it worked in DC2, and after restarting ecchornos from DC1, it worked as well, so I guess that something is happening during startup, as it was not happening before, maybe it might be related to the pull requests "Add check during startup" and "Add check during startup to ensure there are enough local nodes.", maybe something happened in startup that make rest not to get up I don't know.

Also, in the logs, there is this line, which may be related to that, but not sure:

{"message":"[s0-admin-0] [com.datastax.oss.driver.internal.core.metadata.token.NetworkTopologyReplicationStrategy] [s0] Error while computing token map for replication settings {class=org.apache.cassandra.locator.NetworkTopologyStrategy, datacenter1=3, datacenter2=1}: could not achieve replication factor 1 for datacenter datacenter2 (found only 0 replicas)."}

What more information do you need? Please let me know, and I can provide it.

[paulchandler]

The http interface ( and hence ecctool working ) is only opened after a connection to cassandra has been confirmed. So the new code added for "Add check during startup" and "Add check during startup to ensure there are enough local nodes." will delay the availability of the port, if the cassandra cluster is not ready.

[cezarpaulo16]

I see, interesting, so maybe something could have happened in the retry mechanism? Because I left this environment from one day to another, and the error was still happening, after that, I restarted the Pod, and it worked. Is the retry inside the connection.cql that is being used?

[paulchandler]

That shouldn't be the case, once the cluster is available to provide local_quorum on the ecchronos keyspace it should continue and the port becomes available.

You should see the following in the log file, after that ecctool should work:

Tomcat started on port 8080 (http) with context path '/'

If it happens again, capture the ecchronos log and send it over.

[paulchandler]

Although looking again, if the port is not open you should get the following error "ecChronos is not running" so it might not be that.

[cezarpaulo16]

I see, this error only happened one time so far.