CNDB-12466: Wait for OutboundConnection to notice disconnect when simulating disconnect in ConnectionTest.testMessageDeliveryOnReconnect

[jkni]

What is the issue

ConnectionTest.testMessageDeliveryOnReconnect is flaky, where sometimes an outbound message is not delivered to the handler.

What does this PR fix and why was it fixed

If the OutboundConnection does not see that its channel is closed until flushing the message in Delivery#doRun, the message will be handled via onFailedSerialize and dropped. This PR changes the test to ensure the OutboundConnection is disconnected before enqueuing a message, triggering a reconnect. This fix has been tested 200 times in CI and did not fail.

Checklist before you submit for review

• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-1507 rejected by Butler

---

2 new test failure(s) in 1 builds
See build details here

---

Found 2 new test failures

Test	Explanation	Branch history	Upstream history
...id>>,ascii>>,wide=false,scenario=SSTABLE_QUERY]	regression	🔴	🔵🔵🔵🔵🔵🔵🔵
...yIteratorTest.testTotalAndReadBytesManySSTables	regression	🔴	🔵🔵🔵🔵🔵🔵🔵

---

Found 4 known test failures

[pkolaczk]

If the OutboundConnection does not see that its channel is closed until flushing the message in Delivery#doRun, the message will be handled via onFailedSerialize and dropped.

But is that the correct behavior?
If there exists an auto-reconnect mechanism, shouldn't it attempt to reconnect and resubmit the message in those cases as well instead of dropping the message when the timing is not right?

I just wonder if the flakiness of this test is not the result of the test problem, but the actual problem with the code under test and maybe we should improve the code.

[jkni]

It's a good question. My reading of the conversations around the messaging service rewrite is that this change was intentional, but let me open it up to a broader discussion.

[jkni]

After some further research and discussion, I do think this behavior is in line with the current design for eventually consistent messaging. The Messaging Service rewrite intentionally eliminated the concept of droppable verbs/retrying droppable verbs. Where retrying has been re-implemented for timeouts/disconnects of certain messages, it was implemented as a higher level https://issues.apache.org/jira/browse/CASSANDRA-18816. The prevailing view is that the messaging service at this level is in line with the eventually consistent design of most messages.