Add test for RequiredHostAbsent error

[Lorak-mmk]

Based on #1473 (which was merged) and #1479
The above PR changes the semantics in a way that allows us to write the test for RequiredHostAbsent error.
The test adds proxy rules that drop connection when query to system.local is received on node 1 (which is coordinator for DDL) and also prevent opening new connections to this node.
Now schema awaiting will perform check twice:

• First check will return broken connection error for coordinator, and drop all connections
• In second (and each subsequent) check checking schema agreement will require RequiredHostAbsent

Test needs to wait until schema agreement timeout - the semantics change means that it won't return error earlier.
If we need to make this test faster, then we need to make check_schema_agreement_with_required_node public and call it directly.

Fixes: #1362

Pre-review checklist

• I have split my patch into logically separate commits.
• All commit messages clearly explain what they change and why.
• I added relevant tests for new features and bug fixes.
• All commits compile, pass static checks and pass test.
• PR description sums up the changes and reasons why they should be introduced.
• I have provided docstrings for the public items that I want to introduce.
• I have adjusted the documentation in ./docs/source/.
• I added appropriate Fixes: annotations to PR description.

[github-actions]

cargo semver-checks found no API-breaking changes in this PR.
Checked commit: 8a8625c

[Copilot]

Pull Request Overview

This PR adds comprehensive test coverage for the RequiredHostAbsent error condition and refactors the schema agreement logic to better handle transient failures. The changes are based on semantic improvements from PR #1473 that allow detecting when a required coordinator node becomes unavailable during schema agreement checks.

Key Changes

• Refactored await_schema_agreement_with_required_node to track the last attempt result and return more informative errors on timeout
• Added test_schema_await_with_transient_failure to verify schema agreement succeeds even when initial checks fail
• Added test_schema_await_required_host_absent to test the RequiredHostAbsent error condition by simulating connection failures to the coordinator node

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
scylla/tests/integration/session/schema_agreement.rs	Adds two new integration tests that verify schema agreement behavior under failure conditions using proxy rules to simulate network issues
scylla/src/client/session.rs	Refactors schema agreement implementation to eliminate the helper method and improve error reporting by tracking last attempt results

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Lorak-mmk]

Rebased on main

[Lorak-mmk]

Rebased on current version of base PR, addressed comments.

[Lorak-mmk]

Rebased on main

[Copilot]

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The CREATE KEYSPACE statement is split across multiple lines without using a line continuation backslash (\). This will include a newline and indentation whitespace in the SQL string, which is inconsistent with the rest of the codebase.

Consider reformatting to a single line like other tests in this file (e.g., line 43), or use a backslash for line continuation like in simple_strategy.rs:

let mut request = Statement::new(format!(
    "CREATE KEYSPACE {ks} WITH REPLICATION = {{'class' : 'NetworkTopologyStrategy', 'replication_factor' : 1}}"
));

Suggested change

	"CREATE KEYSPACE {ks} WITH
	REPLICATION = {{'class' : 'NetworkTopologyStrategy', 'replication_factor' : 1}}"
	"CREATE KEYSPACE {ks} WITH REPLICATION = {{'class' : 'NetworkTopologyStrategy', 'replication_factor' : 1}}"

[wprzytula]

Doesn't this close #1349?

[Lorak-mmk]

I don't think so. It doesn't test the things that were changed in that PR (fetching only from one connection).

[Lorak-mmk]

The test failed on Cassandra, but not in the main part of the test but in droopping the keyspace at the end. Weird.
The error is:

thread 'session::schema_agreement::test_schema_await_required_host_absent' (12643) panicked at scylla/tests/integration/session/schema_agreement.rs:312:18:
called `Result::unwrap()` on an `Err` value: SchemaAgreementError(Timeout(1s))
stack backtrace:

The code is

            // Cleanup
            running_proxy.running_nodes[1].change_request_rules(Some(vec![]));
            session.await_schema_agreement().await.unwrap();
            session
                .query_unpaged(format!("DROP KEYSPACE {ks}"), &[])
                .await
                .unwrap();

with unwrap being line 312.

What is weird is that the error mentions SchemaAgreementError alone, not as part of ExecutionError.
Maybe the error message is wrong and the test failed somewhere else? This doesn't make much sense because in the logs I can see that the DROP was issued.

[Lorak-mmk]

Rebased on main. Added a check that uuid in error is correct.
Let's see if the error is reproducible.

[Lorak-mmk]

Now the test passed. I'll rerun it a few times.

[Lorak-mmk]

Cassandra failed again, but on a different test :D

thread 'session::pager::test_pager_timeouts' (12341) panicked at scylla/tests/integration/session/pager.rs:332:22:
called `Result::unwrap()` on an `Err` value: NextPageError(RequestFailure(RequestTimeout(100ms)))