Backport ccm main #1296

Lorak-mmk · 2025-03-25T14:43:36Z

@muzarski @wprzytula This is a draft of the CCM backport into main branch.

I mostly took the current state from branch-hackathon. One important change I made is to remove separate ccm-integration test target. Instead ccm tests are now a module in integration.
This avoids the problem with sharing utils. It should also be quicker to compile - no need to link 2 separate binaries.

Apart from that I did not really modify CCM integration. Now the question is: what do we do with it.
Are we satisfied with the API? Probably not.
If not, what should the API be like?

This is fully internal to the crate, so we can change it freely, so there is no need to spend too much time on it - we can always improve later.
Still, we should retain some reasonable level of quality, so I'd like to discuss this a bit.

I did not yet pick up @muzarski 's changes that integrated auth and TLS workflows into CCM. I can do that after we agree to the rest of this PR.

On the above matter @muzarski : How should I adapt CCM given that we now support multiple TLS backends?
I see we have in mod.rs DB_TLS_CERT_PATH, DB_TLS_KEY_PATH and CA_TLS_CERT_PATH (which I removed for now because it was guarded by old feature name).
I also see that on branch-hackathon you wrote a TLS test. We should probably make a test per backend, right?
What about those vars? Do they make sense for all the tests? In that case we just need to change feature guard on CA_TLS_CERT_PATH to be activated when any backend is active - or to even make it always active because why not.

Pre-review checklist

I have split my patch into logically separate commits.
All commit messages clearly explain what they change and why.
I added relevant tests for new features and bug fixes.
All commits compile, pass static checks and pass test.
PR description sums up the changes and reasons why they should be introduced.
I have provided docstrings for the public items that I want to introduce.
I have adjusted the documentation in ./docs/source/.
I added appropriate Fixes: annotations to PR description.

Cleanup unused dependancies

github-actions · 2025-03-25T14:45:50Z

cargo semver-checks found no API-breaking changes in this PR.
Checked commit: a02254c

It was released over 6 months ago so it is ok to use it. It allows us to use LazyCell.

ccm module will contain tests that require ccm. It's lib submodule will contain the ccm integration. Why do it this way - which is different than what we did during a hackathon? - Old way required ugly hacks to share test utils between integration test targets, and those hacks did not work well with rust-analyzer. - One target means better compilation time CCM tests will be guardded by a cfg, so we will still be able to run the subset that we want: - All tests: run integration tests with the required cfg - Only CCM tests: as above, but filter by ccm folder - Only non-ccm test: run without the cfg

Lorak-mmk · 2025-03-25T15:48:20Z

Possible improvements / API changes after a brief glance at the code:

Should NodeStartOptions be a struct? In other words, does it make sense to enable e.g. no_wait and wait_other_notice at the same time? We need to know the exact semantics of the wait-related flags to know that (cc @fruch because I don't think this is documented anywhere in this cursed software).
Same for NodeStopOptions
We hold nodes in Arc<RwLock<>>, and methods that give user nodes return that. Maybe we could return refs / mut refs and get rid of Arc<RwLock<>>? I'm not sure.
In the future when we have custom test runner, we could make new struct (ClusterPreferences + NodePreferences) or extend *Options structs. Why? Tests may not care about some parameters, which could help test runner to provide less clusters.

muzarski · 2025-03-25T16:37:25Z

On the above matter @muzarski : How should I adapt CCM given that we now support multiple TLS backends? I see we have in mod.rs DB_TLS_CERT_PATH, DB_TLS_KEY_PATH and CA_TLS_CERT_PATH (which I removed for now because it was guarded by old feature name). I also see that on branch-hackathon you wrote a TLS test. We should probably make a test per backend, right? What about those vars? Do they make sense for all the tests? In that case we just need to change feature guard on CA_TLS_CERT_PATH to be activated when any backend is active - or to even make it always active because why not.

Notice that I implemented all of this when we had old certificates in the repository. We had to update them, because of the errors thrown by rustls.

Why old certs worked for `openssl` but did not for `rustls`?

It's because rustls supports hostname verification by default, while openssl does not. The CN (common name) in db certificate was not matching the hostname, thus rustls was throwing an error.

What changed in the certificates, compared to the previous version?

I generated the db certificates assigned to static IP (172.44.0.2 - one we currently use in CI for TLS single-node cluster). In other words, the extensions to certificate request in openssl config looked like:

[v3_req]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
subjectAltName = IP:172.44.0.2
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

Thanks to that, rustls is able to verify the hostname using node's IP. It checks whether the node that we try to connect to has the same IP as the one defined in certificate (under subjectAltName).

Current state

Currently, our SSL "tests" are limited to simply running the tls-openssl and tls-rustls examples in SSL CI workflow.
On hackathon-branch, however, I removed the SSL workflow and migrated the example test to ccm. It obviously did it for openssl only, as rustls was not supported back then.

Implementing the corresponding ccm test for rustls backed and removing SSL CI workflow.

Well, this is a bit tricky. While, it was not an issue with openssl, for the reasons stated above (no hostname verification), it won't work for rustls. This is because we use dynamic IPs in ccm tests.

The temporary solution I see: migrate the tests from SSL workflow to ccm only for openssl and limit rustls "tests" to just running the example against the global cluster (as it is currently done in CI).

If we ever decide that we want to have ccm tests for rustls backend, I list the possible solutions to the dynamic IP problem:

Generate certificates on the fly, during the test. We would firstly receive some IP from the IpAllocator (or some other mechanism in the future) and then we could generate the self-signed cert for this IP. Then we could ccm updateconf and provide the path to generated certificate. There are some crates we could use, e.g. https://docs.rs/rcgen/latest/rcgen/.
Disable hostname verification during the tests. I think this can be configured via the trait: rustls::client::danger::ServerCertVerifier. I'm not entirely sure, though. This needs some research - I think this is placed in danger module for a reason. OTOH, we already use it in scylla::cloud::config and implement it for our NoCertificateVerification struct.

Lorak-mmk · 2025-03-26T06:51:14Z

Moving such workflows to CCM has 2 advantages:

We get rid of custom images
We get rid of a GHA workflow

If we move only part of it, we get neither. So I'll cherry pick commits that move auth, and skip TLS for now. We can do that in the future.

Btw is it possible to use domain names instead of ip addresses with scylla? In other words, can we have domain names instead of ip addresses in system.peers in driver-relevant columns?
If it was possible, we could use certs with hostnames instead of ips.

fruch · 2025-03-26T07:26:40Z

Moving such workflows to CCM has 2 advantages:

We get rid of custom images

We get rid of a GHA workflow

If we move only part of it, we get neither. So I'll cherry pick commits that move auth, and skip TLS for now. We can do that in the future.

Btw is it possible to use domain names instead of ip addresses with scylla? In other words, can we have domain names instead of ip addresses in system.peers in driver-relevant columns? If it was possible, we could use certs with hostnames instead of ips.

scylla can use hostnames, but then you need a dns server to map them.

I think generating certs as needed is the best approach, and also give the flexibility to try more variants as needed.
that's what we are doing in dtest, and in SCT.

Lorak-mmk · 2025-03-26T07:48:41Z

scylla can use hostnames, but then you need a dns server to map them.

I think generating certs as needed is the best approach, and also give the flexibility to try more variants as needed. that's what we are doing in dtest, and in SCT.

Is there functionality in CCM to generate certs? Or do we have to do it other way?

If Scylla can use hostnames, then we should test it too.

@fruch one other question for you. Could you describe (or point to documentation if such exists) what exactly wait-related flags do in CCM, and how do they interact if I specify more than one?

fruch · 2025-03-26T08:06:55Z

scylla can use hostnames, but then you need a dns server to map them.
I think generating certs as needed is the best approach, and also give the flexibility to try more variants as needed. that's what we are doing in dtest, and in SCT.

Is there functionality in CCM to generate certs? Or do we have to do it other way?

If Scylla can use hostnames, then we should test it too.

@fruch one other question for you. Could you describe (or point to documentation if such exists) what exactly wait-related flags do in CCM, and how do they interact if I specify more than one?

you are more then welcome to document it.

Lorak-mmk · 2025-03-26T08:17:06Z

you are more then welcome to document it.

I'd be happy to make a PR that improves descriptions, but I would have to first understand those options myself.
I don't know ccm's codebase at all, and it is not really friendly to new contributors, that's why I asked you to explain those options.

Lorak-mmk · 2025-03-26T09:23:00Z

I backported the commits that move auth to CCM. I also removed TLS support from CCM for now.

Lorak-mmk · 2025-03-26T13:45:41Z

Marking as ready. The way I see it the only improvement I can make here is better CCM API - which needs input from others, which is basically a review.

muzarski

We hold nodes in Arc<RwLock<>>, and methods that give user nodes return that. Maybe we could return refs / mut refs and get rid of Arc<RwLock<>>? I'm not sure.

As of now, there is no use case for Arc<RwLock<>> - I think we can return refs/mutrefs for now. We could always revert this in the future. It also simplifies the API - I believe NodeList is no longer necessary then. Instead, we can expose nodes_iter_[mut]() and get_node_[mut]_by_id methods on Cluster.

Currently, append_node() and add_node() methods return Arc<RwLock<>>. They could return node id instead.

scylla/tests/integration/ccm/example.rs

scylla/tests/integration/ccm/lib/ip_allocator.rs

Co-authored-by: Mikołaj Uzarski <[email protected]> Co-authored-by: Dmitry Kropachev <[email protected]>

Lorak-mmk · 2025-03-28T18:57:31Z

Addressed @muzarski 's comments
Removed all the Arc<Mutex> stuff, now we just operate on Nodes.
I decided to retain NodeList because I welcome any kind of separation and structure in this code. I made its method simpler using iterator methods.
Method that adds node return mut reference to this node. I think it is more useful than id.

Lorak-mmk · 2025-03-28T19:06:22Z

I have one more idea: we can split off another file from cluster.rs, I would call it ccm_cmd.rs.
It would be a simple wrapper over CCM, providing builder-style commands.
The purpose of this module would be to provide convenient way to call CCM, and encode all its commands and flags into Rust types.
cluster.rs would be responsible for providing user-facing API, handling config dirs etc. Its code would hopefully become cleaner.

Lorak-mmk · 2025-03-28T20:40:22Z

I did this for 2 commands as an experiment, in additional commit. I like the new version, so unless anyone has different opinion I'll convert the rest of the command to this.

@dkropachev I see that both ccm create and ccm populate accept ipprefix argument. Why? What are their respective semantics?

For now it will run on each PR. If at some point it becomes too slow we can switch it to running manually and before release.

Co-authored-by: Mikołaj Uzarski <[email protected]>

Auth tests are now run as a part of CCM test suite.

muzarski · 2025-03-31T14:17:09Z

I did this for 2 commands as an experiment, in additional commit. I like the new version, so unless anyone has different opinion I'll convert the rest of the command to this.

I love the idea. The code in cluster.rs looks much cleaner.

dkropachev · 2025-03-31T19:24:02Z

I did this for 2 commands as an experiment, in additional commit. I like the new version, so unless anyone has different opinion I'll convert the rest of the command to this.

@dkropachev I see that both ccm create and ccm populate accept ipprefix argument. Why? What are their respective semantics?

ccm create calls ccm populate, if you provide number of nodes.
Flow of ccm create + ccm add + ccm start is broken, so you better make ccm create to call ccm populate to create a cluster in the single blow.

wprzytula · 2025-04-01T16:20:14Z

Flow of ccm create + ccm add + ccm start is broken

Could you elaborate? What bad happens in such scenario?

wprzytula

❓ cluster.rs contains a lot of functions which are not used and whose goal is not always clear to me. Are we going to merge them as-is, only later wondering about their intended use and possibly adding documentation?

wprzytula · 2025-03-27T14:09:33Z

scylla/src/cloud/config.rs

+            .inspect_err(|_err| {
                warn!(


🔧 Note that as #1298 has been merged, there is one more spot that should be updated in this commit.

wprzytula · 2025-03-27T14:11:21Z

.github/workflows/rust.yml

@@ -13,7 +13,7 @@ env:
  CARGO_TERM_COLOR: always
  RUSTFLAGS: -Dwarnings
  RUST_BACKTRACE: full
-  rust_min: 1.70.0 # <- Update this when bumping up MSRV
+  rust_min: 1.80.0 # <- Update this when bumping up MSRV


🔧 Not to forget about it, I suggest making a follow-up commit that replaces lazy_static uses with LazyLock. Let's remove deps providing functionality that is already present in the standard library.

wprzytula · 2025-03-27T14:12:42Z

scylla/Cargo.toml

@@ -84,12 +84,15 @@ num-bigint-04 = { package = "num-bigint", version = "0.4" }
 bigdecimal-04 = { package = "bigdecimal", version = "0.4" }
 scylla-proxy = { version = "0.0.3", path = "../scylla-proxy" }
 ntest = "0.9.3"
-criterion = "0.4"                                                      # Note: v0.5 needs at least rust 1.70.0
-tokio = { version = "1.34", features = ["test-util"] }
+criterion = "0.4"                                                       # Note: v0.5 needs at least rust 1.70.0


📌 maybe let's bump it, as now we can?

wprzytula · 2025-03-27T14:14:29Z

scylla/Cargo.toml

+anyhow = "1"
+tempfile = "3"


🔧 As you mentioned elsewhere @Lorak-mmk, omitting minor number can result in build problems in some scenarios. Let's write minors explicitly.

wprzytula · 2025-04-01T16:26:37Z

scylla/tests/integration/ccm/example.rs

+#[tokio::test]
+#[cfg_attr(not(ccm_tests), ignore)]
+async fn test_cluster_lifecycle1() {
+    setup_tracing();
+    async fn test(cluster: Arc<Mutex<Cluster>>) {
+        let cluster = cluster.lock().await;
+        let session = cluster.make_session_builder().await.build().await.unwrap();
+
+        let rows = session
+            .query_unpaged("select data_center from system.local", &[])
+            .await
+            .expect("failed to execute query")
+            .into_rows_result()
+            .expect("failed to get rows")
+            .rows::<(String,)>()
+            .expect("failed to deserialize rows")
+            .map(|res| res.map(|row| row.0))
+            .collect::<Result<Vec<_>, _>>()
+            .unwrap();
+        debug!("{:?}", rows);
+    }
+    run_ccm_test(cluster_1_node, test).await;
+}
+
+#[tokio::test]
+#[cfg_attr(not(ccm_tests), ignore)]
+async fn test_cluster_lifecycle2() {
+    setup_tracing();
+    async fn test(cluster: Arc<Mutex<Cluster>>) {
+        let cluster = cluster.lock().await;
+        let session = cluster.make_session_builder().await.build().await.unwrap();
+
+        let rows = session
+            .query_unpaged("select data_center from system.local", &[])
+            .await
+            .expect("failed to execute query")
+            .into_rows_result()
+            .expect("failed to get rows")
+            .rows::<(String,)>()
+            .expect("failed to deserialize rows")
+            .map(|res| res.map(|row| row.0))
+            .collect::<Result<Vec<_>, _>>()
+            .unwrap();
+        debug!("{:?}", rows);
+    }
+    run_ccm_test(cluster_1_node, test).await;
+}


❓ Is there any difference between these two tests? I can't find any.

wprzytula · 2025-04-01T16:47:29Z

scylla/tests/integration/ccm/lib/ccm_cmd.rs

+        let nodes_string = nodes
+            .iter()
+            .map(|node| node.to_string())
+            .collect::<Vec<String>>()
+            .join(":");


⛏️ Let's use Itertools::join. Unnecessary collect() pains my eyes even though I know these are just tests...

wprzytula · 2025-04-01T16:49:47Z

scylla/tests/integration/ccm/lib/ccm_cmd.rs

+pub(crate) struct ClusterCreate<'ccm> {
+    ccm: &'ccm mut Ccm,
+    name: String,
+    version: String,
+    ip_prefix: String,
+    db_type: DBType,
+}
+
+impl Ccm {
+    pub(crate) fn cluster_create(
+        &mut self,
+        name: String,
+        version: String,
+        ip_prefix: NetPrefix,
+        db_type: DBType,
+    ) -> ClusterCreate<'_> {
+        ClusterCreate {
+            ccm: self,
+            name,
+            version,
+            ip_prefix: ip_prefix.to_string(),
+            db_type,
+        }
+    }
+}
+
+impl ClusterCreate<'_> {
+    pub(crate) async fn run(self) -> Result<ExitStatus, anyhow::Error> {
+        let mut args: Vec<&str> = vec![
+            "create",
+            self.name.as_str(),
+            "--version",
+            self.version.as_str(),
+            "--ipprefix",
+            self.ip_prefix.as_str(),
+            "--config-dir",
+            self.ccm.config_dir.as_str(),
+        ];
+        if self.db_type == DBType::Scylla {
+            args.push("--scylla");
+        }
+        self.ccm
+            .cmd
+            .run_command("ccm", &args, RunOptions::new())
+            .await
+    }
+}


💯 🤩 What a great, clean design! Congrats!

Right now all parameters are accepted in the constructor of the command. I'll switch it to builder pattern soon, and of course move other commands too.

wprzytula · 2025-04-01T16:52:45Z

scylla/tests/integration/ccm/lib/ip_allocator.rs

+    pub(super) fn to_str(self) -> String {
        match self.0 {
            IpAddr::V4(v4) => {
                let octets = v4.octets();


🤔 I doubted to_* functions should accept self by value, even in case of Copy types.
However, a quick look at i64::to_be() convinced me. TIL.

I would like to change this name anyway, but not because of to_ part. The str part is the problem imo, as the function returns String, not &str.

wprzytula · 2025-04-01T17:25:28Z

scylla/tests/integration/ccm/lib/ip_allocator.rs

+
+impl std::fmt::Display for NetPrefix {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", self.to_str())


⛏️ Should have a bit lower overhead (at least compile-time):

Suggested change

write!(f, "{}", self.to_str())

f.write_str(self.to_str())

wprzytula · 2025-04-01T17:27:30Z

scylla/tests/integration/ccm/lib/ip_allocator.rs

+                let ip_port: Vec<&str> = ip_hex.split(':').collect();
+                if let Some(ip_hex) = ip_port.first() {


Suggested change

let ip_port: Vec<&str> = ip_hex.split(':').collect();

if let Some(ip_hex) = ip_port.first() {

let mut ip_port = ip_hex.split(':');

if let Some(ip_hex) = ip_port.next() {

scylla: cleanup Cargo.toml

8ba3fc3

Cleanup unused dependancies

Lorak-mmk force-pushed the backport-ccm-main branch from 2f94f52 to 394dcec Compare March 25, 2025 15:00

Lorak-mmk added 2 commits March 25, 2025 16:11

Bump MSRV to 1.80

7a34172

It was released over 6 months ago so it is ok to use it. It allows us to use LazyCell.

Lorak-mmk force-pushed the backport-ccm-main branch 2 times, most recently from ad0c3aa to 6e14276 Compare March 25, 2025 15:20

Lorak-mmk force-pushed the backport-ccm-main branch from 6e14276 to aeabc5e Compare March 26, 2025 08:56

Lorak-mmk marked this pull request as ready for review March 26, 2025 13:45

Lorak-mmk requested review from wprzytula and muzarski and removed request for wprzytula March 26, 2025 13:45

muzarski assigned Lorak-mmk Mar 26, 2025

muzarski suggested changes Mar 26, 2025

View reviewed changes

scylla/tests/integration/ccm/example.rs Outdated Show resolved Hide resolved

scylla/tests/integration/ccm/example.rs Outdated Show resolved Hide resolved

scylla/tests/integration/ccm/lib/ip_allocator.rs Outdated Show resolved Hide resolved

Add CCM integration and example test

7a38b56

Co-authored-by: Mikołaj Uzarski <[email protected]> Co-authored-by: Dmitry Kropachev <[email protected]>

Lorak-mmk force-pushed the backport-ccm-main branch from aeabc5e to d546b53 Compare March 28, 2025 19:01

Lorak-mmk force-pushed the backport-ccm-main branch from d546b53 to f05e4dc Compare March 28, 2025 20:38

Lorak-mmk force-pushed the backport-ccm-main branch 2 times, most recently from 55fced0 to b605f97 Compare March 28, 2025 20:50

Extract cluster initialization into ccm_cmd module

41e26cf

Lorak-mmk force-pushed the backport-ccm-main branch from b605f97 to a02254c Compare March 28, 2025 20:51

Lorak-mmk and others added 4 commits March 28, 2025 21:51

Makefile: Add target to run CCM tests

dbbecb7

Add CCM CI pipeline

d6f3fb2

For now it will run on each PR. If at some point it becomes too slow we can switch it to running manually and before release.

IT: Move authenticate tests to CCM

2b0c18c

Co-authored-by: Mikołaj Uzarski <[email protected]>

ci: remove auth workflow and related dockerfile

a02254c

Auth tests are now run as a part of CCM test suite.

wprzytula requested changes Apr 2, 2025

View reviewed changes

Lorak-mmk mentioned this pull request Apr 7, 2025

Add flatten attribute to derive SerializeRow #1144

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport ccm main #1296

Backport ccm main #1296

Lorak-mmk commented Mar 25, 2025 •

edited

Loading

github-actions bot commented Mar 25, 2025 •

edited

Loading

Lorak-mmk commented Mar 25, 2025

muzarski commented Mar 25, 2025

Lorak-mmk commented Mar 26, 2025

fruch commented Mar 26, 2025

Lorak-mmk commented Mar 26, 2025

fruch commented Mar 26, 2025

Lorak-mmk commented Mar 26, 2025

Lorak-mmk commented Mar 26, 2025

Lorak-mmk commented Mar 26, 2025

muzarski left a comment

Lorak-mmk commented Mar 28, 2025 •

edited by wprzytula

Loading

Lorak-mmk commented Mar 28, 2025

Lorak-mmk commented Mar 28, 2025

muzarski commented Mar 31, 2025

dkropachev commented Mar 31, 2025

wprzytula commented Apr 1, 2025

wprzytula left a comment

wprzytula Mar 27, 2025

wprzytula Mar 27, 2025

wprzytula Mar 27, 2025

wprzytula Mar 27, 2025

wprzytula Apr 1, 2025

wprzytula Apr 1, 2025

wprzytula Apr 1, 2025

Lorak-mmk Apr 2, 2025

wprzytula Apr 1, 2025

Lorak-mmk Apr 2, 2025

wprzytula Apr 1, 2025

wprzytula Apr 1, 2025

		let ip_port: Vec<&str> = ip_hex.split(':').collect();
		if let Some(ip_hex) = ip_port.first() {

		anyhow = "1"
		tempfile = "3"

Backport ccm main #1296

Are you sure you want to change the base?

Backport ccm main #1296

Conversation

Lorak-mmk commented Mar 25, 2025 • edited Loading

Pre-review checklist

github-actions bot commented Mar 25, 2025 • edited Loading

Lorak-mmk commented Mar 25, 2025

muzarski commented Mar 25, 2025

Why old certs worked for openssl but did not for rustls?

What changed in the certificates, compared to the previous version?

Current state

Implementing the corresponding ccm test for rustls backed and removing SSL CI workflow.

Lorak-mmk commented Mar 26, 2025

fruch commented Mar 26, 2025

Lorak-mmk commented Mar 26, 2025

fruch commented Mar 26, 2025

Lorak-mmk commented Mar 26, 2025

Lorak-mmk commented Mar 26, 2025

Lorak-mmk commented Mar 26, 2025

muzarski left a comment

Choose a reason for hiding this comment

Lorak-mmk commented Mar 28, 2025 • edited by wprzytula Loading

Lorak-mmk commented Mar 28, 2025

Lorak-mmk commented Mar 28, 2025

muzarski commented Mar 31, 2025

dkropachev commented Mar 31, 2025

wprzytula commented Apr 1, 2025

wprzytula left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lorak-mmk commented Mar 25, 2025 •

edited

Loading

github-actions bot commented Mar 25, 2025 •

edited

Loading

Why old certs worked for `openssl` but did not for `rustls`?

Lorak-mmk commented Mar 28, 2025 •

edited by wprzytula

Loading