CQL Vector support #1165

smoczy123 · 2025-01-06T22:36:37Z

This PR adds serialization and deserialization of CQL Vector (as implemented in Cassandra) therefore achieving compatibility with Cassandra's Vector type. It's important to note that Cassandra implements Vector serialization and deserialization in a way that
contradicts the CQL protocol, using [unsigned vint] instead of [int] as the element size encoding for variable type length vectors.

Fixes #1014

Pre-review checklist

I have split my patch into logically separate commits.
All commit messages clearly explain what they change and why.
I added relevant tests for new features and bug fixes.
All commits compile, pass static checks and pass test.
PR description sums up the changes and reasons why they should be introduced.
I have provided docstrings for the public items that I want to introduce.
I have adjusted the documentation in ./docs/source/.
I added appropriate Fixes: annotations to PR description.

github-actions · 2025-01-06T22:41:42Z

cargo semver-checks found no API-breaking changes in this PR.
Checked commit: 322cb09

smoczy123 · 2025-01-10T03:19:05Z

I'm not sure this is the correct way to split this PR into commits (I'm pretty sure it isn't, as the commits won't compile), however I can't think of a proper way.

muzarski

I only reviewed the first commit (introduction of TypeParser)

Some general comments:

The logic of TypeParser is quite complex. I suggest adding some docstrings next to the type definitions and methods. For example, I have no idea what TypeParser::from_hex does. Docstrings will also help a lot in the future in case some other developer touches this piece of code.
It's worth adding some comments next to the non-intuitive parts of the code. Example:

        if name.is_empty() {
            if !self.is_eos() {
                return Err(CqlTypeParseError::AbstractTypeParseError());
            }
            return Ok(ColumnType::Blob);
        }

It's not obvious why we return Blob if name is empty. A link to the corresponding part of original source code would be helpful.

Please, add some unit tests. I saw that there is some small test of TypeParser in a later commit. I think we should add more tests and try to handle as many parsing cases as we can. In addition, I think that in this case, unit tests should be added in the same commit (they help during review - it's easier to reason about the complex code when there are some use case examples one can look at)
This implementation is based on some existing (probably Java) implementation, correct? If so, please, provide the link to the source in the commit. Ideally, the link should be placed in the comments in code as well.

scylla-cql/src/frame/frame_errors.rs

muzarski · 2025-01-13T15:52:37Z

scylla-cql/src/frame/response/type_parser.rs

+type UDTParameters<'result> = (
+    Cow<'result, str>,
+    Cow<'result, str>,
+    Vec<(Cow<'result, str>, ColumnType<'result>)>,
+);


nit: I'd prefer to have it as a struct instead of a type alias. Cow<'result, str> type appears twice and it's hard to reason about it without explicit field names.

nit: I'd prefer to have it as a struct instead of a newtype.

Actually, a struct is called a newtype. type is a type alias, which is not a new type, yet just a new name for an existing type.

muzarski · 2025-01-13T15:54:12Z

scylla-cql/src/frame/response/type_parser.rs

+pub(crate) struct TypeParser<'result> {
+    pos: usize,
+    str: Cow<'result, str>,
+}


nit: Probably need to rename it to CustomTypeParser (or AbstractTypeParser if we decide to stick to abstract naming convention). Same goes for the name of the module - type_parser.rs is not specific enough IMO.

scylla-cql/src/frame/response/type_parser.rs

muzarski · 2025-01-13T15:57:10Z

scylla-cql/src/frame/response/type_parser.rs

+        if !self.is_eos() && self.str.as_bytes()[self.pos] == b':' {
+            self.pos += 1;
+            let _ = usize::from_str_radix(&name, 16)
+                .map_err(|_| CqlTypeParseError::AbstractTypeParseError());
+            name = self.read_next_identifier();
+        }


What does this part do? Is it tested somewhere?

smoczy123 · 2025-01-13T16:52:16Z

Whole TypeParser logic was ripped straight out of ScyllaDB's vector implementation, however, as it still in development and probably won't be merged for a while, it will be hard to link directly. IIRC there is a lot of tests there for this functionality, so thay also can be borrowed.

muzarski · 2025-01-13T16:58:14Z

Whole TypeParser logic was ripped straight out of ScyllaDB's vector implementation, however, as it still in development and probably won't be merged for a while, it will be hard to link directly. IIRC there is a lot of tests there for this functionality, so thay also can be borrowed.

Ok, makes sense. And let's borrow the tests in such case :)

wprzytula

💯 What a great piece of code! Thank you for the contribution!

There are quite many comments, though.
I think that the new parser module needs much more unit tests.
Also, tests for particular errors upon serialization and deserialization of Vector are missing.

wprzytula · 2025-01-15T13:11:56Z

scylla-cql/src/frame/response/type_parser.rs

+type UDTParameters<'result> = (
+    Cow<'result, str>,
+    Cow<'result, str>,
+    Vec<(Cow<'result, str>, ColumnType<'result>)>,
+);


nit: I'd prefer to have it as a struct instead of a newtype.

Actually, a struct is called a newtype. type is a type alias, which is not a new type, yet just a new name for an existing type.

scylla-cql/src/frame/response/type_parser.rs

wprzytula · 2025-01-15T16:17:37Z

scylla-cql/src/types/serialize/writers.rs

 pub struct CellWriter<'buf> {
    buf: &'buf mut Vec<u8>,
+    cell_len: Option<usize>,
+    size_as_uvarint: bool,
 }


@Lorak-mmk As you know the serialization framework quite well, could you please aid in review of this commit?

scylla-cql/src/types/serialize/writers.rs

wprzytula · 2025-01-15T16:33:55Z

scylla-cql/src/types/serialize/writers.rs

+        if let Some(buffer) = self.variable_length_buffer {
+            let value_len = buffer.len();
+            let mut len = Vec::new();
+            types::unsigned_vint_encode(value_len as u64, &mut len);
+            self.buf.extend_from_slice(&len);
+            self.buf.extend_from_slice(&buffer);
+        } else {
+            let value_len: i32 = (self.buf.len() - self.starting_pos - 4)
+                .try_into()
+                .map_err(|_| CellOverflowError)?;
+            self.buf[self.starting_pos..self.starting_pos + 4]
+                .copy_from_slice(&value_len.to_be_bytes());
+        }


💭 The logic around Cell* got much convoluted with these alterations. Let's think how we can make it more digestible. @Lorak-mmk @muzarski

scylla/src/client/session_test.rs

scylla-cql/src/frame/frame_errors.rs

scylla-cql/src/frame/response/type_parser.rs

scylla-cql/src/frame/response/result.rs

docs/source/data-types/vector.md

scylla-cql/src/frame/response/result.rs

scylla-cql/src/types/deserialize/value_tests.rs

scylla-cql/src/frame/response/type_parser.rs

scylla-cql/src/frame/value.rs

scylla-cql/src/types/deserialize/value.rs

smoczy123 · 2025-03-22T19:03:44Z

As of right now all changes requested above should be implemented, the only thing left is the lifetime issue with the new_type_parser in deser_type_generic. @wprzytula @Lorak-mmk

Lorak-mmk · 2025-03-24T13:27:47Z

As of right now all changes requested above should be implemented, the only thing left is the lifetime issue with the new_type_parser in deser_type_generic. @wprzytula @Lorak-mmk

I see a lot of open review comments. Please go over them and:

If you are sure that you fixed the issue that the comment pointed out, mark it as resolved
If you did not, or you are not sure, then respond to the comment to keep the discussion going
If the comment does not point out any issue with your code (like CQL Vector support #1165 (comment) ) then you can ignore it.

Lorak-mmk

I left just 2 comments. I'll continue the review after the second one is addressed.

scylla-cql/src/frame/response/new_type_parser.rs

Lorak-mmk · 2025-03-24T13:22:39Z

scylla-cql/src/frame/response/new_type_parser.rs

Commit: frame: added a parser for Java-class type names

You introduced both new_type_parser.rs and type_parser.rs.

Both contain UDTParameters struct, and TypeParser struct with pub(crate) parse method.

There is no module-level comment on those files

No comments on the aforementioned structs and method.

type_parser seems unused (Does make ci pass on this commit?).

Commit message says nothing about those modules and structs.

How I as a reviewer am supposed to understand the purpose of those modules and structures? Please improve it by addressing some of the points above.

Both of those do the same thing, parse the Java class type name, however the new_type_parser uses the parse.rs module that was already present in the project, however due to lifetime issues I was not able to make it work correctly. So right now there are two versions of the parser, once we either find a way to fix the lifetime issues or decide to go back to type_parser.rs (that would be quite sad, we would duplicate a lot of parsing code) the other version will be deleted. cc @wprzytula

smoczy123 · 2025-04-02T13:24:26Z

I have closed the resolved conversations, all of the issues in them should be fixed. I have decided to not touch the bigger issues with the type_parser.rs right now, as if we decide to throw it away it would be a waste of time right now. cc @Lorak-mmk

This is needed to deserialize vector metadata as it is implemented as a Custom type with VectorType as its class

Due to the fact that Cassandra implements variable type length vectors in a way that contradicts the CQL protocol, special care must be given when deserializing them as sizes of their elements are encoded as unsigned vint instead of an int

Similarly to the previous commit, special care must be given when serializing variable type length vectors, as sizes of their elements must be written as an unsigned varint

swasik · 2025-04-03T08:11:49Z

I have decided to not touch the bigger issues with the type_parser.rs right now, as if we decide to throw it away it would be a waste of time right now.

If we decide not to fix it right now, could you create the issue with the whole connected covered so that we don't forget about it in the future?

github-actions bot added the semver-checks-breaking cargo-semver-checks reports that this PR introduces breaking API changes label Jan 6, 2025

smoczy123 force-pushed the vector-type branch from 78f489c to cfdf4e5 Compare January 10, 2025 03:13

smoczy123 marked this pull request as ready for review January 10, 2025 03:19

smoczy123 force-pushed the vector-type branch 2 times, most recently from 6aee097 to 440d63a Compare January 13, 2025 12:38

wprzytula requested review from muzarski, Lorak-mmk and wprzytula January 13, 2025 13:29

wprzytula added this to the 0.16.0 milestone Jan 13, 2025

muzarski reviewed Jan 13, 2025

View reviewed changes

wprzytula requested changes Jan 15, 2025

View reviewed changes

wprzytula assigned smoczy123 Jan 15, 2025

piodul requested changes Jan 17, 2025

View reviewed changes

smoczy123 force-pushed the vector-type branch 5 times, most recently from 23d6dad to 8e128e1 Compare January 22, 2025 22:07

Lorak-mmk mentioned this pull request Jan 23, 2025

Remove CqlType in favor of ColumnType #1166

Merged

8 tasks

wprzytula requested review from muzarski, piodul and wprzytula January 26, 2025 15:17

Lorak-mmk modified the milestones: 0.16.0, 1.0.0 Feb 5, 2025

muzarski modified the milestones: 1.0.0, 1.1.0 Feb 6, 2025

smoczy123 force-pushed the vector-type branch from 8e128e1 to 60afb67 Compare March 17, 2025 17:07

github-actions bot removed the semver-checks-breaking cargo-semver-checks reports that this PR introduces breaking API changes label Mar 17, 2025

smoczy123 force-pushed the vector-type branch 4 times, most recently from b572171 to 62f008a Compare March 22, 2025 18:59

Lorak-mmk requested changes Mar 24, 2025

View reviewed changes

smoczy123 force-pushed the vector-type branch 2 times, most recently from 85ef4e7 to 8165b53 Compare April 2, 2025 13:19

wprzytula requested a review from Lorak-mmk April 2, 2025 13:27

smoczy123 added 7 commits April 2, 2025 15:33

scylla-cql/utils: moved parse.rs module to scylla-cql crate

4d27cea

frame: added a parser for Java-class type names

bf001d3

This is needed to deserialize vector metadata as it is implemented as a Custom type with VectorType as its class

frame: added a Vector CqlValue and ColumnType

f9fa37a

types/serialize: added vector serialization

91b308a

Similarly to the previous commit, special care must be given when serializing variable type length vectors, as sizes of their elements must be written as an unsigned varint

test: added tests for vector type

0392bd1

docs: added docs for vector type

322cb09

smoczy123 force-pushed the vector-type branch from 8165b53 to 322cb09 Compare April 2, 2025 13:35

Lorak-mmk modified the milestones: 1.1.0, 1.2.0 Apr 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CQL Vector support #1165

CQL Vector support #1165

smoczy123 commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

smoczy123 commented Jan 10, 2025

muzarski left a comment

muzarski Jan 13, 2025 •

edited

Loading

wprzytula Jan 15, 2025

muzarski Jan 13, 2025

muzarski Jan 13, 2025 •

edited

Loading

smoczy123 commented Jan 13, 2025

muzarski commented Jan 13, 2025

wprzytula left a comment

wprzytula Jan 15, 2025

wprzytula Jan 15, 2025

wprzytula Jan 15, 2025 •

edited

Loading

smoczy123 commented Mar 22, 2025

Lorak-mmk commented Mar 24, 2025

Lorak-mmk left a comment

Lorak-mmk Mar 24, 2025 •

edited

Loading

smoczy123 Mar 24, 2025

smoczy123 commented Apr 2, 2025

swasik commented Apr 3, 2025

CQL Vector support #1165

Are you sure you want to change the base?

CQL Vector support #1165

Conversation

smoczy123 commented Jan 6, 2025 • edited Loading

Pre-review checklist

github-actions bot commented Jan 6, 2025 • edited Loading

smoczy123 commented Jan 10, 2025

muzarski left a comment

Choose a reason for hiding this comment

muzarski Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

wprzytula Jan 15, 2025

Choose a reason for hiding this comment

muzarski Jan 13, 2025

Choose a reason for hiding this comment

muzarski Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

smoczy123 commented Jan 13, 2025

muzarski commented Jan 13, 2025

wprzytula left a comment

Choose a reason for hiding this comment

wprzytula Jan 15, 2025

Choose a reason for hiding this comment

wprzytula Jan 15, 2025

Choose a reason for hiding this comment

wprzytula Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

smoczy123 commented Mar 22, 2025

Lorak-mmk commented Mar 24, 2025

Lorak-mmk left a comment

Choose a reason for hiding this comment

Lorak-mmk Mar 24, 2025 • edited Loading

Choose a reason for hiding this comment

smoczy123 Mar 24, 2025

Choose a reason for hiding this comment

smoczy123 commented Apr 2, 2025

swasik commented Apr 3, 2025

smoczy123 commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

muzarski Jan 13, 2025 •

edited

Loading

muzarski Jan 13, 2025 •

edited

Loading

wprzytula Jan 15, 2025 •

edited

Loading

Lorak-mmk Mar 24, 2025 •

edited

Loading