Skip to content

fix: Remove duplicate projections #434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: Remove duplicate projections #434

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 23 additions & 10 deletions src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,15 @@ mod trigger;
mod version;
mod wal;

use std::{collections::HashMap, io, marker::PhantomData, mem, ops::Bound, pin::pin, sync::Arc};
use std::{
collections::{HashMap, HashSet},
io,
marker::PhantomData,
mem,
ops::Bound,
pin::pin,
sync::Arc,
};

pub use arrow;
use async_lock::RwLock;
Expand Down Expand Up @@ -881,18 +889,23 @@ where
/// fields in projection Record by field indices
pub fn projection(self, projection: &[&str]) -> Self {
let schema = self.mem_storage.record_schema.arrow_schema();
let mut projection = projection
.iter()
.map(|name| {
schema

let mut seen: HashSet<&str> = HashSet::new();
let mut indices: Vec<usize> = Vec::with_capacity(projection.len());

// Remove duplicate projections
for &name in projection {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a little difficult to keep the chaining functions as .filter().map() requires mutable + immutable borrows, .filter_map returns Options

if seen.insert(name) {
let idx = schema
.index_of(name)
.unwrap_or_else(|_| panic!("unexpected field {name}"))
})
.collect::<Vec<usize>>();
.unwrap_or_else(|_| panic!("unexpected field {name}"));
indices.push(idx);
}
}

let primary_key_index = self.mem_storage.record_schema.primary_key_index();
let mut fixed_projection = vec![0, 1, primary_key_index];
fixed_projection.append(&mut projection);
fixed_projection.dedup();
fixed_projection.append(&mut indices);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this won't work. For example, when we query &["primary_key"], the fixed_projection is [0, 1, 2, 2].


let mask = ProjectionMask::roots(
&ArrowSchemaConverter::new().convert(schema).unwrap(),
Expand Down
Loading