Skip to content

feat: Add dynamic filter pushdown support to prune files from scans#2

Merged
peasee merged 19 commits into
spiceaifrom
peasee-patch-3
Nov 10, 2025
Merged

feat: Add dynamic filter pushdown support to prune files from scans#2
peasee merged 19 commits into
spiceaifrom
peasee-patch-3

Conversation

@peasee
Copy link
Copy Markdown

@peasee peasee commented Nov 7, 2025

🗣 Description

  • Adds support for dynamic filter pushdown for pruning entire files. This does not yet update the filters in the ScanBuilder, because that would require re-building the scan. The scan builder probably needs some work to support updating the filters like that, so for now rely on pruning whole files instead.
    • If the files are grouped into sizes around 50mb, still provides a good opportunity for pruning. Needing data from only 1 file of a set of 10 removes 450mb of scanning.

@peasee peasee self-assigned this Nov 7, 2025
@peasee peasee added the enhancement New feature or request label Nov 7, 2025
@peasee peasee merged commit 05284cd into spiceai Nov 10, 2025
14 of 29 checks passed
lukekim pushed a commit that referenced this pull request May 18, 2026
## Summary

Fix for the second part of: vortex-data#7808 

```
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
    at ./nptl/pthread_kill.c:89
#3  0x000076a38cc4527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x000076a38cc288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x000076a38cc297b6 in __libc_message_impl (fmt=fmt@entry=0x76a38cdce8d7 "%s\n")
    at ../sysdeps/posix/libc_fatal.c:134
#6  0x000076a38cca8ff5 in malloc_printerr (
    str=str@entry=0x76a38cdd1bf0 "free(): double free detected in tcache 2")
    at ./malloc/malloc.c:5775
#7  0x000076a38ccab55f in _int_free (av=0x76a38ce03ac0 <main_arena>, p=<optimized out>, 
    have_lock=0) at ./malloc/malloc.c:4541
#8  0x000076a38ccaddce in __GI___libc_free (mem=0x5be5cd9632c0) at ./malloc/malloc.c:3398
#9  0x000076a38eb6807e in alloc::alloc::dealloc (ptr=0x5be5cd9632c0, layout=...)
    at /home/ubuntu/.rustup/toolchains/1.91.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:114
#10 alloc::alloc::{impl#1}::deallocate (self=0x5be5cd95f708, ptr=..., layout=...)
    at /home/ubuntu/.rustup/toolchains/1.91.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:271
#11 0x000076a38ead9a64 in alloc::boxed::{impl#8}::drop<dyn vortex_scan::Partition, alloc::alloc::Global> (self=0x5be5cd95f6f8)
    at /home/ubuntu/.rustup/toolchains/1.91.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1666
#12 0x000076a38ead349e in core::ptr::drop_in_place<alloc::boxed::Box<dyn vortex_scan::Partition, alloc::alloc::Global>> ()
    at /home/ubuntu/.rustup/toolchains/1.91.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:804
#13 0x000076a38e8764de in core::ptr::drop_in_place<vortex_ffi::scan::VxPartitionScan> ()
    at /home/ubuntu/.rustup/toolchains/1.91.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:804
#14 0x000076a38e876fb8 in core::ptr::drop_in_place<alloc::boxed::Box<vortex_ffi::scan::VxPartitionScan, alloc::alloc::Global>> ()
    at /home/ubuntu/.rustup/toolchains/1.91.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:804
#15 0x000076a38e87f2f5 in core::mem::drop<alloc::boxed::Box<vortex_ffi::scan::VxPartitionScan, alloc::alloc::Global>> (_x=0x5be5cd95f6f0)
    at /home/ubuntu/.rustup/toolchains/1.91.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:961
#16 0x000076a38e84efa7 in vortex_ffi::scan::vx_partition_free (ptr=0x5be5cd95f6f0)
    at vortex-ffi/src/macros.rs:295
#17 0x00005be5b0c81126 in operator() (__closure=0x7fff2208a8b0)
    at /home/ubuntu/vortex/vortex-ffi/test/scan.cpp:940
```

## Testing

Verifying existing behavior is maintained.

Signed-off-by: Dergousov Maksim <dergousovmaxim99@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants