🌪 0.49.0

Changes

✨ Features

• feat: show statpopgen benchmarks (#4359) @danking
• feat: statistical and population genetics benchmark queries and dataset (#4175) @danking

🐛 Bug Fixes

• fix: Always use setSafe when writing values from spark (#4367) @robert3005
• fix: python integration docs (#4363) @danking
• chore: revert to 20.04, enforce glibc version == 2.31 (#4354) @danking
• Revert "Re-open DuckDB context between iterations" (#4355) @lwwmanning

🧰 Maintenance

15 changes

• chore: exclude vortex-error from coverage report (#4366) @lwwmanning
• fix(deps): update all patch updates to v1.0.169 (patch) (#4365) @renovate[bot]
• Re-open DuckDB context between iterations (take 2) (#4362) @lwwmanning
• chore(deps): update rust crate pyo3 to 0.25.0 (#4364) @renovate[bot]
• chore(deps): update rust crate noodles-bgzf to 0.43.0 (#4360) @renovate[bot]
• chore(deps): update rust crate noodles-vcf to 0.81.0 (#4361) @renovate[bot]
• chore(deps): lock file maintenance (#4349) @renovate[bot]
• chore: Inline arrow-pyarrow crate to avoid upstream pyo3 version pin (#4342) @robert3005
• chore: update renovate.json (#4357) @lwwmanning
• chore(deps): update rust crate regex to v1.11.2 (#4348) @renovate[bot]
• chore: shard codspeed benchmarks (#4353) @lwwmanning
• chore(deps): update ubuntu docker tag to v24 (#4345) @renovate[bot]
• chore: add PyO3 environment variables (#4350) @connortsui20
• chore(deps): update datafusion and friends to v49.0.2 (patch) (#4351) @renovate[bot]
• Re-open DuckDB context between iterations (#4346) @gatesn

🌪 0.48.0

Changes

✨ Features

• feat: teach Vortex how to implement a Ray Datasource (#4339) @danking
• feat: teach VortexDataset get_fragments (#4340) @danking
• feat: vortex-python/check.sh (#4338) @danking
• feat: teach VortexDataset to filter & document vortex.expr.* (#4336) @danking

🐛 Bug Fixes

• fix: Remove ScalarValue::is_instance_of (#4341) @robert3005

🧰 Maintenance

4 changes

• fix: Java builds with GLIBC 2.31+ compatibility (#4331) @a10y
• chore(deps): update rust crate url to v2.5.7 (#4343) @renovate[bot]
• chore(deps): update actions/upload-pages-artifact action to v4 (#4329) @renovate[bot]
• fix: Buffer Clone bound (#4337) @a10y

🌪 0.47.0

Changes

✨ Features

• feat[vortex-array]: add stepped pipeline execution (#4312) @joseph-isaacs
• feat[dtype]: ptype visitor (#4307) @joseph-isaacs

🚀 Performance

• feat[vortex-array]: add stepped pipeline execution (#4312) @joseph-isaacs

🐛 Bug Fixes

• fix(java): Spark writer properly handles string and array data (#4335) @a10y
• fix: Spawn datafusion write tasks (#4316) @AdamGS
• fix: vortex-cxx examples macos build (#4308) @0ax1
• fix: Remove conversion from usize to scalar (#4304) @robert3005
• fix: downscale FSST offsets (#4301) @a10y
• feat[vortex-array]: support converting to timestamp arrow arrays with tz (#4285) @asubiotto

🧰 Maintenance

28 changes

• chore: pyright (#4289) @danking
• Clean up match statements and misc (#4332) @connortsui20
• chore: vortex-ffi testing & build improvements (#4218) @lwwmanning
• chore[bench]: have --skip-generate option and only create iterators for tpch when needed (#4328) @joseph-isaacs
• Add local .claude folder to .gitignore (#4325) @connortsui20
• chore: configure codecov comments to be more useful (#4326) @lwwmanning
• chore(deps): update all patch updates (patch) (#4323) @renovate[bot]
• chore: return lvalue ref for cxx's scanbuilder functions (#4306) @XinyuZeng
• chore(deps): update rust crate zip to v4.5.0 (#4320) @renovate[bot]
• Add the missing copyright profile file (#4318) @gatesn
• chore: Don't prune uv caches in CI (#4314) @robert3005
• chore: vortex-scalar cleanup (#4313) @connortsui20
• chore(deps): update actions/setup-java action to v5 (#4305) @renovate[bot]
• chore: Make downscale_integer_array a method on PrimitiveArray (#4303) @robert3005
• chore(deps): update rust crate url to v2.5.6 (#4309) @renovate[bot]
• chore(deps): update spiraldb/actions action to v0.15.0 (#4311) @renovate[bot]
• chore(deps): update rust crate zip to v4.4.0 (#4310) @renovate[bot]
• Chore: Add list of list tests (#4298) @connortsui20
• chore: move field_names tests to field_names.rs (#4300) @connortsui20
• Add test coverage badge to README (#4299) @AdamGS
• Report coverage to codecov (#4296) @AdamGS
• chore(deps): update rust crate inventory to v0.3.21 (#4297) @renovate[bot]
• chore(deps): update rust crate taffy to v0.9.1 (#4295) @renovate[bot]
• Improve test coverage for conversion between DataFusion and Vortex objects (#4294) @AdamGS
• chore(deps): update rust crate thiserror to v2.0.16 (#4293) @renovate[bot]
• chore(deps): update rust crate tempfile to v3.21.0 (#4292) @renovate[bot]
• chore: DType documentation (#4291) @connortsui20
• chore(deps): update rust crate cfg-if to v1.0.3 (#4290) @renovate[bot]

🌪 0.46.1

Changes

🧰 Maintenance

• fix: timeout crates publish after 1hr (#4288) @a10y

🌪 0.46.0

Changes

✨ Features

• feat: ArrayOperations infallible, eager validation + new_unchecked (#4177) @a10y
• feat: teach VortexDataset how to limit the batch_size (#4275) @danking
• chore[vortex-duckdb]: duckdb explain custom display (#4272) @joseph-isaacs

🚀 Performance

• Add more non copying stats operations; replace stats replace with inherit_from (#4279) @robert3005

🐛 Bug Fixes

• Fix: converting from duckdb_string_t requires looking at length (#4277) @robert3005
• fix[duckdb]: set max cardinality in duckdb (#4270) @joseph-isaacs

🧰 Maintenance

14 changes

• chore: infinite publish timeout (#4286) @a10y
• chore(deps): update polarsignals/gh-actions-ps-profiling action to v0.6.0 (#4284) @renovate[bot]
• chore: monad vs. monoid (#4283) @connortsui20
• Add Display impl for FieldNames (#4282) @AdamGS
• chore(deps): update rust crate serde_json to v1.0.143 (#4280) @renovate[bot]
• Chore: Cleanup in take (#4278) @connortsui20
• docs: Add documentation to take function (#4274) @connortsui20
• chore: hide aliases from docs (#4273) @danking
• chore: Add rust-analyzer to the rust-toolchain.tom file (#4271) @connortsui20
• Enable TPC-DS benchmarks for datafusion (#4269) @AdamGS
• Remove old dead test code (#4265) @AdamGS
• chore(deps): update datafusion and friends to v49.0.1 (patch) (#4267) @renovate[bot]
• chore(deps): lock file maintenance (#4264) @renovate[bot]
• chore(deps): update rust crate thiserror to v2.0.15 (#4262) @renovate[bot]

🌪 0.45.0

Changes

✨ Features

• feat[spark]: write support in VortexDataSourceV2 (#4246) @lwwmanning
• feat: implement Display for StructFields (#4255) @blaginin
• feat: Support more datafusion features (#4180) @AdamGS
• Support converting TimestampTZ to and from duckdb (#4226) @robert3005
• Use perfetto tracing (#4250) @gatesn
• feat: Remove 'parquet' feature and some other dependency bloat (#4251) @AdamGS
• feat: Make FieldNames comparison more ergonomic (#4239) @AdamGS
• feat(c++): support passing filter and projection expression in scan (#4224) @XinyuZeng
• feat[expr]: add back the expression folder, implemented over Node (#4230) @joseph-isaacs
• feature: Add TryFrom impls for owned scalars to values (#4234) @AdamGS
• docs: add javadoc to docs.vortex.dev (#4229) @a10y
• feat: Expose various expr transformations directly instead of through module (#4212) @AdamGS
• feat[duckdb]: support more operators (#4170) @joseph-isaacs
• feat: teach Vortex how to export list arrays to duckdb (#4169) @danking
• feat(python): Write PyArrow types directly to Vortex (#4161) @a10y
• Inline the logic of the filter layout reader (#4089) @gatesn
• Support exporting Arrow RunArray to Vortex (#4162) @robert3005
• feature: Support converting Arrow DictArray into vortex (#4157) @robert3005
• refactor the LayoutStrategy (#4149) @a10y

🚀 Performance

• perf: Use sum compute function (#4207) @viirya

🐛 Bug Fixes

20 changes

• fix: Validity export using shifted u64 copy (#4244) @robert3005
• fix[bench]: better bench input size (#4243) @joseph-isaacs
• fix[bench-site]: Compression Size group missing charts (#4236) @lwwmanning
• Fix: Return currect field names in SelectExpr::as_include (#4232) @AdamGS
• fix: javadoc workflow (#4231) @a10y
• fix: Correctly convert arrow temporal arrays with timezone (#4227) @robert3005
• fix: runend decompress benchmarks (#4225) @0ax1
• fix: Avoid running projection evaluation for all false masks (#4209) @robert3005
• fix: small resource leak in vortex-io, plus test improvements (#4201) @lwwmanning
• fix: race condition in vortex-layout, plus test improvements (#4198) @lwwmanning
• fix[bench-website]: add tpc-ds to bench website (#4199) @joseph-isaacs
• shared waker in struct writer (#4193) @onursatici
• fix: duckdb dyn expr race on filter constant (#4168) @0ax1
• fix[scan]: don't convert filter to cnf just extract conjunct list (#4174) @joseph-isaacs
• fix: Fix some minor typos and remove the "single-file" feature from clickbench (#4182) @AdamGS
• fix: incorrect decimal scalar nbytes (#4176) @lwwmanning
• fix(pyvortex): encoding-specific dispatch (#4171) @a10y
• fix[duckdb]: validity ref u64 not u8 (#4172) @joseph-isaacs
• ScalarValue operations must go via Scalar (#4152) @robert3005
• Only coerce scalar values when necessary (#4135) @robert3005

🧰 Maintenance

55 changes

• chore: Fix all rustdoc warnings and enforce no warning in CI (#4261) @AdamGS
• fix(deps): update dependency org.junit.jupiter:junit-jupiter to v5.13.4 (#4260) @renovate[bot]
• chore[vortex-expr]: folder docs (#4238) @joseph-isaacs
• fix(deps): update dependency ch.qos.logback:logback-classic to v1.5.18 (#4259) @renovate[bot]
• fix(deps): update slf4j monorepo to v2.0.17 (patch) (#4258) @renovate[bot]
• fix(deps): update dependency org.junit.jupiter:junit-jupiter to v5.10.5 (#4257) @renovate[bot]
• chore: Add docs to vortex-expr (#4253) @AdamGS
• chore(deps): update rust crate cc to v1.2.33 (#4256) @renovate[bot]
• chore[fastlanes]: remove FoR unsigned transmute (#4252) @joseph-isaacs
• chore(deps): update plugin com.gradleup.shadow to v9.0.2 (#4249) @renovate[bot]
• fix(deps): update protobuf monorepo to v4.32.0 (minor) (#4248) @renovate[bot]
• chore(deps): update rust crate arbitrary to v1.4.2 (#4247) @renovate[bot]
• chore(deps): update rust crate async-trait to v0.1.89 (#4245) @renovate[bot]
• chore[vortex-array]: facilitate third-party ToArrowKernel implementations (#4237) @asubiotto
• chore: Allow coveralls step to fail in CI without failing the job (#4240) @AdamGS
• chore: speed up the benchmark page loading (#4235) @a10y
• docs: fill in missing javadoc (#4233) @a10y
• chore: expand runend decompress benchmarks (#4228) @0ax1
• fix(deps): update dependency org.immutables:value to v2.11.3 (#4222) @renovate[bot]
• chore(deps): update rust crate rayon to v1.11.0 (#4223) @renovate[bot]
• chore(deps): update all patch updates (patch) (#4211) @renovate[bot]
• chore: sequential consistency in vortex-scan (#4217) @lwwmanning
• chore: add vortex-ffi example to coverage (#4210) @lwwmanning
• chore: improve test coverage in vortex-scan (#4206) @lwwmanning
• chore: Fix random-access series colours and delete dead code from benchmarking website (#4213) @robert3005
• chore: improve vortex-dtype test coverage (#4204) @lwwmanning
• chore(deps): update all patch updates (patch) (#4205) @renovate[bot]
• chore(deps): update rust crate clap to v4.5.44 (#4202) @renovate[bot]
• Improve BoolArray::from_indices (#4200) @viirya
• chore[bench]: enable tpcds on label and merge (#4196) @joseph-isaacs
• chore(deps): update actions/checkout action to v5 (#4195) @renovate[bot]
• chore[bench]: enable tpcds on label and merge (#4187) @joseph-isaacs
• Remove for whom argument from SegmentSource (#4190) @gatesn
• chore(deps): update rust crate uuid to v1.18.0 (#4192) @renovate[bot]
• chore(deps): update rust crate glob to v0.3.3 (#4189) @renovate[bot]
• chore(deps): lock file maintenance (#4186) @renovate[bot]
• chore(deps): update rust crate sysinfo to 0.37.0 (#4185) @renovate[bot]
• fix(deps): update all patch updates to v1.0.168 (patch) (#4184) @renovate[bot]
• fix(deps): update all patch updates to v1.0.166 (patch) (#4183) @renovate[bot]
• chore(deps): update plugin com.gradleup.shadow to v9.0.1 (#4181) @renovate[bot]
• chore: improve vortex-mask testing (#4178) @lwwmanning
• chore(deps): update rust crate rustversion to v1.0.22 (#4179) @renovate[bot]
• build: Use Create cc to link to libc++ (#4173) @ssmccoy
• chore[bench-website]: add back tpc-ds to query_bench (#4155) @joseph-isaacs
• chore: clippy allow unreachable function (#4167) @0ax1
• chore(deps): update rust crate cc to v1.2.32 (#4166) @renovate[bot]
• fix(deps): update rust crate cxx to v1.0.165 (#4164) @renovate[bot]
• chore(deps): update all patch updates (patch) (#4163) @renovate[bot]
• chore: Bump msrv to 1.89 (#4159) @AdamGS
• chore: add CastKernel implementations for remaining array encodings (#4131) @lwwmanning
• chore: improve vortex-scalar tests, reorganize module structure (#4145) @lwwmanning
• chore: check out ddb version in ext build based on env var (#4153) @0ax1
• chore(deps): update rust crate taffy to 0.9.0 (#4154) @renovate[bot]
• chore(deps): update plugin com.gradleup.shadow to v9 (#4150) @renovate[bot]
• fix(deps): update all patch updates to v1.0.164 (patch) (#4151) @renovate[bot]

🌪 0.44.1

Changes

🐛 Bug Fixes

• Revert "fix: pr benchmark results (#4141)" (#4142) @0ax1
• fix: pr benchmark results (#4141) @0ax1

🧰 Maintenance

• tests: test writing empty file in PyVortex (#4147) @a10y
• chore(deps): update rust crate clap to v4.5.43 (#4143) @renovate[bot]

🌪 0.44.0

Changes

✨ Features

• feat: Require expressions to be Eq (#4139) @AdamGS

🐛 Bug Fixes

• single threaded duckdb copy (#4140) @onursatici
• wake wakers on all branches (#4136) @onursatici
• fix: error-prone f16 Scalar serde (#4113) @lwwmanning
• fix: Correctly handle sum of chunked float array with all null chunks (#4125) @robert3005

🧰 Maintenance

9 changes

• chore: Improve perf and make the sidebar not force full re-render of page (#4128) @AdamGS
• chore(deps): update actions/download-artifact action to v5 (#4134) @renovate[bot]
• fix(deps): update all patch updates to v1.0.163 (patch) (#4133) @renovate[bot]
• chore: Reapply "feat: implement Cast Kernel everywhere" (#4110) @lwwmanning
• chore: improve consistency tests (#4114) @lwwmanning
• chore: fuzz only uploads new corpus if it hasn't been updated in the meantime (#4124) @robert3005
• chore: playing around with the benchmark website own perf (#4126) @AdamGS
• chore[bench-website]: search hide empty sections (#4122) @joseph-isaacs
• chore: move clippy cpp.rs allow list to build.rs (#4123) @0ax1

🌪 0.43.0

Changes

⚠️ Breaks

• Remove scan async API (#4005) @gatesn

✨ Features

• feat: duckdb:vortex s3 support (#4100) @0ax1
• feat: implement Cast Kernel everywhere (#4086) @lwwmanning
• feat: Make all built-in expressions Eq + replace dyn-eq with our trait (#4091) @AdamGS
• feat: Add serialization support for Select expression (#4077) @AdamGS
• zip compute function (#4031) @onursatici
• feat: teach SparseArray to canonicalize lists (#3992) @danking
• feature: Add From implementation to the vtable in addition to dedicated trait (#4037) @AdamGS
• Remove scan async API (#4005) @gatesn

🚀 Performance

• perf[vortex-array]: use all_valid instead of invalid_count() == 0 (#4108) @joseph-isaacs
• perf[duckdb]: add file footer caching (#4104) @joseph-isaacs
• varbinview zip kernel (#4054) @onursatici
• varbinview builder buffer deduplication (#4036) @onursatici
• perf[bench]: dont delete duckdb.db between db runs (#4017) @joseph-isaacs

🐛 Bug Fixes

16 changes

• fix[duckdb]: glob expansion canonicalize input path (#4121) @joseph-isaacs
• fix[duckdb]: use executor in ddb writer (#4117) @joseph-isaacs
• TransposedStream in struct writer to wake other potentially waiting streams (#4107) @onursatici
• fix[duckdb]: make batch number unique over all threads (#4095) @joseph-isaacs
• fix: mobile benchmark rendering (#4096) @lwwmanning
• fix: Add select-expr deserialization (#4082) @AdamGS
• Fix prefetch accounting (#4072) @gatesn
• fix: compress benchmarks opts doesnt conflict with output (#4070) @robert3005
• fix: benchmark compress hang (#4051) @0ax1
• fix tui segment display for auxiliary children (#4044) @robert3005
• fix: Correctly calculate the size of FlatLayouts array flatbuffer size (#4049) @robert3005
• fix: Forbid SequenceScheme for String and Float dictionary codes as well as Integer (#4048) @robert3005
• fix: duckdb ext symbol visibility (#4046) @0ax1
• fix: Implement compact for ListArray (#4002) @robert3005
• fix: fix sql benchmarks option (#4023) @danking
• Empty chunks don't affect arrays validity, empty Mask is always both all true and all false (#4018) @robert3005

🧰 Maintenance

57 changes

• IO Fuzzer uses single threaded reader and writer (#4119) @robert3005
• chore: Add workflow to minimize fuzz corpuses (#4120) @robert3005
• chore(deps): update plugin com.gradleup.shadow to v8.3.9 (#4115) @renovate[bot]
• fix(deps): update dependency org.immutables:value to v2.11.2 (#4112) @renovate[bot]
• chore[bench]: nightly s3 vortex tpc-h(sf=100) (#4111) @joseph-isaacs
• Revert "feat: implement Cast Kernel everywhere" (#4109) @joseph-isaacs
• chore: ArbitraryArray::arbitrary_with (#4105) @blaginin
• chore(deps): lock file maintenance (#4103) @renovate[bot]
• chore[bench-site]: more mobile fixes (#4102) @lwwmanning
• chore[bench-site]: bench site CSS cleanup (#4099) @lwwmanning
• chore[bench-website]: adjust benchmark size and usability features (#4097) @joseph-isaacs
• chore[bench-site]: add aggregates of each group to bench site (#4098) @lwwmanning
• chore: fix some typos (#4088) @XinyuZeng
• chore[bench]: update vortex bench color (#4094) @joseph-isaacs
• chore(deps): update plugin com.palantir.consistent-versions to v2.37.0 (#4093) @renovate[bot]
• chore(deps): update all patch updates (patch) (#4092) @renovate[bot]
• chore: C++ API improvement (#4050) @XinyuZeng
• chore(deps): update rust crate serde_json to v1.0.142 (#4087) @renovate[bot]
• chore: make bench site both responsive and faster (#4084) @lwwmanning
• chore: stricter yaml validation (#4078) @robert3005
• chore: Add roundtrip serialization tests to vortex-expr (#4083) @AdamGS
• chore: rewrite the benchmark website (#4079) @lwwmanning
• chore: Enable polarsignals diffs on benchmarks (#4067) @AdamGS
• chore: Make vortex pretty (#4081) @AdamGS
• chore: No longer need to mount debugfs and tracefs (#4080) @robert3005
• chore: Manually mount tracefs and debugfs until we bump ami (#4076) @robert3005
• chore(deps): update rust crate divan to v3.0.5 (#4073) @renovate[bot]
• [chore] improve binary_numeric conformance (#4069) @lwwmanning
• Clean up decompress benchmarks (#4068) @gatesn
• [chore] cross-operation consistency tests (#4055) @lwwmanning
• chore: cleanup commented-out code in vortex-expr (#4066) @AdamGS
• chore: add into_cpu_stream (#4053) @blaginin
• test: block_in_place (#4059) @0ax1
• chore(deps): update rust crate clap to v4.5.42 (#4060) @renovate[bot]
• chore(deps): update rust crate crossterm to 0.29 (#2925) @renovate[bot]
• chore: reject warnings on clippy but not on check (#4052) @blaginin
• Clean up stats propagation for slicing (#3356) @gatesn
• [chore] conformance testing for TakeFn (#4043) @lwwmanning
• [chore] use rstest for filter & mask conformance test harnesses (#4045) @lwwmanning
• chore: add missing {doctest} (#4042) @blaginin
• [chore] add filter conformance test & fill existing conformance testing gaps (#4041) @lwwmanning
• chore: cleanup COMPUTE_FN definitions (#4040) @a10y
• docs: Document all canonical array types (#4039) @a10y
• chore: Timeout data downloads in benchmarks (#4038) @AdamGS
• use arc slice for buffers in varbinview (#4035) @onursatici
• chore: bump datafusion to 49 (#4006) @AdamGS
• chore(deps): lock file maintenance (#4030) @renovate[bot]
• chore(deps): update rust crate rstest to v0.26.1 (#4029) @renovate[bot]
• chore(deps): update rust crate rstest to 0.26 (#4026) @renovate[bot]
• chore(deps): update rust crate tokio to v1.47.0 (#4025) @renovate[bot]
• chore(deps): update rust crate roaring to v0.11.2 (#4024) @renovate[bot]
• chore: benchmark individual encoding throughputs (#4014) @lwwmanning
• chore: optionally delete the duckdb database on engine creation. (#4022) @danking
• chore: Install llvm in fuzzer for slightly better logging and remove submodules from checkout (#4015) @robert3005
• Remove segment source factory (#4016) @gatesn
• Add Claude Code GitHub Workflow (#4009) @robert3005
• chore: unit tests for Arrow conversion (#4007) @a10y

🌪 0.42.1

Changes

🐛 Bug Fixes

• fix[ci]: use cargo +nightly in publish (#4011) @joseph-isaacs