Skip to content

Commit 86cc337

Browse files
lukekimrobert3005myrrcjoseph-isaacsconnortsui20
authored
Fix decimal casts to primitive arrays (#51)
* AllInvalid arrays have non empty distinct info (#7369) This is required to handle cases where we are attempting to dict compress and a sample ends up being all null Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Robert Kruszewski <github@robertk.io> * duckdb: Reset validity instead of filling (#7365) 1. Reset validity if it's "all true" instead of filling manually. 2. Set validity to true if parent array is constant (slicing won't affect that). 3. Count ones while copying to destination buffer Signed-off-by: Mikhail Kot <to@myrrc.dev> * use expect over allow in clippy (#7373) remove #[allow(....) * Remove portable_simd take implementation and remove unused cargo flags (#7388) First of all we never use nightly so portable simd never gets enabled, second of all it looks like avx512 doesn't bring any benefits beyond avx2 for simd gather per gcc mailing list and Intel GDS mailing list. AVX2 on modern intel machines is comparable to Zen 5 machines and on Zen 5 machines AVX512, AVX2 and scalar versions behave the same. Signed-off-by: Robert Kruszewski <github@robertk.io> * Decompose TurboQuant into 2 `ScalarFnArray`s and `DictArray` (#7374) ## Summary Tracking issue: https://github.com/vortex-data/vortex/issues/7297 Decomposes `TurboQuant` into: ```text ScalarFnArray(L2Denorm, [ ScalarFnArray(SorfTransform, [ Extension<Vector>( FixedSizeListArray( DictArray(codes=Primitive<u8>, values=Primitive<f32>), padded_dim ) ) ]), norms ]) ``` This makes the implementation more modular and turns the TurboQuant-specific pieces into reusable building blocks. Also defines SORF sign generation with a frozen local `SplitMix64` implementation and cleans up related vector compute code. ## API Changes - add `SorfTransform` - remove the `TurboQuant` encoding/array type - keep `TurboQuantScheme` as the compressor entry point - simplify a all scalar fn APIs as we no longer need `ApproxOptions` Note that `ApproxOptions` doesn't actually make sense here because we are doing exact compute here, it is the encoding itself that is lossy. Until we figure out what the exact semantics are of a lossy encoding I will remove this. ## Testing More tests: - TurboQuant roundtrip / structural coverage - SORF roundtrip + norm preservation - deterministic SplitMix64 / sign generation coverage - readthrough behavior (when children are normalized) for vector similarity ops --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> * add vector similarity benchmark (#7391) ## Summary Tracking issue: https://github.com/vortex-data/vortex/issues/7297 Adds basic benchmarking setup for vector similarity. Right now it is just a bunch of random vectors. Note that the numbers dont really mean anything right now as we have yet to optimize anything (namely I have not yet added the inner product / cosine similarity optimizations pushed through both the SORF transform and the dictionary for constant array). In the future we will add proper benchmarking on real datasets (likely in `vortex-bench`, and also maybe we will integrate https://github.com/zilliztech/vectordbbench ## Testing N/A Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> * perf: `Arc::ptr_eq` short-circuit to DType eq comparisons (#7398) FieldName, FieldNames, and StructFields all wrap Arc types but their derived PartialEq always dereferences through the Arc to compare contents. Since DTypes are frequently cloned (every slice, filter, execute copies the DType), the cloned Arcs share pointers - making pointer equality a reliable fast path. DataFusion ClickBench full-suite [apmc](https://github.com/0ax1/apmc) measurement for vortex, averaged over two runs: - Cycles: -5.7% (861B → 812B) — fewer total CPU cycles - IPC: +5.8% (1.88 → 1.99) — more instructions per cycle - MAP_STALL: -5.5% (360B → 340B) — less CPU stalls waiting on memory Signed-off-by: Alexander Droste <alexander.droste@protonmail.com> * Update dependency io.netty:netty-bom to v4.2.12.Final (#7403) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [io.netty:netty-bom](https://netty.io/) ([source](https://redirect.github.com/netty/netty)) | `4.2.7.Final` → `4.2.12.Final` | ![age](https://developer.mend.io/api/mc/badges/age/maven/io.netty:netty-bom/4.2.12.Final?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/maven/io.netty:netty-bom/4.2.7.Final/4.2.12.Final?slim=true) | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/357) for more information. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMTAuMiIsInVwZGF0ZWRJblZlciI6IjQzLjExMC4yIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update react monorepo to v19.2.5 (#7404) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [react](https://react.dev/) ([source](https://redirect.github.com/facebook/react/tree/HEAD/packages/react)) | [`19.2.4` → `19.2.5`](https://renovatebot.com/diffs/npm/react/19.2.4/19.2.5) | ![age](https://developer.mend.io/api/mc/badges/age/npm/react/19.2.5?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/react/19.2.4/19.2.5?slim=true) | | [react-dom](https://react.dev/) ([source](https://redirect.github.com/facebook/react/tree/HEAD/packages/react-dom)) | [`19.2.4` → `19.2.5`](https://renovatebot.com/diffs/npm/react-dom/19.2.4/19.2.5) | ![age](https://developer.mend.io/api/mc/badges/age/npm/react-dom/19.2.5?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/react-dom/19.2.4/19.2.5?slim=true) | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/357) for more information. --- ### Release Notes <details> <summary>facebook/react (react)</summary> ### [`v19.2.5`](https://redirect.github.com/facebook/react/releases/tag/v19.2.5): 19.2.5 (April 8th, 2026) [Compare Source](https://redirect.github.com/facebook/react/compare/v19.2.4...v19.2.5) ##### React Server Components - Add more cycle protections ([#&#8203;36236](https://redirect.github.com/facebook/react/pull/36236) by [@&#8203;eps1lon](https://redirect.github.com/eps1lon) and [@&#8203;unstubbable](https://redirect.github.com/unstubbable)) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMTAuMiIsInVwZGF0ZWRJblZlciI6IjQzLjExMC4yIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update rust-wasm-bindgen monorepo (#7406) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [js-sys](https://wasm-bindgen.github.io/wasm-bindgen/) ([source](https://redirect.github.com/wasm-bindgen/wasm-bindgen/tree/HEAD/crates/js-sys)) | dependencies | patch | `0.3.91` → `0.3.95` | | [wasm-bindgen](https://wasm-bindgen.github.io/wasm-bindgen) ([source](https://redirect.github.com/wasm-bindgen/wasm-bindgen)) | dependencies | patch | `0.2.114` → `0.2.118` | | [wasm-bindgen-futures](https://wasm-bindgen.github.io/wasm-bindgen/) ([source](https://redirect.github.com/wasm-bindgen/wasm-bindgen/tree/HEAD/crates/futures)) | workspace.dependencies | patch | `0.4.64` → `0.4.68` | | [web-sys](https://wasm-bindgen.github.io/wasm-bindgen/web-sys/index.html) ([source](https://redirect.github.com/wasm-bindgen/wasm-bindgen/tree/HEAD/crates/web-sys)) | dependencies | patch | `0.3.91` → `0.3.95` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/357) for more information. --- ### Release Notes <details> <summary>wasm-bindgen/wasm-bindgen (wasm-bindgen)</summary> ### [`v0.2.118`](https://redirect.github.com/wasm-bindgen/wasm-bindgen/blob/HEAD/CHANGELOG.md#02118) [Compare Source](https://redirect.github.com/wasm-bindgen/wasm-bindgen/compare/0.2.117...0.2.118) ##### Added - Added `Error::stack_trace_limit()` and `Error::set_stack_trace_limit()` bindings to `js-sys` for the non-standard V8 `Error.stackTraceLimit` property. [#&#8203;5082](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5082) - Added support for multiple `#[wasm_bindgen(start)]` functions, which are chained together at initialization, as well as a new `#[wasm_bindgen(start, private)]` to register a start function without exporting it as a public export. [#&#8203;5081](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5081) - Reinitialization is no longer automatically applied when using `panic=unwind` and `--experimental-reset-state-function`, instead it is triggered by any use of the `handler::schedule_reinit()` function under `panic=unwind`, which is supported from within the `on_abort` handler for reinit workflows. Renamed `handler::reinit()` to `handler::schedule_reinit()` and removed the `set_on_reinit()` handler. The `__instance_terminated` address is now always a simple boolean (`0` = live, `1` = terminated). [#&#8203;5083](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5083) - `handler::schedule_reinit()` now works under `panic=abort` builds. Previously it was a no-op; it now sets the JS-side reinit flag and the next export call transparently creates a fresh `WebAssembly.Instance`. [#&#8203;5099](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5099) ##### Changed - MSRV bump from 1.71 to 1.76 for the CLI, and 1.82 to 1.86 for the API [#&#8203;5102](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5102) ##### Fixed - ES module `import` statements are now hoisted to the top of generated JS files, placed right after the `@ts-self-types` directive. This ensures valid ES module output since `import` declarations must precede other statements. [#&#8203;5103](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5103) - Fixed two CLI issues affecting WASM modules built by rustc 1.94+. First, a panic (`failed to find N in function table`) caused by lld emitting element segment offsets as `global.get $__table_base` or extended const expressions instead of plain `i32.const N` for large function tables; the fix adds a const-expression evaluator in `get_function_table_entry` and guards against integer underflow in multi-segment tables. Second, the descriptor interpreter now routes all global reads/writes through a single `globals` HashMap seeded from the module's own globals, and mirrors the module's actual linear memory rather than a fixed 32KB buffer, so the stack pointer's real value is valid without any override. This fixes panics like `failed to find 32752 in function table` caused by `GOT.func.internal.*` globals being misidentified as the stack pointer. [#&#8203;5076](https://redirect.github.com/wasm-bindgen/wasm-bindgen/issues/5076) [#&#8203;5080](https://redirect.github.com/wasm-bindgen/wasm-bindgen/issues/5080) [#&#8203;5093](https://redirect.github.com/wasm-bindgen/wasm-bindgen/issues/5093) [#&#8203;5095](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5095) ### [`v0.2.117`](https://redirect.github.com/wasm-bindgen/wasm-bindgen/blob/HEAD/CHANGELOG.md#02117) [Compare Source](https://redirect.github.com/wasm-bindgen/wasm-bindgen/compare/0.2.116...0.2.117) ##### Fixed - Fixed a regression introduced in [#&#8203;5026](https://redirect.github.com/wasm-bindgen/wasm-bindgen/issues/5026) where stable `web-sys` methods that accept a union type containing a `[WbgGeneric]` interface (e.g. `ImageBitmapSource`, which includes `VideoFrame`) incorrectly applied typed generics to all union expansions rather than only those whose argument type is itself `[WbgGeneric]`. In practice this caused `Window::create_image_bitmap_with_*` and the corresponding `WorkerGlobalScope` overloads to return `Promise<ImageBitmap>` instead of `Promise<JsValue>` for the stable (non-`VideoFrame`) call sites, breaking `JsFuture::from(promise).await?`. [#&#8203;5064](https://redirect.github.com/wasm-bindgen/wasm-bindgen/issues/5064) [#&#8203;5073](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5073) - Fixed handling logic for environment variable `WASM_BINDGEN_TEST_ADDRESS` in the test runner, when running tests in headless mode. [#&#8203;5087](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5087) ### [`v0.2.116`](https://redirect.github.com/wasm-bindgen/wasm-bindgen/blob/HEAD/CHANGELOG.md#02116) [Compare Source](https://redirect.github.com/wasm-bindgen/wasm-bindgen/compare/0.2.115...0.2.116) ##### Added - Added `js_sys::Float16Array` bindings, `DataView` float16 accessors using `f32`, and raw `[u16]` helper APIs for interoperability with binary16 representations such as `half::f16`. [#&#8203;5033](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5033) ##### Changed - Updated to Walrus 0.26.1 for deterministic type section ordering. [#&#8203;5069](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5069) - The `#[wasm_bindgen]` macro now emits `&mut (impl FnMut(...) + MaybeUnwindSafe)` / `&(impl Fn(...) + MaybeUnwindSafe)` for raw `&mut dyn FnMut` / `&dyn Fn` import arguments instead of a hidden generic parameter and where-clause. The generated signature is cleaner and the `MaybeUnwindSafe` bound is visible directly in the argument position. The ABI and wire format are unchanged. When building with `panic=unwind`, closures that capture non-`UnwindSafe` values (e.g. `&mut T`, `Cell<T>`) must wrap them in `AssertUnwindSafe` before capture; on all other targets `MaybeUnwindSafe` is a no-op blanket impl. [#&#8203;5056](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5056) ### [`v0.2.115`](https://redirect.github.com/wasm-bindgen/wasm-bindgen/blob/HEAD/CHANGELOG.md#02115) [Compare Source](https://redirect.github.com/wasm-bindgen/wasm-bindgen/compare/0.2.114...0.2.115) ##### Added - `console.debug/log/info/warn/error` output from user-spawned `Worker` and `SharedWorker` instances is now forwarded to the CLI test runner during headless browser tests, just like output from the main thread. Works for blob URL workers, module workers, URL-based workers (importScripts), nested workers, and shared workers (including logs emitted before the first port connection). Non-cloneable arguments are serialized via `String()` rather than crashing the worker. The `--nocapture` flag is respected. [#&#8203;5037](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5037) - `js_sys::Promise<T>` now implements `IntoFuture`, enabling direct `.await` on any JS promise without a wrapper type. The `wasm-bindgen-futures` implementation has been moved into `js-sys` behind an optional `futures` feature, which is activated automatically when `wasm-bindgen-futures` is a dependency. All existing `wasm_bindgen_futures::*` import paths continue to work unchanged via re-exports. `js_sys::futures` is also available directly for users who want `promise.await` without depending on `wasm-bindgen-futures`. [#&#8203;5049](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5049) - Added `--target emscripten` support, generating a `library_bindgen.js` file for consumption by Emscripten at link time. Includes support for futures, JS closures, and TypeScript output. A new Emscripten-specific test runner is also included, along with CI integration. [#&#8203;4443](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/4443) - Added `VideoFrame`, `VideoColorSpace`, and related WebCodecs dictionaries/enums to `web-sys`. [#&#8203;5008](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5008) - Added `wasm_bindgen::handler` module with `set_on_abort` and `set_on_reinit` hooks for `panic=unwind` builds. `set_on_abort` registers a callback invoked after the instance is terminated (hard abort, OOM, stack overflow). `set_on_reinit` registers a callback invoked after `reinit()` resets the WebAssembly instance via `--experimental-reset-state-function`. Handlers are stored as Wasm indirect-function-table indices so dispatch is safe even when linear memory is corrupt. ##### Changed - Replaced per-closure generic destructors with a single `__wbindgen_destroy_closure` export. [#&#8203;5019](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5019) - Refactored the headless browser test runner logging pipeline for dramatically improved performance (>400x faster on Chrome, >10x on Firefox, \~5x on Safari). Switched to incremental DOM scraping with `textContent.slice(offset)`, append-only output semantics, unified log capture across all log levels on failure, and browser-specific invisible-div optimizations (`display:none` for Chrome/Firefox, `visibility:hidden` for Safari). [#&#8203;4960](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/4960) - TTY-gated status/clear output in the test runner shell to avoid `\r` control-character artifacts in non-interactive (CI) environments. [#&#8203;4960](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/4960) - Added `bench_console_log_10mb` benchmark alongside the existing 1MB benchmark for the headless test runner. The main branch cannot complete this benchmark at any volume. [#&#8203;4960](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/4960) - Updated to Walrus 0.26 [#&#8203;5057](https://redirect.github.com/wasm-bindgen/walrus/pull/5057) ##### Fixed - Fixed argument order when calling multi-parameter functions in the `wasm-bindgen` interpreter by reversing the args collected from the stack. [#&#8203;5047](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5047) - Added support for per-operation `[WbgGeneric]` in WebIDL, restoring typed generic return types (e.g. `Promise<ImageBitmap>`) for `createImageBitmap` on `Window` and `WorkerGlobalScope` that were lost after the `VideoFrame` stabilization. [#&#8203;5026](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5026) - Fixed missing `#[cfg(feature = "...")]` gates on deprecated dictionary builder methods and getters for union-typed fields (e.g. `{Open,Save,Directory}FilePickerOptions::start_in()`), and fixed per-setter doc requirements to list each setter's own required features. [#&#8203;5039](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5039) - Fixed `JsOption::new()` to use `undefined` instead of `null`, to be compatible with `Option::None` and JS default parameters. [#&#8203;5023](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5023) - Fixed unsound `unsafe` transmutes in `JsOption<T>::wrap`, `as_option`, and `into_option` by replacing `transmute_copy` with `unchecked_into()`. Also tightened the `JsGeneric` trait bound and `JsOption<T>` impl block to require `T: JsGeneric` (which implies `JsCast`), preventing use with arbitrary non-JS types. [#&#8203;5030](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5030) - Fixed headless test runner emitting `\r` carriage-return sequences in non-TTY environments, which polluted captured logs in CI and complicated output-matching tests. [#&#8203;4960](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/4960) - Fixed headless test runner printing incomplete and out-of-order log output on test failures by merging all five log levels into a single unified output div. [#&#8203;4960](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/4960) - Fixed large test outputs (10MB+) causing oversized WebDriver responses that were either extremely slow or crashed completely, by switching to incremental streaming output collection. [#&#8203;4960](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/4960) - Fixed a duplciate wasm export in node ESM atomics, when compiled in debug mode [#&#8203;5028](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5028) - Fixed a type inference regression (`E0283: type annotations needed`) introduced in v0.2.109 where the stable `FromIterator` and `Extend` impls on `js_sys::Array` were changed from `A: AsRef<JsValue>` to `A: AsRef<T>`. Because `#[wasm_bindgen]` generates multiple `AsRef` impls per type, the compiler could not uniquely resolve `T`, breaking code like `Array::from_iter([my_wasm_value])` without explicit annotations. The stable impls are restored to `A: AsRef<JsValue>` (returning `Array<JsValue>`); the generic `A: AsRef<T>` forms remain available under `js_sys_unstable_apis`. [#&#8203;5052](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5052) - Fixed `skip_typescript` not being respected when using `reexport`, causing TypeScript definitions to be incorrectly emitted for re-exported items marked with `#[wasm_bindgen(skip_typescript)]`. [#&#8203;5051](https://redirect.github.com/wasm-bindgen/wasm-bindgen/pull/5051) ##### Removed </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMTAuMiIsInVwZGF0ZWRJblZlciI6IjQzLjExMC4yIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * bump rand to 0.10.1 (#7405) ## Summary Bumps `rand` to `0.10.1`. - https://rustsec.org/advisories/RUSTSEC-2026-0097 - https://github.com/rust-random/rand/pull/1763 ## Testing N/A Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> * Update all patch updates (#7402) This PR contains the following updates: | Package | Type | Update | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---|---|---| | [arc-swap](https://redirect.github.com/vorner/arc-swap) | workspace.dependencies | patch | `1.9.0` → `1.9.1` | ![age](https://developer.mend.io/api/mc/badges/age/crate/arc-swap/1.9.1?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/arc-swap/1.9.0/1.9.1?slim=true) | | [cc](https://redirect.github.com/rust-lang/cc-rs) | workspace.dependencies | patch | `1.2.57` → `1.2.60` | ![age](https://developer.mend.io/api/mc/badges/age/crate/cc/1.2.60?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/cc/1.2.57/1.2.60?slim=true) | | [custom-labels](https://polarsignals.com) ([source](https://redirect.github.com/polarsignals/custom-labels)) | workspace.dependencies | patch | `0.4.5` → `0.4.6` | ![age](https://developer.mend.io/api/mc/badges/age/crate/custom-labels/0.4.6?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/custom-labels/0.4.5/0.4.6?slim=true) | | [env_logger](https://redirect.github.com/rust-cli/env_logger) | workspace.dependencies | patch | `0.11.9` → `0.11.10` | ![age](https://developer.mend.io/api/mc/badges/age/crate/env_logger/0.11.10?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/env_logger/0.11.9/0.11.10?slim=true) | | [fsst-rs](https://redirect.github.com/spiraldb/fsst) | workspace.dependencies | patch | `0.5.9` → `0.5.10` | ![age](https://developer.mend.io/api/mc/badges/age/crate/fsst-rs/0.5.10?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/fsst-rs/0.5.9/0.5.10?slim=true) | | [inventory](https://redirect.github.com/dtolnay/inventory) | workspace.dependencies | patch | `0.3.22` → `0.3.24` | ![age](https://developer.mend.io/api/mc/badges/age/crate/inventory/0.3.24?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/inventory/0.3.22/0.3.24?slim=true) | | [object_store](https://redirect.github.com/apache/arrow-rs-object-store) | workspace.dependencies | patch | `0.13.1` → `0.13.2` | ![age](https://developer.mend.io/api/mc/badges/age/crate/object_store/0.13.2?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/object_store/0.13.1/0.13.2?slim=true) | | [prettier](https://prettier.io) ([source](https://redirect.github.com/prettier/prettier)) | devDependencies | patch | [`3.8.1` → `3.8.2`](https://renovatebot.com/diffs/npm/prettier/3.8.1/3.8.2) | ![age](https://developer.mend.io/api/mc/badges/age/npm/prettier/3.8.2?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/prettier/3.8.1/3.8.2?slim=true) | | [pyo3](https://redirect.github.com/pyo3/pyo3) | workspace.dependencies | patch | `0.28.2` → `0.28.3` | ![age](https://developer.mend.io/api/mc/badges/age/crate/pyo3/0.28.3?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/pyo3/0.28.2/0.28.3?slim=true) | | [rustc-hash](https://redirect.github.com/rust-lang/rustc-hash) | workspace.dependencies | patch | `2.1.1` → `2.1.2` | ![age](https://developer.mend.io/api/mc/badges/age/crate/rustc-hash/2.1.2?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/crate/rustc-hash/2.1.1/2.1.2?slim=true) | | com.gradleup.shadow | plugin | patch | `9.4.0` → `9.4.1` | ![age](https://developer.mend.io/api/mc/badges/age/maven/com.gradleup.shadow:com.gradleup.shadow.gradle.plugin/9.4.1?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/maven/com.gradleup.shadow:com.gradleup.shadow.gradle.plugin/9.4.0/9.4.1?slim=true) | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/357) for more information. --- ### Release Notes <details> <summary>vorner/arc-swap (arc-swap)</summary> ### [`v1.9.1`](https://redirect.github.com/vorner/arc-swap/blob/HEAD/CHANGELOG.md#191) [Compare Source](https://redirect.github.com/vorner/arc-swap/compare/v1.9.0...v1.9.1) - One more SeqCst :-| ([#&#8203;204](https://redirect.github.com/vorner/arc-swap/issues/204)). </details> <details> <summary>rust-lang/cc-rs (cc)</summary> ### [`v1.2.60`](https://redirect.github.com/rust-lang/cc-rs/blob/HEAD/CHANGELOG.md#1260---2026-04-10) [Compare Source](https://redirect.github.com/rust-lang/cc-rs/compare/cc-v1.2.59...cc-v1.2.60) ##### Fixed - *(ar)* suppress warnings from `D` modifier probe ([#&#8203;1700](https://redirect.github.com/rust-lang/cc-rs/pull/1700)) ### [`v1.2.59`](https://redirect.github.com/rust-lang/cc-rs/blob/HEAD/CHANGELOG.md#1259---2026-04-03) [Compare Source](https://redirect.github.com/rust-lang/cc-rs/compare/cc-v1.2.58...cc-v1.2.59) ##### Fixed - *(ar)* deterministic archives with `D` modifier ([#&#8203;1697](https://redirect.github.com/rust-lang/cc-rs/pull/1697)) ##### Other - Regenerate target info ([#&#8203;1698](https://redirect.github.com/rust-lang/cc-rs/pull/1698)) - Fix target abi parsing for sanitiser targets ([#&#8203;1695](https://redirect.github.com/rust-lang/cc-rs/pull/1695)) ### [`v1.2.58`](https://redirect.github.com/rust-lang/cc-rs/blob/HEAD/CHANGELOG.md#1258---2026-03-27) [Compare Source](https://redirect.github.com/rust-lang/cc-rs/compare/cc-v1.2.57...cc-v1.2.58) ##### Other - Update Compile-time Requirements to add info about clang-cl.exe ([#&#8203;1693](https://redirect.github.com/rust-lang/cc-rs/pull/1693)) </details> <details> <summary>rust-cli/env_logger (env_logger)</summary> ### [`v0.11.10`](https://redirect.github.com/rust-cli/env_logger/blob/HEAD/CHANGELOG.md#01110---2026-03-23) [Compare Source](https://redirect.github.com/rust-cli/env_logger/compare/v0.11.9...v0.11.10) ##### Internal - Update dependencies </details> <details> <summary>spiraldb/fsst (fsst-rs)</summary> ### [`v0.5.10`](https://redirect.github.com/spiraldb/fsst/releases/tag/0.5.10) [Compare Source](https://redirect.github.com/spiraldb/fsst/compare/0.5.9...0.5.10) #### Changes - feat: prune low-value symbols from table on small inputs ([#&#8203;203](https://redirect.github.com/spiraldb/fsst/issues/203)) [@&#8203;CommanderStorm](https://redirect.github.com/CommanderStorm) - chore(deps): update codspeedhq/action digest to [`db35df7`](https://redirect.github.com/spiraldb/fsst/commit/db35df7) ([#&#8203;201](https://redirect.github.com/spiraldb/fsst/issues/201)) @&#8203;[renovate\[bot\]](https://redirect.github.com/apps/renovate) - chore(deps): update taiki-e/cache-cargo-install-action digest to [`511847d`](https://redirect.github.com/spiraldb/fsst/commit/511847d) ([#&#8203;198](https://redirect.github.com/spiraldb/fsst/issues/198)) @&#8203;[renovate\[bot\]](https://redirect.github.com/apps/renovate) - chore(deps): pin dtolnay/rust-toolchain action to [`3c5f7ea`](https://redirect.github.com/spiraldb/fsst/commit/3c5f7ea) ([#&#8203;197](https://redirect.github.com/spiraldb/fsst/issues/197)) @&#8203;[renovate\[bot\]](https://redirect.github.com/apps/renovate) - chore(deps): lock file maintenance ([#&#8203;196](https://redirect.github.com/spiraldb/fsst/issues/196)) @&#8203;[renovate\[bot\]](https://redirect.github.com/apps/renovate) - chore(deps): pin release-drafter/release-drafter action to [`139054a`](https://redirect.github.com/spiraldb/fsst/commit/139054a) ([#&#8203;194](https://redirect.github.com/spiraldb/fsst/issues/194)) @&#8203;[renovate\[bot\]](https://redirect.github.com/apps/renovate) - chore(deps): update taiki-e/cache-cargo-install-action digest to [`7824a3d`](https://redirect.github.com/spiraldb/fsst/commit/7824a3d) ([#&#8203;195](https://redirect.github.com/spiraldb/fsst/issues/195)) @&#8203;[renovate\[bot\]](https://redirect.github.com/apps/renovate) - Update commments in the decompress method ([#&#8203;193](https://redirect.github.com/spiraldb/fsst/issues/193)) [@&#8203;robert3005](https://redirect.github.com/robert3005) </details> <details> <summary>dtolnay/inventory (inventory)</summary> ### [`v0.3.24`](https://redirect.github.com/dtolnay/inventory/releases/tag/0.3.24) [Compare Source](https://redirect.github.com/dtolnay/inventory/compare/0.3.23...0.3.24) - Add support for VxWorks targets ([#&#8203;89](https://redirect.github.com/dtolnay/inventory/issues/89), thanks [@&#8203;elBoberido](https://redirect.github.com/elBoberido)) ### [`v0.3.23`](https://redirect.github.com/dtolnay/inventory/releases/tag/0.3.23) [Compare Source](https://redirect.github.com/dtolnay/inventory/compare/0.3.22...0.3.23) - Avoid triggering clippy::disallowed\_types in downstream projects that use Loom ([#&#8203;88](https://redirect.github.com/dtolnay/inventory/issues/88), thanks [@&#8203;elBoberido](https://redirect.github.com/elBoberido)) </details> <details> <summary>apache/arrow-rs-object-store (object_store)</summary> ### [`v0.13.2`](https://redirect.github.com/apache/arrow-rs-object-store/blob/HEAD/CHANGELOG.md#v0132-2026-03-19) [Compare Source](https://redirect.github.com/apache/arrow-rs-object-store/compare/v0.13.1...v0.13.2) [Full Changelog](https://redirect.github.com/apache/arrow-rs-object-store/compare/v0.12.5...v0.13.2) **Implemented enhancements:** - `Path::join(Self, &PathPart) -> Self` [#&#8203;665](https://redirect.github.com/apache/arrow-rs-object-store/issues/665) - Support for AWS Encryption Client encryption [#&#8203;647](https://redirect.github.com/apache/arrow-rs-object-store/issues/647) - `LocalFileSystem`: use `read_at` instead of seek + read [#&#8203;622](https://redirect.github.com/apache/arrow-rs-object-store/issues/622) - Avoid reading metadata for `LocalFileSystem::read_ranges` (and other methods) [#&#8203;614](https://redirect.github.com/apache/arrow-rs-object-store/issues/614) - expose `Inner` from `HttpRequestBody` [#&#8203;606](https://redirect.github.com/apache/arrow-rs-object-store/issues/606) - Release object store `0.13.1` (maintenance) - Target Jan 2026 [#&#8203;598](https://redirect.github.com/apache/arrow-rs-object-store/issues/598) - Support AWS\_ENDPOINT\_URL\_S3 in aws backend [#&#8203;589](https://redirect.github.com/apache/arrow-rs-object-store/issues/589) - Release object store `0.12.5` (maintenance) - Target Dec 2025 [#&#8203;582](https://redirect.github.com/apache/arrow-rs-object-store/issues/582) - Support upper-case configuration options in parse\_url\_opts [#&#8203;529](https://redirect.github.com/apache/arrow-rs-object-store/issues/529) - object\_store: Support `{az,abfs,abfss}://container@account.blob.{core.windows.net,fabric.microsoft.com}` URLs [#&#8203;430](https://redirect.github.com/apache/arrow-rs-object-store/issues/430) - Support `Transfer-Encoding: chunked` responses in HttpStore [#&#8203;340](https://redirect.github.com/apache/arrow-rs-object-store/issues/340) - Use reconstructed ListBlobs marker to provide list offset support in `MicrosoftAzure` store [#&#8203;461](https://redirect.github.com/apache/arrow-rs-object-store/issues/461) **Fixed bugs:** - Azure Fabric: Unsigned integer underflow when fetching token causes integer overflow panic [#&#8203;640](https://redirect.github.com/apache/arrow-rs-object-store/issues/640) - Error body missing for 5xx errors after retry exhausted [#&#8203;617](https://redirect.github.com/apache/arrow-rs-object-store/issues/617) - Heavy contention on credentials cache [#&#8203;541](https://redirect.github.com/apache/arrow-rs-object-store/issues/541) - AWS/S3 Default Headers are not considered for signature calculation [#&#8203;484](https://redirect.github.com/apache/arrow-rs-object-store/issues/484) - az:// \<container> not work as expected [#&#8203;443](https://redirect.github.com/apache/arrow-rs-object-store/issues/443) **Performance improvements:** - Preallocate single `Vec` in `get_ranges` for LocalFilesystem [#&#8203;634](https://redirect.github.com/apache/arrow-rs-object-store/issues/634) - Use platform specific `read_at` when available [#&#8203;628](https://redirect.github.com/apache/arrow-rs-object-store/pull/628) ([AdamGS](https://redirect.github.com/AdamGS)) - Avoid metadata lookup for `LocalFileSystem::read_ranges` and `chunked_stream` [#&#8203;621](https://redirect.github.com/apache/arrow-rs-object-store/pull/621) ([Dandandan](https://redirect.github.com/Dandandan)) **Closed issues:** - \[Security Alert] Exposed API key(s) detected: AWS Access Key [#&#8203;659](https://redirect.github.com/apache/arrow-rs-object-store/issues/659) - AWS S3 token expired on multi-threaded app with Arc usage [#&#8203;655](https://redirect.github.com/apache/arrow-rs-object-store/issues/655) - Emulator tests fail in CI due to an unsupported service version header in Azurite [#&#8203;626](https://redirect.github.com/apache/arrow-rs-object-store/issues/626) **Merged pull requests:** - Replace `Path::child` with `Path::join` [#&#8203;666](https://redirect.github.com/apache/arrow-rs-object-store/pull/666) ([Kinrany](https://redirect.github.com/Kinrany)) - Support --xa-s3 suffix for S3 Express One Zone bucket access points [#&#8203;663](https://redirect.github.com/apache/arrow-rs-object-store/pull/663) ([pdeva](https://redirect.github.com/pdeva)) - docs: clarify `Clone` behavior [#&#8203;656](https://redirect.github.com/apache/arrow-rs-object-store/pull/656) ([crepererum](https://redirect.github.com/crepererum)) - Implement Clone for local and memory stores [#&#8203;653](https://redirect.github.com/apache/arrow-rs-object-store/pull/653) ([DoumanAsh](https://redirect.github.com/DoumanAsh)) - Unify `from_env` behaviours [#&#8203;652](https://redirect.github.com/apache/arrow-rs-object-store/pull/652) ([miraclx](https://redirect.github.com/miraclx)) - docs: add examples to the aws docs where appropriate [#&#8203;651](https://redirect.github.com/apache/arrow-rs-object-store/pull/651) ([CommanderStorm](https://redirect.github.com/CommanderStorm)) - Switch TokenCache to RWLock [#&#8203;648](https://redirect.github.com/apache/arrow-rs-object-store/pull/648) ([tustvold](https://redirect.github.com/tustvold)) - Minimize futures dependency into relevant sub-crates [#&#8203;646](https://redirect.github.com/apache/arrow-rs-object-store/pull/646) ([AdamGS](https://redirect.github.com/AdamGS)) - Clarify ShuffleResolver doc-comments [#&#8203;645](https://redirect.github.com/apache/arrow-rs-object-store/pull/645) ([jkosh44](https://redirect.github.com/jkosh44)) - Introduce a "tokio" to allow pulling a trait-only build [#&#8203;644](https://redirect.github.com/apache/arrow-rs-object-store/pull/644) ([AdamGS](https://redirect.github.com/AdamGS)) - fix(azure): fix integer overflow in Fabric token expiry check [#&#8203;641](https://redirect.github.com/apache/arrow-rs-object-store/pull/641) ([desmondcheongzx](https://redirect.github.com/desmondcheongzx)) - chore: upgrade to `rand` 0.10 [#&#8203;637](https://redirect.github.com/apache/arrow-rs-object-store/pull/637) ([crepererum](https://redirect.github.com/crepererum)) - fix(aws): Include default headers in signature calculation ([#&#8203;484](https://redirect.github.com/apache/arrow-rs-object-store/issues/484)) [#&#8203;636](https://redirect.github.com/apache/arrow-rs-object-store/pull/636) ([singhsaabir](https://redirect.github.com/singhsaabir)) - fix(azure): correct Microsoft Fabric blob endpoint domain [#&#8203;631](https://redirect.github.com/apache/arrow-rs-object-store/pull/631) ([kevinjqliu](https://redirect.github.com/kevinjqliu)) - Unblock emulator based tests [#&#8203;627](https://redirect.github.com/apache/arrow-rs-object-store/pull/627) ([AdamGS](https://redirect.github.com/AdamGS)) - Azure ADLS list\_with\_offset support [#&#8203;623](https://redirect.github.com/apache/arrow-rs-object-store/pull/623) ([omar](https://redirect.github.com/omar)) - Implement tests for range and partial content responses [#&#8203;619](https://redirect.github.com/apache/arrow-rs-object-store/pull/619) ([vitoordaz](https://redirect.github.com/vitoordaz)) - fix: missing 5xx error body when retry exhausted [#&#8203;618](https://redirect.github.com/apache/arrow-rs-object-store/pull/618) ([jackye1995](https://redirect.github.com/jackye1995)) - build(deps): update nix requirement from 0.30.0 to 0.31.1 [#&#8203;616](https://redirect.github.com/apache/arrow-rs-object-store/pull/616) ([dependabot\[bot\]](https://redirect.github.com/apps/dependabot)) - Clarify behavior of `parse_url_opts` with regards to case sensitivity [#&#8203;613](https://redirect.github.com/apache/arrow-rs-object-store/pull/613) ([AdamGS](https://redirect.github.com/AdamGS)) - Fix logical format conflict [#&#8203;605](https://redirect.github.com/apache/arrow-rs-object-store/pull/605) ([tustvold](https://redirect.github.com/tustvold)) - Fix Azure URL parsing [#&#8203;604](https://redirect.github.com/apache/arrow-rs-object-store/pull/604) ([tustvold](https://redirect.github.com/tustvold)) - build(deps): update quick-xml requirement from 0.38.0 to 0.39.0 [#&#8203;602](https://redirect.github.com/apache/arrow-rs-object-store/pull/602) ([dependabot\[bot\]](https://redirect.github.com/apps/dependabot)) - Only read file metadata once in `LocalFileSystem::read_ranges` [#&#8203;595](https://redirect.github.com/apache/arrow-rs-object-store/pull/595) ([AdamGS](https://redirect.github.com/AdamGS)) - feat: Add support for AWS\_ENDPOINT\_URL\_S3 environment variable [#&#8203;590](https://redirect.github.com/apache/arrow-rs-object-store/pull/590) ([rajatgoel](https://redirect.github.com/rajatgoel)) - feat: impl MultipartStore for PrefixStore [#&#8203;587](https://redirect.github.com/apache/arrow-rs-object-store/pull/587) ([ddupg](https://redirect.github.com/ddupg)) - Implement typos-cli [#&#8203;570](https://redirect.github.com/apache/arrow-rs-object-store/pull/570) ([jayvdb](https://redirect.github.com/jayvdb)) - feat (azure): support for '.blob.core.windows.net' in "az://" scheme [#&#8203;431](https://redirect.github.com/apache/arrow-rs-object-store/pull/431) ([vladidobro](https://redirect.github.com/vladidobro)) \* *This Changelog was automatically generated by [github\_changelog\_generator](https://redirect.github.com/github-changelog-generator/github-changelog-generator)* </details> <details> <summary>prettier/prettier (prettier)</summary> ### [`v3.8.2`](https://redirect.github.com/prettier/prettier/compare/3.8.1...fbf300f9d89820364ddc9b2efa05b92b8c01b692) [Compare Source](https://redirect.github.com/prettier/prettier/compare/3.8.1...3.8.2) </details> <details> <summary>pyo3/pyo3 (pyo3)</summary> ### [`v0.28.3`](https://redirect.github.com/pyo3/pyo3/blob/HEAD/CHANGELOG.md#0283---2026-04-02) [Compare Source](https://redirect.github.com/pyo3/pyo3/compare/v0.28.2...v0.28.3) ##### Fixed - Fix compile error with `#[pyclass(get_all)]` on a type named `Probe`. [#&#8203;5837](https://redirect.github.com/PyO3/pyo3/pull/5837) - Fix compile error in debug builds related to `_Py_NegativeRefcount` with Python < 3.12. [#&#8203;5847](https://redirect.github.com/PyO3/pyo3/pull/5847) - Fix a race condition where `Python::attach` or `try_attach` could return before `site.py` had finished running. [#&#8203;5903](https://redirect.github.com/PyO3/pyo3/pull/5903) - Fix unsoundness in `PyBytesWriter::write_vectored` with Python 3.15 prerelease versions. [#&#8203;5907](https://redirect.github.com/PyO3/pyo3/pull/5907) - Fix deadlock in `.into_pyobject()` implementation for C-like `#[pyclass]` enums. [#&#8203;5928](https://redirect.github.com/PyO3/pyo3/pull/5928) </details> <details> <summary>rust-lang/rustc-hash (rustc-hash)</summary> ### [`v2.1.2`](https://redirect.github.com/rust-lang/rustc-hash/blob/HEAD/CHANGELOG.md#212) [Compare Source](https://redirect.github.com/rust-lang/rustc-hash/compare/v2.1.1...v2.1.2) - [Refactor byte hashing to remove unreachable panic](https://redirect.github.com/rust-lang/rustc-hash/pull/65) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMTAuMiIsInVwZGF0ZWRJblZlciI6IjQzLjExMC4yIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore[vortex-array]: add integer casting benchmark (#7400) The motivation is to compare an upcoming perf improvement. <!-- Thank you for submitting a pull request! We appreciate your time and effort. Please make sure to provide enough information so that we can review your pull request. The Summary and Testing sections below contain guidance on what to include. --> ## Summary <!-- If this PR is related to a tracked effort, please link to the relevant issue here (e.g., `Closes: #123`). Otherwise, feel free to ignore / delete this. In this section, please: 1. Explain the rationale for this change. 2. Summarize the changes included in this PR. A general rule of thumb is that larger PRs should have larger summaries. If there are a lot of changes, please help us review the code by explaining what was changed and why. If there is an issue or discussion attached, there is no need to duplicate all the details, but clarity is always preferred over brevity. --> <!-- ## API Changes Uncomment this section if there are any user-facing changes. Consider whether the change affects users in one of the following ways: 1. Breaks public APIs in some way. 2. Changes the underlying behavior of one of the engine integrations. 3. Should some documentation be updated to reflect this change? If a public API is changed in a breaking manner, make sure to add the appropriate label. You can run `./scripts/public-api.sh` locally to see if there are any public API changes (and this also runs in our CI). --> ## Testing <!-- Please describe how this change was tested. Here are some common categories for testing in Vortex: 1. Verifying existing behavior is maintained. 2. Verifying new behavior and functionality works correctly. 3. Serialization compatibility (backwards and forwards) should be maintained or explicitly broken. --> Signed-off-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com> Co-authored-by: Adam Gutglick <adam@spiraldb.com> * Ignore dead_code warnings on scalar bit transpose code (#7408) Likely we shouldn't use expect here but we also don't test on architectures where this wouldn't be dead code Signed-off-by: Robert Kruszewski <github@robertk.io> * support multiple globs in DuckDB and MultiFileDataSource (#7390) ## Summary Adds support for multiple globs in DuckDB, similar to Parquet, E.g. instead of `read_vortex('*.vortex')` now you can also do `read_vortex(['s3://bucket1/path/*.vortex', '/a/b/*.vortex'])` Note that multiple file systems are supported in a single scan. This is implemented and wired through into the underlying Rust MultiFileDataSource. ## Testing We add new DuckDB e2e tests for multi-glob Couldn't figure out a great way to test multiple file systems, just multiple globs --------- Signed-off-by: Andrew Duffy <andrew@a10y.dev> * Optimize math expressions with `Constant` children (#7394) ## Summary Tracking issue: https://github.com/vortex-data/vortex/issues/7297 Adds some constant array optimizations to the tensor crate expressions `L2Norm`, `L2Denorm`, and `CosineSimilarity`. The remaining expressions `InnerProduct` and `SorfTransform` are a bit more complicated and deserve their own PR. ## Testing Adds more tests for these optimizations. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> * TurboQuant `InnerProduct` optimizations (#7396) _Stacked on top of https://github.com/vortex-data/vortex/pull/7394._ ## Summary Tracking issue: https://github.com/vortex-data/vortex/issues/7297 Implements the final pushdown / reduction rules needed to make the TurboQuant quantization scheme make sense. TODO ## Testing More tests --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com> * duckdb: Exit earlier on Validity::AllFalse (#7411) 1. Make duckdb all-invalid exporter a unit type: If we set underlying vector's validity to "all false", duckdb won't read underlying values so you don't need to fill them 2. Exit early on exporter branches if validity is all false. Saves a ConstantArray creation + execute Signed-off-by: Mikhail Kot <to@myrrc.dev> * perf: `Arc::ptr_eq` short-circuit for List & FixedSizeList DType eq (#7410) Extend the `Arc::ptr_eq` fast-path from #7398 to cover the remaining Arc-containing DType variants. List and FixedSizeList hold a bare `Arc<DType>` in the enum variant, so the shortcut is applied in `DType`'s manual `PartialEq` impl. `StructFields` already handles its own `Arc::ptr_eq` internally. The mismatch arms enumerate every variant in the first position so that adding a new DType variant produces a non-exhaustive match compile error. DuckDB StatPopGen full-suite [apmc](https://github.com/0ax1/apmc) measurement for vortex, averaged over two runs: - Cycles: -5.4% (5.97B → 5.65B) - Instructions: -15.3% (15.5B → 13.1B) - L1D_CACHE_MISS_LD: -0.8% (56.2M → 55.7M) - MAP_STALL: -0.2% (1.35B → 1.34B) Signed-off-by: Alexander Droste <alexander.droste@protonmail.com> * feat: implement IsNotNull expression in vortex expression library (#6969) ### Summary Closes: #6040 Add a first-class IsNotNull scalar function, replacing the previous Not(IsNull(...)) composition pattern. This simplifies the expression tree and enables direct stat_falsification for zone map pruning. Changes: New is_not_null.rs with ScalarFnVTable implementation, including stat_falsification using is_constant && null_count > 0 (with TODO for future RowCount stat) Updated all integration points: DataFusion, DuckDB, Python/Substrait to use is_not_null(...) directly Replaced the Not(IsNull(...)) fallback in erased.rs validity with IsNotNull Registered IsNotNull in ScalarFnSession and ExprBuiltins/ArrayBuiltins ### AI Assistance Disclosure This PR was developed with AI assistance (Kiro). AI was used for code review, implementing stat_falsification, writing tests, and drafting the PR description. All output was reviewed and validated by the author. API Changes New public APIs: vortex_array::expr::is_not_null(child) — creates an IsNotNull expression Expression::is_not_null() / ArrayRef::is_not_null() via ExprBuiltins/ArrayBuiltins traits Python: vortex._lib.expr.is_not_null(child) ### Testing 9 unit tests covering: return dtype, child replacement, mixed/all-valid/all-invalid evaluation, struct field access, display formatting, null sensitivity, and stat falsification pruning expression generation. --------- Signed-off-by: Xiaoxuan Li <xioxuan@amazon.com> Signed-off-by: Robert Kruszewski <github@robertk.io> Co-authored-by: Robert Kruszewski <github@robertk.io> * More robust types in the compressor (#7415) ## Summary Tracking issue: https://github.com/vortex-data/vortex/issues/7216 Makes the compressor types more robust (removes the possibility for invalid state), which additionally sets up adding tracing easier (draft at https://github.com/vortex-data/vortex/pull/7385) ## API Changes Changes some types: ```rust /// Closure type for [`DeferredEstimate::Callback`]. /// /// The compressor calls this with the same arguments it would pass to sampling. The closure must /// resolve directly to a terminal [`EstimateVerdict`]. #[rustfmt::skip] pub type EstimateFn = dyn FnOnce( &CascadingCompressor, &mut ArrayAndStats, CompressorContext, ) -> VortexResult<EstimateVerdict> + Send + Sync; /// The result of a [`Scheme`]'s compression ratio estimation. /// /// This type is returned by [`Scheme::expected_compression_ratio`] to tell the compressor how /// promising this scheme is for a given array without performing any expensive work. /// /// [`CompressionEstimate::Verdict`] means the scheme already knows the terminal answer. /// [`CompressionEstimate::Deferred`] means the compressor must do extra work before the scheme can /// produce a terminal answer. #[derive(Debug)] pub enum CompressionEstimate { /// The scheme already knows the terminal estimation verdict. Verdict(EstimateVerdict), /// The compressor must perform deferred work to resolve the terminal estimation verdict. Deferred(DeferredEstimate), } /// The terminal answer to a compression estimate request. #[derive(Debug)] pub enum EstimateVerdict { /// Do not use this scheme for this array. Skip, /// Always use this scheme, as it is definitively the best choice. /// /// Some examples include constant detection, decimal byte parts, and temporal decomposition. /// /// The compressor will select this scheme immediately without evaluating further candidates. /// Schemes that return `AlwaysUse` must be mutually exclusive per canonical type (enforced by /// [`Scheme::matches`]), otherwise the winner depends silently on registration order. /// /// [`Scheme::matches`]: crate::scheme::Scheme::matches AlwaysUse, /// The estimated compression ratio. This must be greater than `1.0` to be considered by the /// compressor, otherwise it is worse than the canonical encoding. Ratio(f64), } /// Deferred work that can resolve to a terminal [`EstimateVerdict`]. pub enum DeferredEstimate { /// The scheme cannot cheaply estimate its ratio, so the compressor should compress a small /// sample to determine effectiveness. Sample, /// A fallible estimation requiring a custom expensive computation. /// /// Use this only when the scheme needs to perform trial encoding or other costly checks to /// determine its compression ratio. The callback returns an [`EstimateVerdict`] directly, so /// it cannot request more sampling or another deferred callback. Callback(Box<EstimateFn>), } ``` This will make some changes that we want to make is the future easier as well (tracing, better decision making for what things to try, etc). ## Testing Some new tests Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> * fix[ci]: bench no fail-fast random access (#7418) <!-- Thank you for submitting a pull request! We appreciate your time and effort. Please make sure to provide enough information so that we can review your pull request. The Summary and Testing sections below contain guidance on what to include. --> ## Summary <!-- If this PR is related to a tracked effort, please link to the relevant issue here (e.g., `Closes: #123`). Otherwise, feel free to ignore / delete this. In this section, please: 1. Explain the rationale for this change. 2. Summarize the changes included in this PR. A general rule of thumb is that larger PRs should have larger summaries. If there are a lot of changes, please help us review the code by explaining what was changed and why. If there is an issue or discussion attached, there is no need to duplicate all the details, but clarity is always preferred over brevity. --> Closes: #000 <!-- ## API Changes Uncomment this section if there are any user-facing changes. Consider whether the change affects users in one of the following ways: 1. Breaks public APIs in some way. 2. Changes the underlying behavior of one of the engine integrations. 3. Should some documentation be updated to reflect this change? If a public API is changed in a breaking manner, make sure to add the appropriate label. You can run `./scripts/public-api.sh` locally to see if there are any public API changes (and this also runs in our CI). --> ## Testing <!-- Please describe how this change was tested. Here are some common categories for testing in Vortex: 1. Verifying existing behavior is maintained. 2. Verifying new behavior and functionality works correctly. 3. Serialization compatibility (backwards and forwards) should be maintained or explicitly broken. --> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> * Remove getrandom feature flag overrides (#7421) These have been required at some point in our codebase to get getrandom v0.2 and v0.3 to coexsit in wasm setup. However, the world has moved from getrandom v0.2 and override situation is not as complicated anymore. It seems we can wholesale remove these overrides that we kept around for way too long. fix #7382 Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Robert Kruszewski <github@robertk.io> * perf[vortex-array]: use from_trusted_len_iter in primitive casts (#7401) Some of our scan profiles show 10% of scan cpu time is spent in integer widening casts (nullable dictionary codes). This commit simplifies and optimizes primitive casts by hoisting a lot of hot loop branching logic. Specifically, this commit relies on values_fit_in to verify representability so that we can avoid a potential validity and error check in the hot loop. Additionally from_trusted_len_iter lets the destination BufferMut optimize the actual cast instead of using push_unchecked for each element. Results locally running the benchmark from #7400. Before: ``` $ cargo bench -p vortex-array --bench cast_primitive Finished `bench` profile [optimized + debuginfo] target(s) in 0.16s Running benches/cast_primitive.rs (target/release/deps/cast_primitive-598823f32b8f3db0) Timer precision: 41 ns cast_primitive fastest │ slowest │ median │ mean │ samples │ iters ╰─ cast_u16_to_u32 384.2 µs │ 491.6 µs │ 395.1 µs │ 397.3 µs │ 100 │ 100 ``` After: ``` cargo bench -p vortex-array --bench cast_primitive Finished `bench` profile [optimized + debuginfo] target(s) in 0.17s Running benches/cast_primitive.rs (target/release/deps/cast_primitive-598823f32b8f3db0) Timer precision: 41 ns cast_primitive fastest │ slowest │ median │ mean │ samples │ iters ╰─ cast_u16_to_u32 6.874 µs │ 543.7 µs │ 6.999 µs │ 12.7 µs │ 100 │ 100 ``` Signed-off-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com> * Update dependency pytest to v9 [SECURITY] (#7419) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [pytest](https://redirect.github.com/pytest-dev/pytest) ([changelog](https://docs.pytest.org/en/stable/changelog.html)) | `8.4.2` → `9.0.3` | ![age](https://developer.mend.io/api/mc/badges/age/pypi/pytest/9.0.3?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/pytest/8.4.2/9.0.3?slim=true) | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/357) for more information. ### GitHub Vulnerability Alerts #### [CVE-2025-71176](https://nvd.nist.gov/vuln/detail/CVE-2025-71176) pytest through 9.0.2 on UNIX relies on directories with the `/tmp/pytest-of-{user}` name pattern, which allows local users to cause a denial of service or possibly gain privileges. --- ### Release Notes <details> <summary>pytest-dev/pytest (pytest)</summary> ### [`v9.0.3`](https://redirect.github.com/pytest-dev/pytest/releases/tag/9.0.3) [Compare Source](https://redirect.github.com/pytest-dev/pytest/compare/9.0.2...9.0.3) ### pytest 9.0.3 (2026-04-07) #### Bug fixes - [#&#8203;12444](https://redirect.github.com/pytest-dev/pytest/issues/12444): Fixed `pytest.approx` which now correctly takes into account `~collections.abc.Mapping` keys order to compare them. - [#&#8203;13634](https://redirect.github.com/pytest-dev/pytest/issues/13634): Blocking a `conftest.py` file using the `-p no:` option is now explicitly disallowed. Previously this resulted in an internal assertion failure during plugin loading. Pytest now raises a clear `UsageError` explaining that conftest files are not plugins and cannot be disabled via `-p`. - [#&#8203;13734](https://redirect.github.com/pytest-dev/pytest/issues/13734): Fixed crash when a test raises an exceptiongroup with `__tracebackhide__ = True`. - [#&#8203;14195](https://redirect.github.com/pytest-dev/pytest/issues/14195): Fixed an issue where non-string messages passed to <span class="title-ref">unittest.TestCase.subTest()</span> were not printed. - [#&#8203;14343](https://redirect.github.com/pytest-dev/pytest/issues/14343): Fixed use of insecure temporary directory (CVE-2025-71176). #### Improved documentation - [#&#8203;13388](https://redirect.github.com/pytest-dev/pytest/issues/13388): Clarified documentation for `-p` vs `PYTEST_PLUGINS` plugin loading and fixed an incorrect `-p` example. - [#&#8203;13731](https://redirect.github.com/pytest-dev/pytest/issues/13731): Clarified that capture fixtures (e.g. `capsys` and `capfd`) take precedence over the `-s` / `--capture=no` command-line options in `Accessing captured output from a test function <accessing-captured-output>`. - [#&#8203;14088](https://redirect.github.com/pytest-dev/pytest/issues/14088): Clarified that the default `pytest_collection` hook sets `session.items` before it calls `pytest_collection_finish`, not after. - [#&#8203;14255](https://redirect.github.com/pytest-dev/pytest/issues/14255): TOML integer log levels must be quoted: Updating reference documentation. #### Contributor-facing changes - [#&#8203;12689](https://redirect.github.com/pytest-dev/pytest/issues/12689): The test reports are now published to Codecov from GitHub Actions. The test statistics is visible [on the web interface](https://app.codecov.io/gh/pytest-dev/pytest/tests). \-- by `aleguy02` ### [`v9.0.2`](https://redirect.github.com/pytest-dev/pytest/releases/tag/9.0.2) [Compare Source](https://redirect.github.com/pytest-dev/pytest/compare/9.0.1...9.0.2) ### pytest 9.0.2 (2025-12-06) #### Bug fixes - [#&#8203;13896](https://redirect.github.com/pytest-dev/pytest/issues/13896): The terminal progress feature added in pytest 9.0.0 has been disabled by default, except on Windows, due to compatibility issues with some terminal emulators. You may enable it again by passing `-p terminalprogress`. We may enable it by default again once compatibility improves in the future. Additionally, when the environment variable `TERM` is `dumb`, the escape codes are no longer emitted, even if the plugin is enabled. - [#&#8203;13904](https://redirect.github.com/pytest-dev/pytest/issues/13904): Fixed the TOML type of the `tmp_path_retention_count` settings in the API reference from number to string. - [#&#8203;13946](https://redirect.github.com/pytest-dev/pytest/issues/13946): The private `config.inicfg` attribute was changed in a breaking manner in pytest 9.0.0. Due to its usage in the ecosystem, it is now restored to working order using a compatibility shim. It will be deprecated in pytest 9.1 and removed in pytest 10. - [#&#8203;13965](https://redirect.github.com/pytest-dev/pytest/issues/13965): Fixed quadratic-time behavior when handling `unittest` subtests in Python 3.10. #### Improved documentation - [#&#8203;4492](https://redirect.github.com/pytest-dev/pytest/issues/4492): The API Reference now contains cross-reference-able documentation of `pytest's command-line flags <command-line-flags>`. ### [`v9.0.1`](https://redirect.github.com/pytest-dev/pytest/releases/tag/9.0.1) [Compare Source](https://redirect.github.com/pytest-dev/pytest/compare/9.0.0...9.0.1) ### pytest 9.0.1 (2025-11-12) #### Bug fixes - [#&#8203;13895](https://redirect.github.com/pytest-dev/pytest/issues/13895): Restore support for skipping tests via `raise unittest.SkipTest`. - [#&#8203;13896](https://redirect.github.com/pytest-dev/pytest/issues/13896): The terminal progress plugin added in pytest 9.0 is now automatically disabled when iTerm2 is detected, it generated desktop notifications instead of the desired functionality. - [#&#8203;13904](https://redirect.github.com/pytest-dev/pytest/issues/13904): Fixed the TOML type of the verbosity settings in the API reference from number to string. - [#&#8203;13910](https://redirect.github.com/pytest-dev/pytest/issues/13910): Fixed <span class="title-ref">UserWarning: Do not expect file\_or\_dir</span> on some earlier Python 3.12 and 3.13 point versions. #### Packaging updates and notes for downstreams - [#&#8203;13933](https://redirect.github.com/pytest-dev/pytest/issues/13933): The tox configuration has been adjusted to make sure the desired version string can be passed into its `package_env` through the `SETUPTOOLS_SCM_PRETEND_VERSION_FOR_PYTEST` environment variable as a part of the release process -- by `webknjaz`. #### Contributor-facing changes - [#&#8203;13891](https://redirect.github.com/pytest-dev/pytest/issues/13891), [#&#8203;13942](https://redirect.github.com/pytest-dev/pytest/issues/13942): The CI/CD part of the release automation is now capable of creating GitHub Releases without hav…
1 parent c765c90 commit 86cc337

3 files changed

Lines changed: 153 additions & 9 deletions

File tree

  • vortex-array/src
  • vortex-datafusion/src/persistent

vortex-array/src/arrays/decimal/compute/cast.rs

Lines changed: 147 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,27 @@
22
// SPDX-FileCopyrightText: Copyright the Vortex contributors
33

44
use vortex_buffer::Buffer;
5+
use vortex_buffer::BufferMut;
56
use vortex_error::VortexExpect;
67
use vortex_error::VortexResult;
78
use vortex_error::vortex_bail;
9+
use vortex_error::vortex_err;
810
use vortex_error::vortex_panic;
11+
use vortex_mask::AllOr;
12+
use vortex_mask::Mask;
913

1014
use crate::ArrayRef;
1115
use crate::ExecutionCtx;
16+
use crate::IntoArray;
1217
use crate::arrays::DecimalArray;
1318
use crate::arrays::DecimalVTable;
19+
use crate::arrays::primitive::PrimitiveArray;
1420
use crate::dtype::DType;
1521
use crate::dtype::DecimalType;
1622
use crate::dtype::NativeDecimalType;
23+
use crate::dtype::NativePType;
1724
use crate::match_each_decimal_value_type;
25+
use crate::match_each_native_ptype;
1826
use crate::scalar_fn::fns::cast::CastKernel;
1927
use crate::vtable::ValidityHelper;
2028

@@ -24,17 +32,39 @@ impl CastKernel for DecimalVTable {
2432
dtype: &DType,
2533
_ctx: &mut ExecutionCtx,
2634
) -> VortexResult<Option<ArrayRef>> {
27-
// Early return if not casting to decimal
28-
let DType::Decimal(to_decimal_dtype, to_nullability) = dtype else {
29-
return Ok(None);
30-
};
3135
let DType::Decimal(from_decimal_dtype, _) = array.dtype() else {
3236
vortex_panic!(
3337
"DecimalArray must have decimal dtype, got {:?}",
3438
array.dtype()
3539
);
3640
};
3741

42+
if let DType::Primitive(to_ptype, to_nullability) = dtype {
43+
let new_validity = array
44+
.validity()
45+
.clone()
46+
.cast_nullability(*to_nullability, array.len())?;
47+
let mask = new_validity.to_mask(array.len());
48+
49+
return Ok(Some(match_each_native_ptype!(*to_ptype, |T| {
50+
match_each_decimal_value_type!(array.values_type(), |F| {
51+
PrimitiveArray::new(
52+
cast_decimal_buffer_to_primitive::<F, T>(
53+
array.buffer::<F>(),
54+
from_decimal_dtype.scale(),
55+
mask,
56+
)?,
57+
new_validity,
58+
)
59+
.into_array()
60+
})
61+
})));
62+
}
63+
64+
let DType::Decimal(to_decimal_dtype, to_nullability) = dtype else {
65+
return Ok(None);
66+
};
67+
3868
// Scale changes are not yet supported
3969
if from_decimal_dtype.scale() != to_decimal_dtype.scale() {
4070
vortex_bail!(
@@ -139,6 +169,57 @@ fn upcast_decimal_buffer<F: NativeDecimalType, T: NativeDecimalType>(from: Buffe
139169
.collect()
140170
}
141171

172+
fn cast_decimal_buffer_to_primitive<F, T>(
173+
from: Buffer<F>,
174+
scale: i8,
175+
mask: Mask,
176+
) -> VortexResult<Buffer<T>>
177+
where
178+
F: NativeDecimalType,
179+
T: NativePType,
180+
{
181+
let scale_factor = 10_f64.powi(i32::from(scale));
182+
183+
match mask.bit_buffer() {
184+
AllOr::All => {
185+
let mut buffer = BufferMut::<T>::with_capacity(from.len());
186+
for value in from {
187+
let value = cast_decimal_value_to_primitive::<F, T>(value, scale_factor)?;
188+
buffer.push(value);
189+
}
190+
Ok(buffer.freeze())
191+
}
192+
AllOr::None => Ok(Buffer::zeroed(from.len())),
193+
AllOr::Some(validity) => {
194+
let mut buffer = BufferMut::<T>::with_capacity(from.len());
195+
for (value, valid) in from.iter().zip(validity.iter()) {
196+
if valid {
197+
let value = cast_decimal_value_to_primitive::<F, T>(*value, scale_factor)?;
198+
buffer.push(value);
199+
} else {
200+
buffer.push(T::default());
201+
}
202+
}
203+
Ok(buffer.freeze())
204+
}
205+
}
206+
}
207+
208+
fn cast_decimal_value_to_primitive<F, T>(value: F, scale_factor: f64) -> VortexResult<T>
209+
where
210+
F: NativeDecimalType,
211+
T: NativePType,
212+
{
213+
let value = value
214+
.to_f64()
215+
.ok_or_else(|| vortex_err!(Compute: "Failed to cast decimal value {value} to f64"))?
216+
/ scale_factor;
217+
218+
T::from(value).ok_or_else(
219+
|| vortex_err!(Compute: "Failed to cast decimal value {value} to {:?}", T::PTYPE),
220+
)
221+
}
222+
142223
#[cfg(test)]
143224
mod tests {
144225
use rstest::rstest;
@@ -154,6 +235,7 @@ mod tests {
154235
use crate::dtype::DecimalDType;
155236
use crate::dtype::DecimalType;
156237
use crate::dtype::Nullability;
238+
use crate::dtype::PType;
157239
use crate::validity::Validity;
158240
use crate::vtable::ValidityHelper;
159241

@@ -282,6 +364,67 @@ mod tests {
282364
assert_eq!(casted.values_type(), DecimalType::I128);
283365
}
284366

367+
#[test]
368+
fn cast_decimal_to_f64_applies_scale() {
369+
let array = DecimalArray::new(
370+
buffer![12345i64, -50, 0],
371+
DecimalDType::new(15, 2),
372+
Validity::NonNullable,
373+
);
374+
let dtype = DType::Primitive(PType::F64, Nullability::NonNullable);
375+
376+
let casted = array.to_array().cast(dtype.clone()).unwrap().to_primitive();
377+
378+
assert_eq!(casted.dtype(), &dtype);
379+
assert_eq!(casted.validity(), &Validity::NonNullable);
380+
let values = casted.as_slice::<f64>();
381+
assert!((values[0] - 123.45).abs() < 0.000000000001);
382+
assert_eq!(values[1], -0.5);
383+
assert_eq!(values[2], 0.0);
384+
}
385+
386+
#[test]
387+
fn cast_nullable_decimal_to_nullable_f64_preserves_validity() {
388+
let array = DecimalArray::from_option_iter(
389+
[Some(12345i64), None, Some(-50)],
390+
DecimalDType::new(15, 2),
391+
);
392+
let dtype = DType::Primitive(PType::F64, Nullability::Nullable);
393+
394+
let casted = array.to_array().cast(dtype.clone()).unwrap().to_primitive();
395+
396+
assert_eq!(casted.dtype(), &dtype);
397+
let mask = casted.validity_mask().unwrap();
398+
assert!(mask.value(0));
399+
assert!(!mask.value(1));
400+
assert!(mask.value(2));
401+
let values = casted.as_slice::<f64>();
402+
assert!((values[0] - 123.45).abs() < 0.000000000001);
403+
assert_eq!(values[2], -0.5);
404+
}
405+
406+
#[test]
407+
fn cast_nullable_decimal_to_non_nullable_f64_fails() {
408+
let array = DecimalArray::from_option_iter(
409+
[Some(12345i64), None, Some(-50)],
410+
DecimalDType::new(15, 2),
411+
);
412+
let dtype = DType::Primitive(PType::F64, Nullability::NonNullable);
413+
414+
let result = array
415+
.to_array()
416+
.cast(dtype)
417+
.and_then(|a| a.to_canonical().map(|c| c.into_array()));
418+
419+
assert!(result.is_err());
420+
assert!(
421+
result
422+
.unwrap_err()
423+
.to_string()
424+
.contains("Cannot cast array with invalid values to non-nullable type")
425+
);
426+
}
427+
285428
#[test]
286429
fn cast_to_non_decimal_returns_err() {
287430
let array = DecimalArray::new(

vortex-array/src/scalar_fn/fns/list_contains/mod.rs

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -280,9 +280,10 @@ fn or_arrays_balanced(mut arrays: Vec<ArrayRef>) -> VortexResult<ArrayRef> {
280280
arrays = next;
281281
}
282282

283-
Ok(arrays
284-
.pop()
285-
.expect("or_arrays_balanced must be called with at least one array"))
283+
let Some(array) = arrays.pop() else {
284+
vortex_bail!("or_arrays_balanced must be called with at least one array");
285+
};
286+
Ok(array)
286287
}
287288

288289
/// Returns a [`BoolArray`] where each bit represents if a list contains the scalar.
@@ -452,15 +453,14 @@ fn list_is_not_empty(
452453
mod tests {
453454
use std::sync::Arc;
454455

455-
use super::or_arrays_balanced;
456-
457456
use itertools::Itertools;
458457
use rstest::rstest;
459458
use vortex_buffer::BitBuffer;
460459
use vortex_buffer::Buffer;
461460
use vortex_utils::aliases::hash_map::HashMap;
462461
use vortex_utils::aliases::hash_set::HashSet;
463462

463+
use super::or_arrays_balanced;
464464
use crate::Array;
465465
use crate::ArrayRef;
466466
use crate::IntoArray;

vortex-datafusion/src/persistent/opener.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -538,6 +538,7 @@ mod tests {
538538
use object_store::memory::InMemory;
539539
use rstest::rstest;
540540
use vortex::VortexSessionDefault;
541+
use vortex::array::ArrayRef;
541542
use vortex::array::arrow::FromArrowArray;
542543
use vortex::buffer::Buffer;
543544
use vortex::file::WriteOptionsSessionExt;

0 commit comments

Comments
 (0)