Upgrade spiceai-52 to Vortex 0.68.0

[lukekim]

Summary

• sync spiceai-52 to upstream Vortex 0.68.0 (ae906c7bfebb830f73401f3d5f8fcdeb9c5fc033)
• preserve SpiceAI-specific DF52 compatibility and custom date->timestamp extension casting during the merge
• regenerate public API lockfiles after the sync

Validation

• RUSTC_WRAPPER= cargo test --no-run -p vortex-array -p vortex-datafusion
• RUSTC_WRAPPER= cargo test -p vortex-array cast_date_days_to_timestamp -- --nocapture
• RUSTC_WRAPPER= cargo test -p vortex-array test_map_array_conversion -- --nocapture
• RUSTC_WRAPPER= cargo test -p vortex-datafusion case_when -- --nocapture
• cargo +nightly fmt --all
• ./scripts/public-api.sh
• RUSTC_WRAPPER= cargo clippy --all-targets --all-features

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Syncs the repo to upstream Vortex 0.68.0, updating encoding kernels to the newer ArrayView/session-based execution APIs while also modernizing benchmark tooling, docs, and CI automation around fuzzing/benchmarks/web assets.

Changes:

• Refactors multiple encoding compute kernels (datetime-parts, bytebool, ALP/ALPRD) to new kernel/adaptor traits and execution contexts.
• Updates benchmarks and benchmark website tooling, plus documentation and contribution/agent guidance.
• Reworks CI workflows for fuzzing/benchmarks/web and introduces compatibility fixture generation/validation workflows.

Reviewed changes

Copilot reviewed 134 out of 1795 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
encodings/datetime-parts/src/compute/mod.rs	Exposes is_constant module and updates tests to new array construction/execution APIs
encodings/datetime-parts/src/compute/mask.rs	Ports masking kernel to ArrayView and new DateTimeParts API
encodings/datetime-parts/src/compute/kernel.rs	Updates parent kernel adaptors (Take/Compare) for new module paths/types
encodings/datetime-parts/src/compute/is_constant.rs	Replaces old kernel registration with dyn aggregate kernel implementation
encodings/datetime-parts/src/compute/filter.rs	Ports filter reduce implementation to ArrayView-based API
encodings/datetime-parts/src/compute/compare.rs	Ports compare kernel signatures to ArrayView and ArrayRef
encodings/datetime-parts/src/compute/cast.rs	Ports cast reduce to ArrayView and updates conformance tests
encodings/datetime-parts/src/compress.rs	Adjusts temporal split validity handling and refactors TryFrom for new data type
encodings/datetime-parts/src/canonical.rs	Updates decode pipeline to new buffer/validity/execution APIs and tests
encodings/bytebool/src/slice.rs	Ports slice reduce to ArrayView and new validity API
encodings/bytebool/src/rules.rs	Updates rule adaptors and encoding type
encodings/bytebool/src/kernel.rs	Updates take adaptor path and parent kernel set type
encodings/bytebool/Cargo.toml	Adds vortex-mask dependency
encodings/alp/src/lib.rs	Adds session initialization helper for registering encodings + ALP NaN-count kernel
encodings/alp/src/alp_rd/slice.rs	Ports slice kernel to ArrayView type and new constructor API
encodings/alp/src/alp_rd/rules.rs	Updates optimizer rules to new encoding type
encodings/alp/src/alp_rd/ops.rs	Updates scalar access vtable to ArrayView and dtype-based float selection
encodings/alp/src/alp_rd/mod.rs	Refactors RD decode pipeline; adds fast-path combining and adjusts patches ownership
encodings/alp/src/alp_rd/kernel.rs	Updates parent kernels to new adaptor module paths/types
encodings/alp/src/alp_rd/compute/take.rs	Ports take execution to ArrayView, ArrayRef, and ctx-aware patches
encodings/alp/src/alp_rd/compute/mod.rs	Updates consistency tests to use into_array()
encodings/alp/src/alp_rd/compute/mask.rs	Ports mask reduce to new scalar-fn factory API and patches ownership
encodings/alp/src/alp_rd/compute/filter.rs	Ports filter kernel to ArrayView and ctx-aware patches filtering
encodings/alp/src/alp_rd/compute/cast.rs	Ports cast reduce to ArrayView and ctx-aware patches handling
encodings/alp/src/alp/rules.rs	Updates parent kernels/rules to new adaptor paths and encoding type
encodings/alp/src/alp/ops.rs	Ports scalar access vtable to ArrayView and dtype-based ptype selection
encodings/alp/src/alp/mod.rs	Exposes compute module and updates protobuf metadata test serialization
encodings/alp/src/alp/decompress.rs	Requires ExecutionCtx for execution/patching and adjusts chunk helpers
encodings/alp/src/alp/compute/take.rs	Ports take execution to ArrayView and ctx-aware patches
encodings/alp/src/alp/compute/slice.rs	Ports slice kernel to ArrayView
encodings/alp/src/alp/compute/nan_count.rs	Replaces old kernel registration with dyn aggregate NaN-count kernel
encodings/alp/src/alp/compute/mod.rs	Exposes nan_count module for registration and updates tests
encodings/alp/src/alp/compute/mask.rs	Ports mask reduce/kernel to ctx-aware validity masking + patch value casting
encodings/alp/src/alp/compute/filter.rs	Ports filter kernel to ArrayView and ctx-aware patch filtering
encodings/alp/src/alp/compute/compare.rs	Ports compare kernel to ArrayView and ArrayRef rhs
encodings/alp/src/alp/compute/cast.rs	Ports cast reduce to ArrayView and simplifies unchecked ctor usage
encodings/alp/src/alp/compute/between.rs	Ports between reduce to ArrayView and ArrayRef bounds
encodings/alp/src/alp/compress.rs	Updates validity API usage and returns new encoding type
encodings/alp/benches/alp_compress.rs	Updates benchmark setup for new RNG API + ctx-required decompress
docs/pyproject.toml	Updates Sphinx minimum version
docs/project/contributing.md	Expands AI assistance policy and repo agent expectations
docs/developer-guide/internals/session.md	Updates session registration examples
docs/developer-guide/internals/architecture.md	Fixes table formatting and updates Spark module path naming
docs/developer-guide/benchmarking.md	Major expansion of benchmarking best practices and references
docs/api/python/store.rst	Replaces monolithic module doc with toctree pages + from_url function
docs/api/python/store/aws.rst	Adds S3 API docs page
docs/api/python/store/azure.rst	Adds Azure API docs page
docs/api/python/store/config.rst	Adds common config API docs page
docs/api/python/store/gcs.rst	Adds GCS API docs page
docs/api/python/store/http.rst	Adds HTTP store API docs page
docs/api/python/store/local.rst	Adds local store API docs page
docs/api/python/store/memory.rst	Adds memory store API docs page
docs/api/python/arrays.rst	Updates renamed Python API type (SerializedArray)
docs/api/cpp/index.rst	Updates documented minimum C++ standard
docs/api/c/dtypes.rst	Removes function docs entry for clone helper
clippy.toml	Disallows Result::and due to footgun risk
benchmarks/random-access-bench/src/main.rs	Updates RNG API and benchmark output writer creation
benchmarks/lance-bench/src/main.rs	Switches SQL runner API to BenchmarkMode + result wrapper type
benchmarks/lance-bench/src/convert.rs	Uses Arc::clone for schemas/columns
benchmarks/lance-bench/src/compress.rs	Uses Arc::clone for Parquet schema
benchmarks/duckdb-bench/src/main.rs	Switches to BenchmarkMode instead of early-return explain path
benchmarks/duckdb-bench/src/lib.rs	Adds BenchmarkQueryResult wrapper for DuckDB results
benchmarks/datafusion-bench/src/main.rs	Switches to BenchmarkMode + result wrapper; refactors object store usage
benchmarks/datafusion-bench/src/lib.rs	Uses explicit Arc::clone when registering object stores
benchmarks/compress-bench/src/vortex.rs	Updates array stream conversion to into_array()
benchmarks/compress-bench/src/parquet.rs	Uses Arc::clone for schema
benchmarks/compress-bench/src/main.rs	Updates benchmark output writer creation
benchmarks/compress-bench/src/lib.rs	Updates chunked array iteration APIs and RecordBatch conversion
benchmarks-website/src/styles/index.css	Adds dark-mode theme variables and UI color tweaks
benchmarks-website/src/components/Sidebar.jsx	Updates benchmark data download URL
benchmarks-website/src/components/Header.jsx	Adds light/dark theme logo switching via <picture>
benchmarks-website/src/components/ChartContainer.jsx	Refactors imports and adjusts chart typography/tooltip styling
benchmarks-website/server.js	Updates data/commits URLs and normalizes one chart label
benchmarks-website/package.json	Bumps Vite major version
benchmarks-website/config.js	Removed legacy config module (migrated elsewhere)
benchmarks-website/utils.js	Removed legacy utils module (migrated elsewhere)
benchmarks-website/zoom-sync.js	Removed legacy zoom sync module (migrated elsewhere)
benchmarks-website/worker-manager.js	Removed legacy worker manager module (migrated elsewhere)
benchmarks-website/data-worker.js	Removed legacy data worker (migrated elsewhere)
benchmarks-website/data-shared.js	Removed legacy shared grouping logic (migrated elsewhere)
benchmarks-website/data-processor.js	Removed legacy processor wrapper (migrated elsewhere)
benchmarks-website-v2/index.html	Removes legacy v2 site entrypoint
benchmarks-website-v2/public/site.webmanifest	Removes legacy v2 webmanifest
_typos.toml	Expands ignore lists and excludes additional paths
REUSE.toml	Updates licensing annotations and includes JSON under Apache-2.0 override
CONTRIBUTING.md	Aligns top-level AI policy with expanded docs
CLAUDE.md	Renames/repoints to AGENTS.md
AGENTS.md	Adds consolidated agent/development guidelines document
.github/workflows/web.yml	Adds web build/lint/deploy workflow for vortex-web
.github/workflows/wasm-fuzz.yml	Updates artifact upload action version
.github/workflows/typos.yml	Updates typos action version
.github/workflows/sql-pr.yml	Converts PR SQL benchmarks to workflow_call entrypoint
.github/workflows/run-fuzzer.yml	Refactors fuzz runner inputs, corpus naming, fork-mode repro, and runner selection
.github/workflows/report-fuzz-crash.yml	Adds FUZZ_NAME plumbing and updates artifact download versions
.github/workflows/release-drafter.yml	Updates release-drafter action version and simplifies token usage
.github/workflows/publish.yml	Updates artifact download versions and adds compat upload workflow call
.github/workflows/publish-benchmarks-website.yml	Retargets publish to benchmarks-website/** and updates docker actions
.github/workflows/package.yml	Updates artifact upload versions
.github/workflows/nightly-bench.yml	Updates benchmark matrix and dataset bucket for S3 runs
.github/workflows/minimize_fuzz_corpus_workflow.yml	Updates runner selection and cargo-fuzz installer revision
.github/workflows/fuzzer-fix-automation.yml	Updates cargo-fuzz installer revision
.github/workflows/fuzz.yml	Increases schedule frequency and adds additional fuzz jobs + fork parallelism
.github/workflows/fuzz-coverage.yml	Updates runner selection and artifact upload version
.github/workflows/docs.yml	Updates Doxygen action and Spark javadoc path; updates artifact download version
.github/workflows/compat-validation.yml	Adds scheduled compat validation workflow
.github/workflows/compat-gen-upload.yml	Adds compat fixture gen/upload workflow with dry-run gating
.github/workflows/close-fixed-fuzzer-issues.yml	Runs nightly and updates runner selection + cargo-fuzz installer revision
.github/workflows/bench.yml	Updates AWS roles/buckets, runner selection, and bench invocation wrapper
.github/workflows/bench-pr.yml	Converts to workflow_call and updates runner selection + bench invocation wrapper
.github/workflows/bench-dispatch.yml	Adds label-driven dispatcher to avoid phantom check suites
.github/workflows/approvals.yml	Adds PR approval requirement workflow (2 approvals for bot-authored PRs)
.github/scripts/run-sql-bench.sh	Adds --iterations passthrough to benchmark binaries
.github/scripts/fuzz_report/templates/new_issue.md	Improves issue template formatting and reproduction instructions
.github/scripts/fuzz_report/cli.py	Uses FUZZ_NAME for issue titles (fallback to FUZZ_TARGET)
.github/runs-on.yml	Switches to private runner config extension
.github/actions/setup-rust/action.yml	Reworks toolchain install flow and adds optional cache timestamping
.github/actions/setup-prebuild/action.yml	Adds AMI-aware setup wrapper with fork fallback
.github/actions/check-rebuild/action.yml	Adds helper action to verify rebuild is a no-op
.config/nextest.toml	Adds nextest override priority for a specific test filter

Files not reviewed (3)

• .idea/copyright/profiles_settings.xml: Language not supported
• .idea/scopes/web_files.xml: Language not supported
• benchmarks-website/package-lock.json: Language not supported

Comments suppressed due to low confidence (1)

encodings/datetime-parts/src/compress.rs:1

• This TryFrom<TemporalArray> implementation computes days/seconds/subseconds (and validates them) but then returns Ok(DateTimePartsData {}) without carrying those computed arrays forward. If DateTimePartsData is expected to represent the encoded data, this would make the conversion silently discard the actual payload. Consider returning a DateTimePartsData value that retains the split components (or, if DateTimePartsData is intentionally a marker/ZST, remove this TryFrom impl and expose a constructor that returns the actual array type instead).

// SPDX-License-Identifier: Apache-2.0

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.