Merge upstream Vortex 0.70.0 into spiceai-52

.agents/skills/ci-failure-analysis/SKILL.md

@@ -0,0 +1,100 @@
+---
+name: ci-failure-analysis
+description: Analyze Vortex GitHub Actions CI failures. Use when asked to investigate failed CI runs, failed jobs, or when the user mentions "/ci-failure-analysis".
+---
+
+# Vortex CI Failure Analysis Skill
+
+Analyze failed GitHub Actions runs for the Vortex repository and identify whether the failure is
+caused by the PR, pre-existing flakiness, infrastructure, or an unrelated main-branch issue.
+
+## Inputs
+
+Use any PR number, repository, run ID, failed job metadata, or log snippets supplied by the user or
+automation prompt. If a needed value is missing, discover it with the narrowest `gh` command that
+can answer the question.
+
+## Workflow
+
+1. List failed jobs for the workflow run:
+
+   ```bash
+   gh run view <run-id> --repo <owner/repo> --json jobs
+   ```
+
+2. Fetch only failed job logs first:
+
+   ```bash
+   gh run view <run-id> --repo <owner/repo> --job <job-id> --log-failed
+   ```
+
+   If that fails, use the Actions API:
+
+   ```bash
+   gh api repos/<owner/repo>/actions/jobs/<job-id>/logs
+   ```
+
+3. If any `gh` command fails with `error connecting to api.github.com` in a sandbox, rerun it with
+   escalated network permissions immediately.
+
+4. Classify each failure:
+
+   - Rust build errors: compiler diagnostics, spans, trait bound failures, feature-gate issues.
+   - Rust test failures: failing test name, panic/assertion output, expected vs actual values,
+     source path and line.
+   - Clippy failures: lint name, file path, line, and suggested fix if shown.
+   - Formatting or public API failures: changed files and commands needed to regenerate output.
+   - Python/docs failures: pytest, maturin, Sphinx, doctest, or packaging output.
+   - Infrastructure failures: toolchain download, cache, runner, network, disk, timeout, or
+     service issues.
+
+5. Fetch the PR diff and metadata only after the failing log section is understood:
+
+   ```bash
+   gh pr view <pr-number> --repo <owner/repo> --json title,body,baseRefName,headRefName,files,commits
+   gh pr diff <pr-number> --repo <owner/repo>
+   ```
+
+6. Reproduce narrowly when practical:
+
+   ```bash
+   cargo test -p <crate-name> <test-name>
+   cargo clippy -p <crate-name> --all-targets --all-features
+   make -C docs doctest
+   uv run --all-packages pytest <path>
+   ```
+
+7. Check whether the same failure appears on recent main-branch runs or open issues before calling
+   it PR-caused.
+
+## Report Format
+
+Post or return one concise Markdown report:
+
+````markdown
+## CI Failure Analysis
+
+### Status
+<PR-caused | likely pre-existing | infrastructure | inconclusive>
+
+### Failed Jobs
+- `<job name>`: <build | test | clippy | fmt | docs | infra>
+
+### Relevant Log Output
+```text
+<only the failing lines needed to understand the issue>
+```
+
+### Correlation With PR Changes
+<Explain whether the diff touches the failing area and cite files/functions.>
+
+### Recommended Next Step
+<One or two concrete commands or code fixes.>
+````
+
+## Rules
+
+- Show relevant failure excerpts, not full logs.
+- If many tests fail, detail the first few distinct failures and summarize the rest.
+- Do not guess. If causation is unclear, say what was checked and what would resolve it.
+- Prefer one PR comment or one final report over multiple fragmented updates.

.agents/skills/pr-review/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: pr-review
+description: Review Vortex pull requests for correctness, Rust soundness, performance, API compatibility, and test coverage. Use when reviewing PRs, reviewing code changes, or when the user mentions "/pr-review".
+---
+
+# Vortex PR Review Skill
+
+Review Vortex changes for issues that CI may miss: semantic correctness, Rust soundness,
+zero-copy and alignment invariants, performance, API compatibility, and missing regression
+coverage.
+
+## Usage Modes
+
+### GitHub Actions Mode
+
+When invoked by a PR automation, the prompt may already include PR metadata, issue body,
+comments, and changed files. In this mode, use git commands for the diff and history:
+
+```bash
+git diff origin/<base-branch>...HEAD
+git diff --stat origin/<base-branch>...HEAD
+git log origin/<base-branch>..HEAD --oneline
+```
+
+If the base ref is missing, fetch only that ref:
+
+```bash
+git fetch origin <base-branch> --depth=1
+```
+
+Do not refetch data already provided in the prompt. Use the supplied comments and metadata as
+review context.
+
+### Local CLI Mode
+
+When the user provides a PR number or URL, use `gh` to fetch the PR metadata, comments, and diff:
+
+```bash
+gh pr view <PR_NUMBER> --json title,body,author,baseRefName,headRefName,files,additions,deletions,commits
+gh pr view <PR_NUMBER> --json comments,reviews
+gh pr diff <PR_NUMBER>
+```
+
+If `gh` cannot connect to `api.github.com` because of sandbox networking, rerun with escalated
+network permissions.
+
+## Review Workflow
+
+1. Read `AGENTS.md` and any nested `AGENTS.md` for project conventions.
+2. Identify the change intent from the PR title, body, commits, and tests.
+3. Group changed files by area: arrays, encodings, buffers, file/layout, integrations, bindings,
+   docs, or CI.
+4. Trace changed behavior through callers, trait implementations, dtype/nullability handling,
+   validity masks, and tests.
+5. Focus findings on actionable defects. Avoid commenting on formatting or issues already covered
+   by clippy, rustfmt, or generated API checks.
+6. Scope verification to the change. Rust/API changes need Rust checks; docs-only, agent-only,
+   symlink-only, and metadata-only changes should use targeted validation such as Markdown review,
+   `ls`, `find`, `git status`, or relevant config linters.
+
+## Review Areas
+
+| Area | Focus |
+| --- | --- |
+| Correctness | Length and dtype invariants, nullability, validity masks, offset math, canonicalization, boundary conditions, empty arrays, scalar vs array behavior |
+| Rust soundness | `unsafe` blocks, aliasing, lifetimes, alignment, FFI boundaries, panic safety, ownership of buffers and arrays |
+| Compression and IO | Encoding metadata, statistics, layout evolution, file compatibility, scan projection/filter behavior, async IO edge cases |
+| Performance | Unnecessary copies, lost zero-copy behavior, avoidable allocations, poor cache locality, quadratic loops, excessive dynamic dispatch in hot paths |
+| Error handling | Correct `vortex_err!` and `vortex_bail!` usage, useful messages, no accidental panics on user data |
+| API compatibility | Public API docs, public-api lock updates, feature flags, crate boundaries, Python/Java binding impacts |
+| Tests | Regression coverage, edge cases, parameterized cases with `rstest`, use of `assert_arrays_eq!`, docs doctests when docs change |
+| Verification scope | Avoid requesting or running expensive workspace checks when the PR only changes docs, agent files, symlinks, or metadata |
+
+## Output
+
+Lead with findings, ordered by severity. For each finding include:
+
+- File and line reference.
+- Why the issue is a real bug or material risk.
+- A concrete fix or verification path when possible.
+
+Use inline review comments when the environment supports them and a precise changed line is the
+best place for the feedback. Keep broad design feedback in the summary.
+
+If no issues are found, say so explicitly and mention any residual risk or tests not run.
+
+## Principles
+
+- Review the code that changed, but inspect enough surrounding code to validate invariants.
+- Do not infer causation from commit messages alone. Verify with code, tests, or logs.
+- Do not ask for broad rewrites when a narrow fix would address the risk.
+- Do not downgrade a correctness or soundness issue to a nit because it is inconvenient.
+- Be specific and proportionate.

.agents/skills/query/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: query
+description: Answer questions about the Vortex codebase or pull requests. Use when asked a question via "/query" or when the user wants to understand code, architecture, behavior, or implementation details.
+---
+
+# Vortex Query Skill
+
+Answer questions about the Vortex project, its pull requests, and its implementation.
+
+## Key Context
+
+- Vortex is a Rust workspace for columnar arrays, compression encodings, file IO, and scan
+  integrations.
+- `vortex-array` defines the core array traits, dtype system, canonical arrays, and base
+  encodings.
+- `vortex-buffer` owns aligned zero-copy buffers.
+- `vortex-file` and `vortex-layout` implement file and layout reading.
+- `encodings/*` contains specialized compressed encodings.
+- Python, Java, DuckDB, and DataFusion integrations live in their own workspace areas.
+
+## Workflow
+
+1. Read `AGENTS.md` and any closer scoped `AGENTS.md` before relying on conventions.
+2. Use `rg` and targeted file reads to identify the relevant crate, module, and tests.
+3. If the question is about a PR, inspect the diff and comments before answering.
+4. If the question is about behavior, trace the implementation through public entry points,
+   encoding-specific implementations, and tests.
+5. Answer with concrete file paths and line numbers when they help.
+
+## Answering Guidelines
+
+- Separate confirmed facts from inference.
+- Prefer precise code references over broad descriptions.
+- Mention important uncertainty and describe what would verify it.
+- Do not invent architecture. If the repository does not answer the question, say what you
+  checked and what is still missing.

.cargo/config.toml

@@ -3,9 +3,6 @@ rustflags = [
     "-C", "force-frame-pointers=yes",
 ]
 
-[target.wasm32-unknown-unknown]
-rustflags = ['--cfg', 'getrandom_backend="wasm_js"', '-C', 'target-feature=+atomics']
-
 [alias]
 xtask = "run -p xtask --"
 vx = "run -p vortex-tui --"

.claude

@@ -0,0 +1 @@
+.agents

.github/AGENTS.md

@@ -5,7 +5,6 @@ Nightly is required for:
 - `-Z` flags: sanitizers (`-Zsanitizer=address`), miri (`-Zmiri-*`), publish (`-Zpublish-timeout`)
 - `cargo-fuzz` (requires nightly)
 - `public-api` xtask (nightly rustdoc JSON)
-- `--cfg vortex_nightly` (enables `portable_simd` feature gate)
 
 Everything else (build, clippy, tests, docs, benchmarks, packaging) should use stable.
 

.github/ISSUE_TEMPLATE/epic.md

@@ -0,0 +1,77 @@
+---
+name: Epic
+about: Umbrella issue for a major initiative tracked via sub-issues.
+title: "Epic: "
+labels: epic
+---
+
+<!--
+An Epic is the umbrella for work that ships as several tracking issues. Attach
+them with GitHub's "Create sub-issue" or "Add existing sub-issue" button. The
+sub-issue panel tracks progress, so this issue does not need a checklist.
+
+Keep this issue readable. Route discussion elsewhere:
+
+- Open-ended scope or product questions: open a GitHub Discussion and link it
+  here.
+- Design debate scoped to one piece of work: open a tracking issue as a
+  sub-issue and discuss it there.
+- Bug reports: file a bug issue and reference this Epic.
+
+Short clarifying comments are fine. Long threads should be redirected.
+-->
+
+This Epic is for ...
+
+<!--
+One or two sentences explaining this Epic.
+
+Please also link any roadmap, design, or external context that helps readers
+understand the scope.
+-->
+
+## Status
+
+<!--
+Proposed / Active / Blocked / Complete. Briefly explain the current state.
+-->
+
+**Proposed.**
+
+## Goal
+
+<!--
+Cover both:
+
+- What "done" looks like for a user or downstream consumer. Name the concrete
+  API, format, encoding, or benchmark.
+- What people might reasonably expect this Epic to cover but it will not.
+  Non-goals stop scope creep.
+-->
+
+## Motivation
+
+<!--
+Why this, why now? Cover:
+
+- The concrete problem and who/what hits it. Examples: a downstream integration
+  (DataFusion, DuckDB, Python, Java), a benchmark gap (TPC-H, ClickBench,
+  vortex-bench), or a file format or encoding limitation.
+- What is currently blocked, slower, or larger than it should be. Link the
+  benchmark numbers, profile output, or related issues.
+
+If the answer is "it would be nice to have", the Epic is not ready.
+-->
+
+## Unresolved questions
+
+<!--
+Scope-level questions that must be answered before this Epic closes.
+Implementation questions belong in the relevant tracking issue.
+
+Each entry should be specific enough that a future reader can tell whether it
+has been answered. If a question grows large, please create a new tracking
+issue or Discussion and link it.
+-->
+
+- [ ] None yet.