[pull] trunk from spiceai:trunk#849
Merged
Merged
Conversation
Bumps the docker-dependencies group with 1 update: ubuntu. Updates `ubuntu` from 24.04 to 26.04 --- updated-dependencies: - dependency-name: ubuntu dependency-version: '26.04' dependency-type: direct:production dependency-group: docker-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… (1→128 concurrency, sqlite + turso lanes) (#10943) * feat(cayenne): scale metastore pool to 32 + vs_duckdb_scaling benches Lifts the metastore-pool ceiling from K=8 to K=32 on the SQLite backend so metadata-heavy workloads (concurrent scans on tables with deletions, multi-table CDC ingest) stop bottlenecking at 8 in-flight metastore ops on boxes with more cores. SQLite WAL mode permits many concurrent readers per file; writes still serialize at the WAL layer regardless of pool size, so the change is read-side leverage only — Turso already runs K=16, this brings SQLite closer to parity. Adds vs_duckdb_scaling — concurrent throughput benchmarks at 1/2/4/8/16/32/64 workers for both reads (warm-session SELECT COUNT(*) against a 1M-row table) and writes (sustained background insert pumping using the proven CayenneBgWriter pattern from vs_duckdb_concurrent). Local results from this branch: reads @ 64 concurrent: Cayenne 2.08 ms vs DuckDB 4.52 ms (2.18× faster) writes @ 64 concurrent: Cayenne 2.03 M rows/s vs DuckDB 775 K rows/s (2.62× faster) Cayenne wins at every concurrency ≥ 1 on writes and at every concurrency ≥ 2 on reads, and the lead grows with concurrency on both — DuckDB plateaus at concurrency 8 while Cayenne keeps climbing. Also bundles three pre-requisite fixes the bench sweep surfaced: * Cargo.toml: add publish = false to [workspace.package] so chbench-driver's publish.workspace = true (commit 5a60299) resolves — the workspace currently fails to load entirely without it. * vs_duckdb_delete.rs, vs_duckdb_in_list_delete.rs: add the missing `use datafusion::catalog::TableProvider` so the benches actually compile (they error E0599 on `delete_from` without the trait in scope). * vs_duckdb_scan.rs + common.rs: add a `cayenne_warm/` lane parallel to the existing `cayenne/` (cold-session) lane, sharing one long-lived `SessionContext` across iterations — measures the steady-state path a running Spice runtime actually pays vs the cold-CLI path, ~80 µs/query difference. Pure addition; the cold lane is preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cayenne): add CDC pipelined scaling lane to vs_duckdb_scaling The first version of vs_duckdb_scaling exercised the generic insert path (table.insert_into → AppendMutationWriter) but not the production CDC pipelined path (table.write_cdc_append_stream → AppendMutationWriter:: write_cdc_pipelined), which is what spiced's refresh_mode: changes loop actually drives. Add a third lane that pumps `write_cdc_append_stream` + `finish()` end-to-end so we can see whether the Stage-A / Stage-B split translates to higher sustained CDC throughput vs the generic insert path, and compare both against DuckDB INSERT at the same shape. Helper: `cayenne_cdc_write(table, batch)` in vs_duckdb_helpers/common.rs wraps a single RecordBatch into a SendableRecordBatchStream, calls write_cdc_append_stream, returns the row count from Stage A, then awaits finish() so the rows are fully committed before the helper returns. Mirrors the test pattern in `crates/cayenne/tests/staged_append_test.rs:99`. Bench: `vs_duckdb_scaling_cdc/cayenne_cdc/{1,2,4,8,16,32,64}` vs `vs_duckdb_scaling_cdc/duckdb/{1,2,4,8,16,32,64}` — sustained CDC pipelined throughput against ONE table at 7 concurrency levels. Pattern is the proven CayenneBgWriter shape from vs_duckdb_concurrent; new `CayenneCdcBgWriter` is the CDC-pipelined twin. Local results from this branch: cayenne_cdc @ 1 : 949 K rows/s vs duckdb 210 K (4.53× faster) cayenne_cdc @ 64: 1.91 M rows/s vs duckdb 732 K (2.61× faster) Essentially identical scaling shape to the generic insert lane shipped in the parent commit, which makes sense: CDC pipelined and generic inserts share the same underlying mutation_writer and staging_wal infrastructure. Cayenne wins at every concurrency level. Also parameterise DuckDbBgWriter on table_name so the writes lane (table "scaling_write") and the cdc lane (table "scaling_cdc") can share the same writer implementation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cayenne): address PR #10943 review — pooled bench workers + channel wait Two Copilot review threads on vs_duckdb_scaling.rs: 1. Per-iteration worker spawn distorts the measurement. The original read lane re-spawned N OS threads (DuckDB) or N tokio tasks (Cayenne) inside every `b.iter` closure — spawn/join cost lands inside the timed region and grows with concurrency. Refactored to per-worker `mpsc` channel pairs: each concurrency level spawns N workers exactly once, and the bench thread coordinates per-iteration via cheap channel sends/receives. Net effect on the 64-way DuckDB read: previously 4.52 ms (carries N thread spawns), now 2.86 ms (steady-state only). Cayenne 64-way reads moved from 2.08 ms to 2.02 ms — much less affected since tokio task spawn is already cheaper than OS-thread spawn. 2. `std::hint::spin_loop` busy-wait on the bench thread distorts the write measurement and can hang on writer panic. Replaced the AtomicUsize counter + spin loop on both writes and CDC lanes with a per-iteration `mpsc::Receiver::recv_timeout(30s)` wait via a new helper `wait_for_completions`. Blocking, no CPU contention with the writer tasks, panics with a useful diagnostic if a writer stalls or dies. Also reverts the now-redundant `publish = false` addition to `[workspace.package]` — trunk #10939 (b63c480) fixed the underlying chbench-driver build issue by changing `publish.workspace = true` to `publish = false` directly in that crate's Cargo.toml. After merging trunk in, the workspace-root config is no longer load-bearing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(cayenne): apply rustfmt + update module doc to mention CDC lane cargo fmt --all rolled up the `CayenneBgWriter::spawn` and `DuckDbReader::spawn` parameter lists to fit on one line each. Also updated the file-level doc comment that still said "Two workloads" — there are three now (reads, writes, cdc) — and documented the channel-based worker pool + bounded-timeout wait pattern introduced in the previous commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cayenne): address second round of PR #10943 review Two active Copilot threads: 1. vs_duckdb_scaling.rs:97 — `ReadFixtures` stored `duckdb::Connection` in an Arc and gave each spawned reader thread a `try_clone()` of it. Other benches in this suite (vs_duckdb_concurrent.rs:108) treat DuckDB connections as non-Send and use `Connection::open(&db_path)` inside each thread. Switched ReadFixtures to store `duckdb_db_path: PathBuf` and DuckDbReader opens its own Connection from the path inside the spawned thread, matching the rest of the suite. The cost per connection is a few µs on first open (outside the bench's timed region — workers spawn once per concurrency level). 2. metastore/sqlite.rs:208 — the doc said pool size is `min(available_parallelism, 32)` but the implementation falls back to `4` when `available_parallelism()` fails. Documented the fallback, plus the >=2 clamp, so the comment matches the code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cayenne): round 3 of PR #10943 review — must_use drain, reader timeout Four new Copilot threads, all valid: 1. `cayenne_writers.drain(..);` / `duckdb_writers.drain(..);` as statements discard a `#[must_use]` `Drain` iterator. Under `-D warnings` CI this becomes an error. Replaced all four occurrences with `drop(vec)` so the intent (`stop the writers + join`) is explicit and lint-clean. 2. Cayenne reader bench had no per-completion timeout on `cayenne_done_rx.recv().await`. If a reader task panicked or stalled, the bench would hang. Wrapped the `recv().await` in `tokio::time::timeout(Duration::from_secs(30), …)` with a labeled panic naming the lane and completion index, mirroring the `wait_for_completions` helper used by the other lanes. 3. Bench comment on the writes lane said "Wait for one full pass (each writer contributes ≥1 insert)" but the implementation drains `concurrency` total messages from a shared channel — one fast writer can contribute multiple of them. Reworded the comment to match what's actually measured (system throughput across N writers, not per-writer fairness) and noted the `Throughput::Elements` divider that lets criterion report rows/sec. 4. File-level doc still claimed completion drains used `mpsc::Receiver::recv_timeout` exclusively. The async Cayenne reader lane uses `tokio::time::timeout` over `tokio::sync::mpsc` (different crate, same shape). Updated the doc to name both, and called out that stalls/panics surface as labeled panics rather than hangs. Also removed the now-unnecessary `mut` from the four worker `Vec` declarations (the drain → drop swap means the bindings don't mutate). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cayenne): add Turso metastore lane to vs_duckdb_scaling benches User caught that the scaling benches only tested the SQLite metastore. The other `vs_duckdb_*` benches that take a metastore (burst, concurrent, groupby, join, upsert, upsert_scaling) iterate over `CAYENNE_LANES` — which expands to `Sqlite` + `Turso` when the `turso` feature is enabled — but the scaling bench used `setup_cayenne(name)` which hardcodes `Metastore::Sqlite`. Refactored all three workloads (reads, writes, cdc) to wrap their per-concurrency loop inside `for &lane in CAYENNE_LANES { … }`, building fixtures via `setup_cayenne_for(name, lane)` and prefixing the `BenchmarkId` with `lane.lane()` so the two metastore lanes register as: reads: cayenne_warm / cayenne_turso_warm writes: cayenne / cayenne_turso cdc: cayenne_cdc / cayenne_turso_cdc DuckDB stays a single lane (no metastore-backend dimension) — runs once per concurrency level outside the lane loop. For the reads workload, the DuckDB fixture is also factored out of the ReadFixtures struct (`DuckDbReadFixture`) so it can be built once and shared across both Cayenne lane runs without rebuilding the parquet file or re-loading DuckDB. Local smoke (concurrency 1 → 64): reads/cayenne_warm/64 ≈ 1.95 ms reads/cayenne_turso_warm/64 ≈ 2.03 ms ← within noise of sqlite So Turso reads scale comparably to SQLite reads at the 64-way concurrency goal, which is the architectural claim Turso's K=16 pool was designed for. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cayenne): extend vs_duckdb_scaling concurrency sweep to 128 User asked for 128-core/thread coverage. Adds 128 to `CONCURRENCIES` so the per-workload curve covers oversubscription on common 64-core SMT-2 boxes (128 logical threads). Each workload (reads/writes/cdc) now runs at: [1, 2, 4, 8, 16, 32, 64, 128] across both Cayenne metastore lanes (sqlite + turso) plus the DuckDB baseline. With the existing 30 s per-iteration timeout, a stalled writer at 128 will surface as a labeled panic rather than hang the bench. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cayenne): skip turso lane in write/CDC scaling benches (write-write conflict) User pointed out the scaling benches were missing the turso metastore. Adding it to all three workloads surfaced a real bug: the sustained-writes pattern (one Cayenne fixture, many writes from a long-lived writer task) reproducibly trips Turso's optimistic concurrency control on the very first insert at concurrency=1 with: External(Catalog { source: InvalidOperation { message: "Failed to commit inline mutation transaction", source: Database { message: "Failed to commit transaction: Write-write conflict" } }}) This is NOT a bench harness problem on our side: * The existing `vs_duckdb_burst/cayenne_turso/*` and `vs_duckdb_upsert/cayenne_turso/*` benches exercise turso writes and work — they use `iter_batched` with a fresh fixture per criterion sample, so writes never accumulate against one Turso instance. * The reader scaling bench also runs Turso fine — `cayenne_turso_warm` measurements through 128-way concurrency match the SQLite lane within noise. * Only the sustained-writes shape trips it. Hypothesis: Cayenne's mutation path fans out across multiple Turso pool connections (round-robin K=16) for what is logically one transaction; Turso treats those as conflicting concurrent writes. SQLite WAL is more permissive and doesn't surface the same conflict in the same workload. This needs a follow-up in Cayenne's turso metastore wiring (pin a logical mutation to one pool connection), separate from this PR's bench scope. Filing the issue. Until that lands, the sustained-writes / CDC lanes of this bench gate Turso behind `lane_supports_sustained_writes()`. The reads lane still runs both backends. The bench file's per-bench comments call out the gate so the next person doesn't re-discover the panic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cayenne): per-iteration go-signaling on write/CDC benches Copilot reviewer flagged that the write/CDC scaling lanes drained `concurrency` completion messages from a shared mpsc::Receiver per iteration without defining an iteration boundary — sustained writers keep pumping between iterations, so the channel can have a backlog at the start of an iteration and the iteration completes immediately by draining already-produced messages, overstating throughput. Refactored all three writer structs (`CayenneBgWriter`, `DuckDbBgWriter`, `CayenneCdcBgWriter`) and their bench callers to per-iteration go-signaling, matching the read lane shape: - Each writer owns its own `go_rx` channel. - Bench iter sends ONE `()` per writer at the start of the iteration. - Each writer does ONE insert (or CDC append), sends `()` on the shared `done_tx`, then loops back to wait for the next `go`. - Bench iter drains exactly `concurrency` completions before sampling. Each criterion sample now measures **N fresh writes**, not "N messages from a backlog that may have accumulated between iterations". Local smoke test (sqlite cayenne writes, new pattern): 1: 984 µs | 4: 3.44 ms | 16: 11.25 ms | 64: 24.64 ms 2: 2.03 ms | 8: 5.61 ms | 32: 16.61 ms About 10-20% faster than the old sustained-loop numbers, consistent with the previous numbers being slightly inflated by backlog drain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cayenne): reuse SessionContext in CDC writer + clarify throughput doc Two valid Copilot review threads addressed: 1. `cayenne_cdc_write` previously built `SessionContext::new()` on every call. In the scaling bench each writer pumps many batches per second, so that per-batch setup + allocator churn was visible in the throughput numbers and unrelated to the actual CDC pipeline cost. The helper now takes `&Arc<TaskContext>` from the caller; the bench's `CayenneCdcBgWriter` builds one `SessionContext` at spawn time and reuses its `task_ctx` across every batch. That matches how a long-running Spice runtime drives `refresh_mode: changes`. 2. The bench comment on `Throughput::Elements` claimed criterion reports per-worker throughput. Criterion actually treats `Throughput::Elements(N)` as total elements per iteration and reports the aggregate rate. Reworded the comment so the units are unambiguous. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(cayenne): mention pool-size fallback in SqliteConnectionPool doc Copilot noted that the `SqliteConnectionPool` doc comment claimed pool size = `min(available_parallelism, 32)` but didn't mention the `map_or(4, ...)` fallback when `available_parallelism()` fails (rare — seccomp-restricted environments). The inline doc on `pool()` already called out the fallback; this brings the struct-level doc and the `SqliteMetastore::pool` field doc in line so all three locations are consistent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion-configured tables (#10936) * perf(cayenne): background retention via PostWriteMaintenance scheduler `apply_retention_if_configured` ran inline inside `AppendMutationWriter:: write` for every CDC commit. The implementation calls a full `CayenneDeletionSink::delete_from()` scan against the table file set under `write_lock`; cost scales with table size. For a 1M-row table at 1000 CDC batches/sec, retention re-scanned the whole table 1000 times/sec and blocked the next write. Extend the existing `PostWriteMaintenance` debounced scheduler to track retention requests alongside stats and listing-refresh. Writers now post a `retention_requested` signal to the scheduler and return — the maintenance loop coalesces signals (multiple writes inside one debounce window collapse to one retention scan), runs the scan against the published snapshot, and reschedules listing refresh + compaction if any rows were deleted. Add `flush_pending_maintenance(&self)` as `pub` so tests can drain the scheduler before asserting on post-retention state, and so coordinated shutdown can ensure no scheduled retention is lost when the table is dropped. `record_file_pk_keys` / `clear_cached_pk_keyset` decision now resolves on `has_retention_delete_filters()` at scheduling time, not on retention's actual delete count (which is no longer known synchronously). Tables with retention configured pay one fresh disk-scan per insert to rebuild the PK keyset cache — the conservative tradeoff for the bg model. Update `test_retention_filters_apply_on_insert_impl` to call `flush_pending_maintenance` between the insert and the assertion on the delete file. The other four retention tests pass unchanged because they either assert via scan-time filtering (which is independent of bg retention completion) or use the staged-commit shape. This is Phase 1 of the P1-6 fix. Phase 2 (separate PR) removes `has_retention_delete_filters` from the `can_stage_for_pipeline` blockers in `write_cdc_pipelined`, so CDC tables with retention configured join the pipelined fast path automatically. All 251 cayenne tests pass (1 pre-existing flake on `test_high_ concurrency_stress_sqlite` that also fails on trunk and is unrelated). Clippy `-D warnings` clean. Build with `--features turso` clean. * perf(cayenne): enable CDC pipelining for retention-configured tables Phase 2 of the background-retention fix. With retention now scheduled asynchronously by `PostWriteMaintenance`, the historical reason `has_retention_delete_filters()` was in the `can_stage_for_pipeline` blocker list at `mutation_writer.rs:233` no longer applies — retention no longer runs inline under `write_lock`. Remove the blocker so CDC writes against retention-configured tables take the same Stage A (file write + WAL prepare) / Stage B (move + visibility flip) pipelined path that non-retention tables already use. `CayenneStagedAppend::finish` schedules the retention request after publish; the bg scheduler picks it up on the next debounce. Update `finish` to match `write_prepared_stream`'s PK keyset cache treatment: when retention is configured, clear the cache conservatively (retention's outcome is not yet known) rather than recording the new keys. Combined effect of Phase 1 + Phase 2: CDC commit latency on a retention-configured table drops from `staged_write + commit + retention_scan` to `staged_write` (Stage B and retention both run after the writer returns). For a 1M-row table that's a ~10-100 ms latency reduction per commit depending on retention scan time, and unblocks the pipelined throughput for the most common Spice acceleration shape (refresh_mode: changes + retention). All cayenne tests pass (251 pass; 2 pre-existing flakes on the high-concurrency stress paths, also failing on trunk independently). Clippy `-D warnings` clean. Build with `--features turso` clean. * fix: address PR review comments and sync with trunk Addresses two review threads on PR #10936: 1. `apply_retention_filters` now passes `Some(Arc::clone(&self.write_lock))` to `CayenneDeletionSink` so the sink serializes against concurrent inserts / listing refreshes for the duration of the scan. Before this fix, the bg maintenance loop called `apply_retention_filters` without holding `write_lock`, but the sink was constructed with `None` (the pre-fix comment assumed an inline caller held the lock). Updated the doc comment to reflect the new lock invariant. 2. `flush_pending_maintenance` now loops until both the queued maintenance state is empty AND the bg `scheduled` flag is false. The previous implementation returned as soon as the queue was empty, which could race with the bg loop mid-iteration (the loop had taken the state but not yet finished running it). The new shape drains queued state synchronously and, if `scheduled` is still true, sleeps 5 ms before re-checking — covers the in-progress-iteration case. Updated the doc comment. Also fixes a trunk-side workspace regression from PR #10935 (adding `publish.workspace = true` to `tools/chbench-driver/Cargo.toml` without defining `workspace.package.publish`). Sets `publish = false` on `[workspace.package]` so the workspace metadata loads cleanly. This unblocks Rust Lint, Build (release profile), and every other check that runs `cargo metadata` against the workspace. `cargo fmt --all` cleanup to remove a stray blank line in `mutation_writer.rs`. All 5 retention tests pass after the fix. Cayenne-scoped clippy clean. Workspace `cargo metadata` succeeds. * fix: address PR review on retention error handling and listing-refresh Two review comments on `run_maintenance_state`: 1. Retention errors were logged and swallowed; a persistently failing retention scan would have left expired rows undeleted indefinitely until a later successful pass. Now the failure path re-queues `retention_requested = true` in the maintenance state, so the next debounce cycle retries. Log level raised to `error` (was `warn`) because the retry turns a single failure into a steady signal worth alerting on. `flush_pending_maintenance` doesn't need a Result return because its loop will keep draining queued state until empty AND the bg loop is idle — under a persistent retention failure it would spin on the 5 ms re-check, which is acceptable for test/shutdown paths. 2. Listing-table refresh could run twice in one pass — once for `state.refresh_listing` and again for `retention_deleted > 0`. The refresh rebuilds the entire `ListingTable`, so doubling it is real cost. Restructured the function to defer the refresh to the end and run it at most once per pass, when either signal is set. Retention now runs before the refresh so deleted rows are reflected in the rebuilt listing. All 5 retention tests still pass. * chore: drop workspace-level publish=false (#10939 fixed it differently) Trunk merged in #10939 which adds `publish = false` directly to `tools/chbench-driver/Cargo.toml` (replacing the broken `publish.workspace = true` line). My earlier workspace-level fix is now redundant — revert it so this PR's diff stays focused on the Cayenne retention work. Addresses the reviewer's request to either document the Cargo.toml change or move it out of this PR. Going with "move out", since trunk already has the right fix. * fix: improve retention error handling and update flush_pending_maintenance to return errors
* Vendor Vortex DataFusion for Cayenne * refactor: Improve code readability and consistency in Vortex modules * refactor: Improve documentation and code clarity in cache and exprs modules * refactor: Enhance documentation clarity and improve code consistency across multiple modules * refactor: Improve code clarity and consistency in Vortex modules * Expose Cayenne footer cache and CDC metrics (#10929) * Expose Cayenne footer cache and CDC metrics * Restore Cayenne target file size default * Move Cayenne footer cache to runtime params * refactor: streamline VortexConfig initialization and enhance footer cache drift logging * refactor: simplify VortexConfig initialization by using struct update syntax * refactor: Enhance error handling and return types in expression conversion functions * refactor: Improve code clarity and consistency in scalar conversion and access plan modules * refactor: Simplify scalar conversion logic and improve test error messages * refactor: Update column statistics initialization to use table schema field count * refactor: Improve code clarity by using `map_or_else` for optional values and enhancing documentation formatting * refactor: Enhance Clippy configuration for improved linting during tests * Benchmarks: add publish.workspace = true to chbench-driver Cargo.toml (#10935) * chore: Update Cargo.toml and README.md for repository details; enhance metadata documentation * refactor: Enhance dynamic filter handling and pushdown logic in VortexOpener * refactor: Improve logging format in CayenneAccelerator and enhance ProcessedProjection documentation * docs: Update comment for leftover_projection to improve clarity * test: Use named Vortex persistent snapshots * docs: Add DR-008 for vendoring vortex-datafusion * test(vortex): Decimal-to-f64 cast regression for spiceai/vortex#51 * feat(vortex): Auto-tune scan concurrency * fix(vortex): Improve row deletion adjustment and add tests for edge cases * perf(vortex): Project partition columns zero-copy * fix(vortex): Correct operator conversion test cases and improve precision handling in VortexFormat * refactor(vortex): Simplify distinct count calculation and add unit tests * refactor(vortex): Replace scan_listing_tables cache with scan_file_statistics for improved snapshot handling * feat(vortex): introduce ProjectionPushdown enum for scan behavior control and update related implementations * feat(deletion): add iterator method for PositionDeletionVector * fix(sink): use clone method for object_store in write_record_batch_stream_to_files * fix(sink): use Arc::clone for object_store in write_record_batch_stream_to_files * docs: clarify Vortex adapter ownership * fix(scalars): improve UTF-8 conversion handling in TryToDataFusion implementation * fix(docs): improve documentation clarity for Precision conversion and VortexSource methods * fix(docs): improve documentation formatting for Precision conversion and Vortex metrics * fix(docs): improve documentation clarity for Vortex metrics extraction and conversion functions * fix(docs): enhance documentation clarity for Vortex metrics extraction functions * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Sergei Grebnov <sergei.grebnov@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…ict" (#10946) * fix(turso-shared): retry on Turso "Write-write conflict" on commit The retry detector in `is_retryable_write_conflict_message` only recognised SQLite-engine error patterns (`sqlite_busy`, `sqlite_locked`, `database is busy`, `database is locked`). Turso's BEGIN CONCURRENT MVCC reports a different message on commit-time conflicts: "Failed to commit transaction: Write-write conflict" Without that string in the matcher, Cayenne's retry-on-conflict loops in `commit_inlined_mutation`, `commit_on_conflict_deletions`, snapshot publish, and friends would see a non-retryable error on the very first Turso commit conflict and bail out — exactly what the new sustained- writes scaling bench hit at concurrency=1 against a Turso fixture. This was masked on SQLite because the WAL engine serialises writers internally, so the same logical race surfaces as `SQLITE_BUSY` / `database is locked` rather than a Turso-style MVCC reject. The sqlite-only bench lanes converged cleanly through 128-way concurrency. The matcher now also accepts `"write-write conflict"` (case-insensitive), so every existing retry loop converges under sustained Turso writes without further changes. Re-enables the cayenne_turso lane in `vs_duckdb_scaling_writes` and `vs_duckdb_scaling_cdc` — they had been gated off by `lane_supports_sustained_writes()` pending this fix. Local smoke: `vs_duckdb_scaling_writes/cayenne_turso/1` now completes cleanly with a 1.17 ms median (sqlite cayenne lane: 1.03 ms — within noise, as expected for single-writer workloads where the metastore backend is not the bottleneck). Added a positive-case unit test for the new pattern in `turso-shared/src/lib.rs` so a future change to the matcher catches the regression. Existing positive/negative cases still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(cayenne): update stale "Turso writes skipped" comments in scaling bench Two long-form comments in `vs_duckdb_scaling.rs` still described the Turso write/CDC lanes as intentionally skipped. With the matcher fix in `turso-shared/src/lib.rs` from this PR, the existing retry loop in `commit_inlined_mutation` (and siblings) converges and Turso writes complete cleanly through 128-way concurrency. Reworded both `bench_write_scaling` and `bench_cdc_scaling` to reflect the current behaviour — the `lane_supports_sustained_writes()` gate stays as a future switch rather than a Turso-specific opt-out. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… IN-list rewrite + microbench coverage (#10916) * Add Cayenne microbench coverage * Refactor deletion handling and optimize shared index usage * refactor: streamline SQLite connection configuration and improve Turso row handling * feat: add primary key point lookup optimization and rewrite consecutive IN-list to range * Fix Cayenne provider try-cast matching * Fix Cayenne literal cast extraction * Select SQLite bench lane explicitly * refactor: simplify IN-list rewriting to use BETWEEN expression * feat: add logical optimizer rule to rewrite IN-list to BETWEEN for performance improvement * refactor: improve comments for clarity in table provider and logical optimizer * feat: add tests for IN-list to BETWEEN rewrite rule in logical optimizer * feat: expand benchmark sizes for insert and query batch tests * refactor: enhance memory gate checks in same-source join optimization * Tighten anti/semi hash→sort-merge gate around the memory pool The byte gate (memory pool fraction) was previously AND-ed with the row count gate (default 10M rows). That left q21-shaped self-joins over wide projections OOMing: build is small in row count (200K-500K) but large in bytes (>1.5 GB at SF1 on a typical 12 GB pool), so the row gate returned early before the byte gate could fire. When the memory gate is configured (runtime-wired), it is now the *primary* decision gate. The row count threshold only applies when no memory information is available (direct DataFusion users with no runtime config). Same join scope (LeftAnti/RightAnti/LeftSemi/RightSemi only); same default thresholds; no new optimizer rules added. Regression test added for the q21 shape: 200K-row Cayenne self-join, LeftAnti, byte estimate exceeds the configured gate -> rewritten to SortMergeJoinExec. * Add tpcds_explain.sh helper for capturing local EXPLAIN ANALYZE plans * Fix workspace publish inheritance after trunk merge * Enhance tpcds_explain.sh to stage spicepod and resolve paths for spiced execution * Add TPC-DS SF1 Cayenne spicepod variant with PKs + target_partitions=4 Declares standard TPC-DS primary keys on all 24 datasets (dimension tables get their single-column surrogate sk; fact tables get the documented composite PK from the TPC-DS spec). Lowers runtime.query.target_partitions from the default (num_cpus) to 4 to reduce shuffle volume on q72-shape multi-way joins. Use via scripts/tpcds_explain.sh: SPICEPOD='test/spicepods/tpcds/sf1/accelerated/file[parquet]-cayenne[file]-pk.yaml' \ scripts/tpcds_explain.sh q21 q72 q9 * Log target_partitions configuration in DataFusionBuilder * test: add unit test for target_partitions configuration in DataFusionBuilder * test: add unit test for nested target_partitions in SpicepodDefinition * fix: improve handling of overflow row IDs in CayenneDeletionSink and optimize filter rewriting in TableProvider * fix: add retrieval of target partitions in CayenneTableProvider execution plan * fix: simplify error handling in position-based deletion and improve in-list expression rewriting
* remove eval related things * bad import --------- Co-authored-by: Jeadie <jeadie@users.noreply.github.com>
* fix(pg catalog): include catalog in FK foreign_table metadata Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(pg): remove redundant catalog_name FK parameter Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Jeadie <jeadie@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: Remove Cayenne Catalog from catalog registration * fix: Ignore cayenne catalog tests --------- Co-authored-by: Jack Eadie <jack@spice.ai> Co-authored-by: Sergei Grebnov <sergei.grebnov@gmail.com>
…et(N, col)') (#10880) * handle NULL sentinel for nullable partition expressions (e.g. 'bucket(N, col)') * handle nullability in PartitionValue * Update 'PartitionValue' * Fix test compile errors after PartitionValue -> HashMap<String, Option<String>> change * linting * change test to handle additional partition * fix bucket count * fix: wrap PartitionValue test fixtures in Some() to match HashMap<String, Option<String>> type * Fix 'test_distributed_acceleration_order_by_limit_pushdown' * fix trunk? * missing colon * fix merge * fmt --------- Co-authored-by: Jeadie <jeadie@users.noreply.github.com> Co-authored-by: Sergei Grebnov <sergei.grebnov@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )