Skip to content

CHANGELOG.md

Latest commit

 

History

History
990 lines (804 loc) · 53 KB

CHANGELOG.md

File metadata and controls

990 lines (804 loc) · 53 KB
Raw

Changelog

All notable changes to Quiver are documented here.

The format is based on Keep a Changelog, and the project adheres to Semantic Versioning. Quiver is pre-1.0: minor releases ship coherent, owner-gated feature sets and may include pre-1.0 API refinements. See docs/roadmap.md for the per-release rationale and Definitions of Done.

Unreleased

0.33.0 — Unburdened — 2026-07-04

Added

  • Streaming, memory-bounded index build — the live wiring (ADR-0071, completing ADR-0070 Increment B). The IVF index rebuild — both the server's off-lock path (ADR-0062) and the embedded open/ensure_indexed path — now streams its vectors from the immutable on-disk .vec segments instead of materialising the whole n·dim corpus in a resident flat array (~51 GiB at 100M×128). Store::capture_vector_source reopens the referenced segments' .vec mmaps as owned, lock-free handles that a concurrent checkpoint cannot disturb; scan_collection captures one for a dense IVF collection and rebuilds the (usually empty) sparse index from a payload-only scan, and both rebuild entry points feed Ivf::build_streaming. Peak build memory for the vectors drops from O(n·dim) to O(sample + nlist·dim + n·m_bytes); the corpus stays on disk, read on demand. No on-disk or wire format change; defaults unchanged. The reference-hardware RSS measurement (≥ 20M) is deferred to the dedicated box and no number is claimed — the change lands on correctness tests (a rebuild spanning a reopened sealed segment and the active buffer) and the code-level elimination of the arena.

0.32.0 — Streaming — 2026-07-04

Foundational work toward the single-box 100M build, plus a cluster-search latency win. No on-disk or wire format change; defaults are unchanged.

Added

  • Streaming, memory-bounded index build — the primitive (ADR-0070). Ivf::build_streaming builds an IVF+PQ index by streaming vectors from a re-iterable source in two bounded passes — a deterministic reservoir-sample train pass, then a stream-encode pass — instead of materializing the whole n·dim corpus in RAM. It is byte-identical to Ivf::build for collections within the training sample, proven by a parity test over L2 and Cosine. This is the core primitive that lifts the batch build's O(n·dim) RAM ceiling toward a single-box 100M; wiring it into the embedded scan→build path is the next increment and is not yet in effect — this release ships the primitive and the accepted ADR, not the end-to-end memory win.

Performance

  • Concurrent cluster scatter-gather. A cluster search queried each shard sequentially, so its latency was the sum of the per-shard latencies. It now fans out with try_join_all and tracks the slowest shard instead. Each shard still returns its local top-k, so the merge and the final result are identical — the exact-ground-truth cluster test passes unchanged.

0.31.0 — Reinforced — 2026-07-02

An adversarial-audit remediation, hardening, and scale pass: 18 confirmed bugs plus one plausible finding fixed (each regression-tested), security/robustness hardening, and IVF scale enhancements proven to 10M vectors on a laptop-class box. No on-disk or wire format change; the single node is unchanged at zero overhead when the new knobs are left at their defaults.

Fixed

  • Critical — Raft state-machine durability. The applied-state pointer and snapshot were held only in memory, so a voter restart re-applied the entire committed log onto the already-durable engine, and a post-compaction cold restart could not reconstruct state or membership. They are now persisted on apply/build/install and reloaded on open, with a create-idempotency guard for the one crash window, and a durable restart-without-reapply regression test.
  • next_collection_id recovery from replayed WAL creates — closes an id-reuse / silent-data-loss window after a crash before the first checkpoint.
  • IVF-PQ metric orientation — the PQ path reported the raw ordering value instead of the true metric, inverting cosine ranking under MVCC and returning negative scores; it now reports the true orientation.
  • Exact large-integer payload filters (no f64 coercion above 2⁵³); cluster-router mutations are audited; un-routed router ops return 501 instead of silently querying the empty local engine; durable raft-log directory fsync; atomic coordinator state writes; fresher online-migration copy; the Python SDK percent-encodes path segments; plus signed-zero secondary-index filters, DCPE tag normalization binding, IVF tombstone-cell skipping, status-based 404 classification, MCP k/limit caps + dim validation, an rrf_k0 guard, malformed-sparse rejection, and out-of-order-safe OpenAI-embedding indexing.

Security

  • Authorize before the write-guard (no follower-role disclosure to out-of-scope keys); GET /cluster/map requires Admin; the decrypted DEK is zeroized before its transient buffer drops; secondary-index range queries binary-search their window.

Performance

  • IVF scale to 10M on a laptop. Codebooks now train on a deterministic 262k sample instead of all N (1M codebook+index build ~15× faster in isolation); nlist ≈ √n replaces the fixed 64, so a query probes a small fraction of cells instead of scanning all of them (1M query p50 66 ms → 13.8 ms, ~10× at scale); the redundant L2/Dot build copy is elided via Cow (what let 10M fit in 15 GiB RAM); and a large bulk load auto-checkpoints at a byte budget (QUIVER_CHECKPOINT_BYTES, default 256 MiB) so ingest stays memory-frugal. Measured on a 15 GiB laptop-class box: 10M vectors built and served at query p50 75.8 ms, 529 B/vector on disk; IVF-Flat (exact) recall@10 = 0.998.

Docs

  • A measured scale characterization (docs/benchmarks/scale-characterization.md), a new "Scale characterization" section in Quiver, Explained (§7.4), and a roadmap Future / next section naming the streaming/chunked index build as the single-box 100M enabler. Honest limits stated plainly: 100M was not run on the box (the batch build still materializes all vectors in RAM) — it awaits the streaming build.

0.30.2 — 2026-07-01

An internal refactor — no behavior, API, on-disk, or wire change; every published crate is functionally identical.

Internal

  • Split quiver-embed's lib.rs into a module tree — the lock-free MVCC serving snapshot (snapshot), the single-writer MVCC write path (mvcc), the on-disk index-snapshot envelopes (envelope), and the multi-vector token-id machinery (multivector) each move to their own module, isolating the lock-free code for review. Purely mechanical: items are re-exported from lib.rs, so the public API and every internal reference are unchanged (verified by an unchanged test suite).

0.30.1 — 2026-07-01

A security and dependency patch — no engine behavior change.

Security

  • Supply-chain hardening — bumped aquasecurity/trivy-action to v0.36.0 (GHSA-69fq-xp46-6x23; the action's tree was briefly compromised before 0.35.0).

Changed

  • Dependency bumps — roaring 0.10 → 0.11 (verified the on-disk .del tombstone-bitmap format round-trips: the crash-recovery gate and segment tests are green), plus ico and criterion (build/dev). GitHub Actions group updated (actions/checkout v7, docker/setup-buildx-action v4). Dependabot is now scoped to the develop branch and configured to skip trait-linked / hardware- validated major bumps (the RustCrypto stack, cudarc, webpki-roots), which are handled by deliberate coordinated upgrades.

0.30.0 — 2026-07-01

Fortified — an audit-remediation and robustness pass. Streaming, memory-bounded segment compaction moved off the checkpoint critical path; lock-free MVCC read correctness (top-k under overlay tombstones) and batch-atomicity fixes; collection-name validation; length-independent constant-time API-key comparison and an HKDF-PRK cache; a release-blocking automated OWASP ZAP DAST gate; and a single canonical cross-language cipher known-answer test gated in CI for Rust, Python, and TypeScript. No on-disk or wire format change; the single node is unchanged at zero overhead.

Testing / CI

  • Single canonical cross-language cipher KAT, gated in CI (F-13). The client ciphers' known-answer vectors (DCPE and the opaque-vector AEAD) now live in one committed file, kat/client-ciphers.json, generated from the Rust reference. The Rust, Python, and TypeScript test suites all load and assert that file (previously each language hardcoded its own copy of the DCPE vector — a drift risk), and CI now runs the Python (pytest) and TypeScript (vitest) SDK suites (they were only built before), so any cross-language cipher drift fails the build.

CI

  • Automated OWASP ZAP DAST gate that blocks a release (ADR-0069). A new reusable dast workflow boots a production-configured live server (encryption-at-rest on, an admin key required, loopback bind), seeds a collection, and runs the OWASP ZAP baseline (spider + passive) and API (active rules over the committed OpenAPI spec, authenticated) scans. A FAIL-level alert (.zap/rules.tsv promotes the injection/disclosure rule classes to FAIL and IGNOREs reviewed benign findings) fails the job; the release job needs it, so a failing scan blocks the release and every package publish — no release until fixed. It also runs on main/develop pushes for early signal (skipped on PRs, which stay on the fast just verify gate). The ZAP reports are uploaded as artifacts on every run.

Security

  • Length-independent constant-time API-key comparison (F-7). The key compare no longer short-circuits on a length mismatch (which leaked the stored key's length via timing); it compares fixed-size SHA-256 digests of both inputs, so timing is independent of input length. The no-early-exit-across-keys property is unchanged. Documented that API keys are shared bearer secrets, plaintext at rest in config/env, and rotated by a zero-downtime add-new → migrate → remove-old config change.
  • Documented the rate limiter's post-authentication scope (F-6). The token- bucket limiter (ADR-0049) is keyed by the authenticated identity and bounded by the configured keys — it does not throttle unauthenticated floods, which is the upstream reverse proxy / WAF's job. Corrected the threat model (per-key rate limiting is shipped, not deferred). No behavior change.

Performance

  • Cache the HKDF PRK in the encryption-at-rest codec (F-8). AeadCodec precomputes the HKDF-extract of the root key once (its output, the PRK, is the same for every page/record) and runs only the per-info HKDF-expand per derive, instead of re-extracting on every page seal/open. Output is byte-identical to the previous full-HKDF derivation (proven by test), so existing encrypted data is unaffected; this removes redundant per-operation work (most noticeable on the small-record WAL path — the 16 KiB page-decrypt still dominates a page read).

Performance

  • MVCC batch upserts coalesce into one overlay republish (ADR-0064). Under lock-free MVCC reads, upsert_batch applied each point to a freshly-cloned, growing overlay and republished per point — O(n·overlay) for a batch of n. It now builds one overlay and publishes it once (O(overlay)), which also makes the batch visible atomically, as a single published snapshot. Single-point upsert behavior is unchanged (it routes through the same batched path with one element). Opt-in MVCC path only.

Changed

  • Collection names are validated at creation (behavior change). create_collection now rejects, with InvalidArgument, an empty name, a name longer than 255 bytes, or a name containing anything outside the documented path-safe charset (ASCII letters, digits, -, _, .) — so a name is always a safe single REST path segment (no /, control characters, whitespace, or non-ASCII). Previously any string was accepted; pathological names now error on create. Enforced in quiver-core, so the embedded API, REST gateway, and MCP server all inherit it.
  • Streaming, memory-bounded segment compaction (ADR-0068). Compaction no longer materialises a collection's whole live set in RAM: a new streaming block-file writer feeds the .vec/.pay columns page-by-page (byte-identical to the old write_blocks, so no format change), and compact_collection streams each row straight from the source segments' mmaps — one row resident at a time instead of ~2× the collection's size. Disk-resident collections now compact without pulling the dataset into RAM. The row directory and (for filterable collections) the secondary index are still assembled in memory — the smaller / opt-in residuals.
  • Compaction bounded off the checkpoint critical path (ADR-0068). A checkpoint now auto-compacts at most one over-threshold collection instead of fanning out across every one, so its added latency is a single collection's streamed merge; the rest amortise across later checkpoints. The atomic manifest swap stays the sole commit point — an interrupted compaction leaves the pre-compaction state intact (new regression test). A fully off-lock background compaction worker is specified and deferred in ADR-0068.

Fixed

  • MVCC snapshot search no longer thins below top-k under overlay tombstones — CollectionSnapshot::search (the lock-free MVCC read path, ADR-0064) asked the base index for exactly k then dropped overlay-tombstoned hits, so deletes/updates shadowing base points (up to the ~20% overlay churn cap) could return fewer than k live results. It now widens the base k/ef by the live fraction — the same compensation the base indexes' own soft-delete paths use — then filters, merges, and truncates to k. Opt-in MVCC path only; the default locked read path is untouched.

Documentation

  • upsert_batch durability contract corrected — the doc previously claimed a crash before the batch's fdatasync leaves none of the batch durable. That is false: WAL recovery is point-in-time and keeps every intact frame up to the first torn one, so an un-acknowledged batch can leave a durable prefix. The comment on Store::upsert_batch (and the Database wrapper) now states standard WAL semantics — acknowledged only after sync() returns; whole-batch retry is safe because upserts are idempotent by external_id. No behavior or on-disk change.

0.29.1 — 2026-06-28

A documentation and tooling patch — no engine change; every published crate is byte-for-byte unchanged.

Documentation

  • Per-subsystem architecture diagrams — 18 Mermaid sources under docs/diagrams/ render to committed SVGs in docs/assets/diagrams/ via just diagrams (mermaid-cli on demand; no new dependency). They cover payload storage, secondary indexes, the audit log, query cost limits, multi-vector MaxSim, BM25, server-side embedding, migration importers, dynamic membership, deferred rebuild, snapshot isolation, gRPC streaming, TLS/mTLS, the envelope key hierarchy, RBAC, observability, MVCC reads, and quantization. A new gallery docs/diagrams.md is linked from the architecture overview, complementing the 31-figure field guide.

CI

  • Visual recap on release — a visual-recap workflow builds a montage of the cockpit screenshots after each release, attaches it plus a gallery markdown as release assets, and appends the gallery to the release notes.

0.29.0 — 2026-06-27

Hardened — a finalization and production-readiness pass: a documented security audit (static review + a dynamic OWASP ZAP scan + the fuzzers re-run), the fixes those probes surfaced, the complete API/CLI/configuration reference, and assorted correctness and coherence fixes. No engine behavior change; the single node is unchanged at zero overhead.

Security

  • Authenticated cluster coordinator — the coordinator's membership API (POST/DELETE /cluster/shards*, /cluster/map, /cluster/health) now enforces the same default-deny RBAC as the data plane: reads need any valid key and the mutating shard ops need an admin key, so a network-reachable coordinator can no longer be reshaped by an unauthenticated caller. A keyless coordinator refuses to start unless insecure (cluster mode only; single-node unaffected).
  • Baseline security response headers — every response now carries X-Content-Type-Options: nosniff and Cross-Origin-Resource-Policy: same-origin (flagged by the OWASP ZAP API scan).
  • Security audit (docs/security/audit-0.29.0.md) — a static OWASP-style review plus a dynamic OWASP ZAP scan of a live server (0 failures across 119 rules) and the three cargo-fuzz targets re-run clean (tens of millions of inputs), with every finding fixed and regression-tested and the residual risks stated honestly. The threat model is extended to cluster mode, the coordinator, and per-shard Raft.

Fixed

  • Crash-recovery gate — a WAL segment too short to hold its 16-byte header (a kill -9 during WAL rotation, beside the prior durable segment) is now treated as an empty torn tail rather than a hard error, so Store::open recovers cleanly. This was the source of the intermittent wal … shorter than its header failure on loaded CI runners; the crash gate (ADR-0005) is honored, not weakened.
  • Field-guide figures 29 & 30 — fixed overlapping labels/arrows.

Documentation

  • Complete API reference — a committed OpenAPI 3.1 spec for every REST endpoint (pinned to the router by a coverage test), and a reconciled docs/api/rest-grpc.md (removed endpoints that were documented but never implemented).
  • CLI reference and an exhaustive configuration reference (every QUIVER_* variable) in the docs site.
  • The "Quiver, Explained" field guide gains §6.5 on the security-robustness evidence (audit + OWASP ZAP + fuzzing), and a systematic bad-input contract test documents that every public entry point rejects malformed input with a 4xx, never a 500.

Internal

  • The quiver demo command seeds a second, text-searchable collection so the cockpit demo exercises every view; community-health files (SUPPORT, CODEOWNERS, Dependabot) and a project wiki round out the repository.

0.28.0 — 2026-06-26

Autopilot — the cluster scales itself and borrows a GPU's muscle where one exists: opt-in automatic scale-out and an optional CUDA distance kernel, both off by default with the single node unchanged at zero overhead.

Added

  • GPU-accelerated batch distance (ADR-0052, behind the off-by-default cuda cargo feature on quiverdb-index). The hot kernel a brute-force / exact scan spends its time in — squared-L2 distance from one query to many vectors — now runs on a CUDA GPU when the feature is on and a device is present (gpu::batch_l2_sq), and falls back to the CPU SIMD kernel otherwise, producing identical results. cudarc dynamically loads the CUDA driver and compiles the kernel with NVRTC at runtime, so a default build links zero CUDA and the feature even compiles without a CUDA toolchain (only running the GPU path needs a device) — the same off-by-default discipline as the raft and otlp features, with the CPU path the always-compiled default and the on-disk format / crash gate untouched. Validated on real hardware: the GPU result is asserted equal to the CPU kernel (the test is a no-op where no GPU is present, so CI builds the feature but only the CPU path is exercised). Wiring it into the planner's exact-scan path is the next step.
  • Coordinator autoscaling — automatic scale-out (ADR-0065 increment 5, opt-in). When enabled, the coordinator samples each shard's point count and, when the busiest crosses a configured high_water_points, grows the cluster into a standby on its own — driving the same safe online migration as a manual POST /cluster/shards/grow, so a busy cluster gains a shard with no operator action and no lost data. It is an explicit policy, not magic: nothing scales without a configured threshold and a standby to grow into, a cooldown bounds the rate, an in-flight migration is never interrupted, and a max_shards cap bounds growth. Configured with an [autoscale] table (or QUIVER_AUTOSCALE_*). Scale-in is deliberately not automated — safe online drain is a later increment, so shrink stays a manual, drained DELETE /cluster/shards/{id}. Single-node and a cluster without the policy are unchanged.

0.27.0 — 2026-06-26

Resilient — surviving a node failure: opt-in per-shard write high availability on an audited Raft core, end to end (durable, log-compacting, elastic), with the single node unchanged at zero overhead.

Added

  • Per-shard Raft state-machine adapter — foundation (ADR-0067, cluster increment 4a). Behind a new off-by-default raft cargo feature on quiverdb-server, the audited openraft consensus core is wired to the engine's apply seam: a committed Raft log entry is one engine write op (WalOp), and the state-machine adapter forwards it through the same apply_replicated path a replication follower already uses (ADR-0030). This increment runs a single-member group to prove the adapter end to end — it commits an op and the engine then serves it; multi-member voting, automatic failover, leader-aware routing, and log compaction land in increments 4b–4d. A default build never links openraft, and single-node / non-Raft clusters are byte-for-byte unchanged. The in-memory Raft log store is vendored from openraft's example memstore (the published openraft-memstore implements the deprecated v1 storage that the storage-v2 API removes); a durable, snapshot-backed store arrives in 4c. Increment 4b adds a real multi-voter group with automatic leader failover (no acknowledged write lost, verified against single-node ground truth) and a gRPC transport (quiver.v1.RaftService) carrying the consensus RPCs between a shard's members over the server's existing tonic stack. The server now runs a per-shard Raft group on its existing gRPC port when configured with a raft_node_id and raft_members (QUIVER_RAFT_NODE_ID / QUIVER_RAFT_MEMBERS): a write is prepared on the leader, proposed through consensus, and committed by a quorum before it is acknowledged, so a leader failover loses no acknowledged write; a write that reaches a non-leader is refused with a "not the leader" redirect (HTTP 421, carrying the leader's URL). An integration test boots a real three-node group, drives writes through the HTTP API, and kills the leader's process to prove automatic failover end to end. The cluster router is now leader-aware: it sends a Raft shard's writes to the shard's current leader, discovering it among the shard's voter URLs ({primary} ∪ replicas), caching it, and re-discovering it on a "not the leader" (HTTP 421) reply or a failover — so a router self-corrects after a leader change without the coordinator on the write path. A non-Raft shard is unchanged (its primary always accepts, so its replicas are never written to). An integration test fronts a three-node Raft shard with a router, kills the leader, and confirms post-failover writes still land and no acknowledged write is lost. The correctness gate asserts the two headline safety properties: a writer driving continuous upserts through the router while the leader's process is killed loses no acknowledged write (every acked id is served by a survivor afterwards), and a minority cannot commit a write (no split-brain) — the latter proven at the consensus-adapter level, where a node can be truly isolated, while it still serves data committed before the partition. The Raft log is now durable (increment 4c): a crash-safe RaftLogStorage persists the granted vote and the log to an append-only file under <data_dir>/raft, fsynced before the call returns and replayed on open (a torn tail from a crash mid-write is discarded), so a restarted voter recovers its log and vote and rejoins safely — the same kill -9 discipline as the engine WAL, proven by a reopen-from-disk test. The committed index stays advisory (recomputed on restart), so the commit path pays no extra fsync. The state-machine snapshot now carries engine state (ADR-0050), so the durable log can be compacted and a far-behind or newly added voter catches up by installing the snapshot instead of replaying a purged log: a snapshot captures the engine as the op stream that recreates it, and installing it resets the target engine and replays — proven both directly and end to end (a fresh voter, added after the leader snapshots and purges its log, catches up purely from the snapshot). A shard's voter set is dynamic: an admin endpoint (POST / DELETE /cluster/raft/voters) adds or removes a voter at runtime — the new node is added as a learner, catches up, and is promoted via a joint-consensus change — so a Raft shard can grow or shrink online without a restart (proven by an end-to-end test that grows a single-member leader to two voters and back). The increment's hardening suite (4d) adds a partition/rejoin gate — a voter cut off from the group misses what the surviving majority commits, then on healing rejoins and catches up with nothing lost — and a property test that replays a long random interleaving of append/truncate/purge/vote and asserts the reopened store equals a reference model, alongside the existing no-lost-acknowledged-write and no-split-brain gates. Raft stays opt-in per shard; a default build still never links openraft.

0.26.0 — 2026-06-25

Elastic — the cluster grows and replicates: per-shard read replicas (increment 2) and dynamic, elastic membership with online rebalancing behind a coordinator (increment 3 — ADR-0066), all opt-in, with single-node unchanged at zero overhead.

Added

  • Cluster online slice migration — data plane (ADR-0066, increment 3c). When a shard joins, the coordinator can add it in a joining state (POST /cluster/shards/joining) and later promote it (POST /cluster/shards/{id}/promote). While a shard is joining, the router dual-writes the migrating slice to both the joining owner and the donor that still serves it, serves searches from the active shards (excluding the joining one, whose donor holds the authoritative slice), and routes gets to the donor — so the slice stays queryable and no write is lost. At promotion the slice's ownership flips atomically (a version bump); a dedup-by-id gather absorbs the brief window where the donor and the promoted shard both hold a slice point. quiver_cluster::ShardMap gains a joining set with add_joining_shard/promote/donor_for/active_shards/ partition_to_donors. An end-to-end test drives a join→copy→flip migration and proves the slice is queryable throughout and every acknowledged write survives the flip.
  • Automated cluster growth (ADR-0066, increment 3c-ii). POST /cluster/shards/grow adds a shard and runs the whole online migration in the background — wait for routers to adopt the joining map (dual-write live), copy the new shard's slice from the donors (paginated scroll, get-if-absent so a concurrent dual-write is never clobbered, provisioning the collection schema on the new shard), promote (flip), then drop the donors' now-stale copies — so an operator grows the cluster with a single call and the slice stays queryable with no lost writes throughout. On any failure the join is reverted. (Single-vector collections; a multivector collection aborts the migration honestly rather than dropping its slice silently.) The point fetch endpoint gains an offset for paginated scroll. Single-node and a static cluster are unaffected.
  • Cluster coordinator + dynamic router refresh (ADR-0066, increment 3b). A new opt-in coordinator mode (QUIVER_COORDINATOR=true) runs a thin, data-plane-free service that owns the authoritative, monotonically versioned shard map and serves a small membership API — GET /cluster/map, POST /cluster/shards, DELETE /cluster/shards/{id}, GET /cluster/health — persisting its map + a never-reused id counter to QUIVER_COORDINATOR_STATE so a restart recovers exactly. A router given QUIVER_COORDINATOR_URL refreshes its shard map from the coordinator on an interval and swaps any newer version into its ArcSwap, so adding or removing a shard propagates with no restart; the coordinator is never a per-query dependency. A router also exposes its currently adopted map read-only at GET /cluster/map. No data migration yet (that is 3c) — a freshly added shard owns only the keys that now hash to it. Single-node and a static cluster are unaffected.

Changed

  • Stable shard ids (ADR-0066, increment-3 groundwork). quiver_cluster::Shard now carries an immutable id: u64 — the HRW key — instead of a positional index: usize, decoupled from the shard's position so a shard can be removed without re-keying (and moving the data of) its survivors. ShardMap keys partition/add_replica by id and gains from_shards for non-contiguous ids (the gap a removed shard leaves); partition now yields (&Shard, group). from_urls is unchanged (ids 0..N), so a static cluster's routing and on-the-wire behaviour are byte-identical and QUIVER_CLUSTER_REPLICAS's <shard_id>=<url> form still targets 0..N. Pure quiver-cluster refactor; single-node unaffected.

Added

  • Cluster read replicas (ADR-0065 increment 2), opt-in via QUIVER_CLUSTER_REPLICAS (a list of <shard_index>=<replica_url> entries). Each shard can now declare one or more read replicas — ordinary leader-follower followers (ADR-0030) of the shard's primary, reused unchanged. Writes, gets and deletes still go to the single primary per shard; searches round-robin across {primary} ∪ replicas to spread read load, falling back to another copy if one is unreachable. Replicas are eventually consistent (replication lag) and refuse direct writes, so a mis-route cannot corrupt one. quiver_cluster::Shard gains replica_urls + read_url/read_order; ShardMap::add_replica attaches them. An end-to-end test boots primaries + followers + a router + a single-node baseline and proves replica-served top-k equals the baseline, a replica refuses writes, and the router tolerates a down replica. Single-node and primary-only shards are unaffected. The "Quiver, Explained" field guide gains §9.9 + a read-replica figure (now 54 pages).

Fixed

  • QUIVER_CLUSTER_SHARDS / QUIVER_CLUSTER_REPLICAS env values are figment array literals and must be bracketed ([http://s1:6333,http://s2:6333]); the .env.example and config docs previously showed an unbracketed comma list, which figment rejects as "expected a sequence" at startup.

0.25.0 — 2026-06-25

Added

  • Cluster mode — sharding + scatter-gather (ADR-0065 increment 1), opt-in via QUIVER_CLUSTER_SHARDS. A non-empty list of shard URLs makes the server a stateless router: it shards writes by point id using rendezvous (HRW) hashing — so adding or removing a shard remaps only ~1/N of ids, the basis for the dynamic, elastic scaling the cluster is designed for — and scatter-gathers searches across all shards, merging the exact global top-k. Each shard is an ordinary single-writer Quiver engine with its own kill -9 crash gate; the cluster is composition, not a rewrite. A new quiver-cluster crate holds the pure primitives (shard map, HRW hashing, merge). An end-to-end test proves a two-shard router returns the same top-k distances as a single-node baseline holding the same data, and that writes shard and routed get/delete work. Single-node stays the zero-overhead default (the router is None unless configured). Later increments: read replicas, online elastic membership + rebalancing + a coordinator, per-shard Raft write-HA, autoscaling hooks.

0.24.0 — 2026-06-25

Added

  • Lock-free MVCC reads (ADR-0064), experimental and default-off behind QUIVER_MVCC_READS. For single-vector, in-memory collections the single writer publishes an immutable CollectionSnapshot (the base index plus a small overlay of writes since the last rebuild) into an arc-swap cell; a reader load()s it and merges base ⊕ overlay without taking any lock, so reads no longer block on a concurrent writer's exclusive lock. Reads served from the snapshot now cover pure-vector, payload/vector, filtered (exact pre-filter and post-filter), and hybrid (dense ⊕ sparse/BM25) — reusing the same store-fetch and RRF logic as the locked path. Durability and the kill -9 crash gate are unchanged — MVCC changes visibility, not durability. Justified by a measured read-during-write contention sweep (docs/benchmarks/results/read-during-write.md): a single concurrent writer of small upserts already retains only ~0.10× of read throughput under the RwLock. At the server, an MVCC-served collection's snapshot cell is cached outside the database lock, so a pure-vector query loads it and searches with no lock at all — it never blocks on a concurrent writer (payload/ filtered/hybrid reads keep the read lock for the store fetch, which is not safe lock-free under a writer, but also serve from the snapshot). Enable with mvcc_reads = true (config) or QUIVER_MVCC_READS=1. A before/after sweep on the same box (docs/benchmarks/results/read-during-write.md) confirms the win: under two small-upsert writers, retained read-QPS goes from 0.00× (RwLock) to 0.79× (MVCC), and from ~0.01× to ~0.67× under four. The flag stays default-off (the proven RwLock path remains the default) until validated on dedicated hardware — absolute QPS is reference-hardware-pending, only the ratio is the honest signal on a shared dev box.
  • Read-during-write contention sweep now measures a grid of write pressure (writer-thread counts × upsert batch sizes), recording the retained-read-QPS ceiling that gates the MVCC build.

[0.23.0] — 2026-06-24

Added

  • Durable on-disk DiskVamana index (ADR-0063): a disk_vamana collection now loads its frugal mmap base on open — mmap the immutable base graph, reconstruct the small in-memory FreshDiskANN delta from the store, and replay the post-checkpoint WAL tail — instead of an O(N) full-RAM rebuild on every restart. The base file is published by atomic rename and a tiny checkpoint blob (base count, tombstones, id map) ties it to the live state; the delta vectors are refetched from the store by id, so the blob stays O(delta ids), not O(N). Any problem falls back to the authoritative rebuild, so the artifact is never load-bearing for correctness and the kill -9 crash gate is preserved by construction. This is what finally lets a running server serve from the PQ-codes-resident path after a restart — the memory-frugality wedge — where earlier releases rebuilt from every full-precision vector on open (the cause of the benchmark's unrepresentative post-build RSS).
  • Cold-reopen-honest memory-frugality evidence: the disk-path wedge benchmark now closes and cold-reopens the server before sampling RSS (#279), so the reported serving memory reflects the durable load path rather than a warm post-build process; a one-command Windows disk-path frugality runner (scripts/bench-disk-frugality.ps1, #275) builds, cold-reopens, and samples on a clean box; and a read-during-write contention sweep (docs/benchmarks/results/read-during-write.md, #281) measures the read-QPS retained under one concurrent writer — the measure-first gate for MVCC. Absolute serving-RAM and QPS stay reference-hardware-pending, never fabricated.
  • Lock-free MVCC reads implementation design (ADR-0064): resolves the tension ADR-0053 left open (indexes are mutated in place, so they cannot simply be shared by Arc) with a per-collection arc-swap snapshot plus a small copy-on-write overlay, published on commit — staged in three increments behind a default-off QUIVER_MVCC_READS flag and gated on the measured contention above. Design only in this release; the read path is unchanged.

Changed

  • QUIVER_DISABLE_DURABLE_DISK_INDEX ops kill switch forces the (always-correct) rebuild-on-open path if the durable load is ever suspected. Durable load is on by default.

Security

  • Addressed code-scanning findings (#282): the TypeScript SDK's unsubscribe-link regex is rewritten to remove a ReDoS vector, and the DCPE "hard-coded cryptographic value" alerts are documented as false-positive/by-design at crates/quiver-crypto/src/dcpe.rs — they are zeroed buffers filled by OsRng, a tag output buffer, a test constant, and one deliberate key-derived deterministic IV, not embedded secrets.

Fixed

  • Bumped vulnerable dependencies flagged by Dependabot (#283): pydantic-settings, langsmith, and the trivy-action CI pin (dev/bench/CI dependencies only; the engine and SDKs are unaffected).

0.22.0 — 2026-06-24

Added

  • Benchmark dimensions (ADR-0061): the v0.22.0 SIFT1M run reports recall at depth 1 / 10 / 100, a saturated-concurrency sweep (1-thread vs 8-thread QPS — up to 1.76× at ef=256), a quantization trade-off (in-memory HNSW vs disk-Vamana + PQ, showing the recall@100 tail collapse), and a filtered selectivity sweep (the planner's pre-filter/post-filter recall valley). Every figure traces to a committed CSV in docs/benchmarks/results/comparison-v0.22.0/; absolute serving-RAM and full-field saturated QPS stay reference-hardware-pending, never fabricated.
  • The "Quiver, Explained" field guide is expanded to v0.22.0 with a whole-system architecture diagram, the off-lock-rebuild timeline, the new benchmark figures, and a cockpit walkthrough; its standalone figures are now committed under docs/assets/explained-figures/.
  • Interactive TUI cockpit (ADR-0060): the retro cockpit gains a query runner (/) — type a query, run a server-side embed-and-search, inspect any result's payload, and recall recent searches — plus a modal keybinding-help overlay (? / F1), a live theme toggle (Ctrl-t, Bronze ↔ Slate), and an ingest-rate sparkline alongside the points trend. Key handling is refactored into a pure, table-tested dispatcher with network I/O pushed to the edge; every screen renders to a buffer and is asserted with ratatui's TestBackend. New committed screenshots (search, help, theme-slate) regenerate from the real render via just tui-shots.
  • OpenTelemetry traces exporter (ADR-0059): opt-in behind the otlp cargo feature and a runtime endpoint (QUIVER_OTLP_ENDPOINT / [otlp] in quiver.toml), exporting the existing #[tracing::instrument] spans over OTLP/gRPC to a collector (Jaeger/Tempo/Grafana). Off by default — no new dependencies in a normal build; the OTLP/gRPC transport reuses the in-tree tonic. A failed exporter degrades to fmt-only rather than failing startup.
  • MCP text tools (ADR-0058): upsert_text and search_text over the MCP server, so an AI agent can store and search documents by text — Quiver embeds them server-side — without running an embedding model itself. Configured with quiver mcp --config <quiver.toml> ([embedding.<collection>] / [rerank.<collection>] tables, the same surface as quiver serve); search_text optionally reranks. This brings the MCP surface to full provider parity with REST, gRPC, and the SDKs.

Changed

  • Index rebuilds run off the exclusive lock (ADR-0062): a write that defers a collection's rebuild no longer stalls concurrent reads. The server serves the prior snapshot while it rebuilds the index off-lock — captured under the shared read lock, built with no lock held, swapped in under a brief write lock — so a rebuild that previously blocked every read for the whole build (measured ~8 s at 20k vectors, ~30 s at 50k, ~77 s at 100k) now keeps reads in the sub-millisecond tail. Server reads are snapshot-isolated and eventually consistent across a rebuild window; embedded &mut searches still rebuild synchronously for read-your-writes. Durability and the crash-recovery gate are unchanged.
  • The embedding/rerank provider seam moved from quiver-server into a new lean quiver-providers crate (ADR-0058) shared by the network and MCP servers; the server re-exports the types, so its public API is unchanged.
  • Crates are now published under the quiverdb-* namespace (ADR-0056): each package is renamed quiverdb-<crate> while its library/extern name stays quiver_<crate> and the binary stays quiver, so source, imports, and cargo install --path are unchanged. This unblocks the (owner-gated) crates.io publish job, since quiver-core / quiver-cli are held by unrelated crates.

0.21.0 — 2026-06-23

Added

  • Packaging & distribution pipeline (ADR-0056): backfilled CHANGELOG.md, crate publish metadata, secret-gated crates.io / PyPI / npm publish jobs, and a Helm chart + Kubernetes manifests under infra/.
  • TypeScript SDK parity with the Python async client: upsertIter (batches a sync or async iterable), scroll (an async generator for export / re-embedding), and deleteByFilter (paged erasure).
  • Go SDK bulk/maintenance helpers: UpsertBatch (batched upload), Scroll (page through a collection via a callback), and DeleteByFilter (paged erasure) — all context-aware, standard-library only.

Changed

  • Concurrent reads (ADR-0057): the server serves searches behind a reader–writer lock instead of a single mutex, so reads run in parallel. The engine gains &self search_snapshot / hybrid_search_snapshot / search_multi_vector_snapshot reads and ensure_indexed; the single-writer model, durability, and crash gate are unchanged. Fully lock-free arc-swap snapshots are the staged successor.

Fixed

  • Go SDK Fetch parsed the wrong response envelope (matches instead of the points the fetch endpoint returns), so it never returned points; now fixed, with a regression test.

0.20.1 — 2026-06-23

Changed

  • Re-ran the multi-DB benchmark on the v0.20.0 engine with the bulk-ingest build path so the build column is an honest time-until-queryable (ADR-0055); folded SIFT1M + GIST1M results across seven competitors into the README, docs, and the "Quiver, Explained" field guide.

Fixed

  • "Quiver, Explained" PDF layout: figure overlaps and the ColBERT callout page break.

0.20.0 — 2026-06-23

Added

  • Online snapshot & restore — consistent whole-directory copy over REST and the MCP server (ADR-0050).
  • Client-streaming gRPC UpsertStream (ADR-0045 fast-follow).
  • Real Prometheus /metrics (counters + latency histogram + security counters), request tracing spans, and an importable Grafana dashboard (ADR-0054).
  • Per-key rate limiting — token bucket with RateLimit-* headers and 429 (ADR-0049).
  • Server-side embedding & reranking hooks, provider-agnostic and opt-in per collection (ADR-0047); BM25 with a Snowball (Porter2) stemmer (ADR-0048).
  • A standard-library Go SDK mirroring the REST surface; snapshot() parity in the Python and TypeScript SDKs.
  • Design-only ADRs for the big bets: distributed/sharded mode (0051), GPU acceleration (0052), and lock-free MVCC reads (0053).

0.19.0 — 2026-06-22

Added

  • Hybrid (dense + sparse) search on every surface — gRPC, MCP, and the TypeScript SDK join REST and Python — backed by a derived sparse inverted index; bulk ingest via POST …/points:bulk (ADR-0045).

Fixed

  • De-noised CI by isolating and retrying the virtualization-sensitive crash-recovery test.

0.18.1 — 2026-06-19

Fixed

  • Automated, tag-triggered Windows release asset so quiver update resolves on Windows; unified multi-platform release packaging (ADR-0044).

0.18.0 — 2026-06-19

Added

  • Query cost limits enforced at the op layer (caps on k, ef_search, dimension, payload, batch), closing an authenticated-DoS vector (ADR-0040).
  • Deep, large-data benchmark dimensions with real SIFT1M and GIST1M multi-DB comparisons and a Docker Milvus-server adapter (ADR-0041).
  • RAG/agentic ergonomics — async Python client, batched upsert / scroll / delete-by-filter, a Haystack DocumentStore, a rerank helper, and an MCP collection_info tool (ADR-0042).
  • Hybrid (dense + sparse) search with RRF fusion over the engine, REST, and the Python SDK (ADR-0043).

0.17.2 — 2026-06-19

Fixed

  • Windows install/update hotfix (fsync_dir behavior) and CDN stale-asset cache bypass.

0.17.1 — 2026-06-19

Changed

  • Verdigris "V" arrowhead in all terminal banners matching the TUI logo; regenerated cockpit screenshots.

0.17.0 — 2026-06-18

Added

  • One-command install (install.sh / install.ps1) and a self-updating quiver update subcommand with checksum verification (ADR-0039).
  • Scientific multi-DB benchmark suite (ADR-0037).

Changed

  • 35× faster upsert build time via batched WAL sync (ADR-0038).

0.16.0 — 2026-06-18

Added

  • The retro cockpit (ADR-0036): a coherent Bronze Quiver brand, a logo whose "V" is a 3-D arrowhead, a vocabulary of retro decorations, a render-to-buffer view API, and reproducible committed screenshots (just tui-shots).

0.15.0 — 2026-06-17

Added

  • mdBook documentation site under apps/docs; a native TypeScript DcpeCipher closing the last DCPE SDK gap (ADR-0035).

Changed

  • DCPE Scale-And-Perturb hardening as a breaking cipher v2 — key-derived component shuffle (an exact L2 isometry) plus an optional ordering-preserving affine normalisation; full per-axis whitening is documented as incompatible with searchable encryption and deliberately not offered (ADR-0035).

0.14.0 — 2026-06-17

Added

  • Multi-vector / ColBERT follow-ups (ADR-0034): incremental multi-vector index maintenance and an opt-in ColBERTv2 residual-compression index with PLAID centroid pruning, creatable across REST/gRPC/MCP and the SDKs. Native variable-stride document rows were deferred honestly, gated on a measured locality win.

0.13.0 — 2026-06-17

Added

  • Graph-index incremental updates — Vamana and DiskVamana adopt FreshDiskANN's StreamingMerge model (in-memory delta + O(1) tombstones + churn-threshold consolidation), the last index family that previously rebuilt on every write (ADR-0033).

0.12.0 — 2026-06-16

Fixed

  • Install honesty (the quiver-cli crates.io name is an unrelated crate → install from source), static README badges that render everywhere, and UTF-8 mojibake in the status banner. No functional change.

0.11.0 — 2026-06-16

Added

  • Semantically secure client-side vector encryption — vector_encryption = client_side seals each vector with XChaCha20-Poly1305 and stores an opaque blob, the server does no ranking, and retrieval is a client-side fetch-and-rank (ADR-0032). The flag migrated from encrypted_vectors to a three-valued vector_encryption enum (byte-compatible on disk).

0.10.0 — 2026-06-15

Added

  • Experimental DCPE vector-encryption mode — the published Scale-And-Perturb distance-comparison-preserving scheme, opt-in and off by default, with honest limits (L2-only, not semantically secure) (ADR-0031).

0.9.0 — 2026-06-15

Added

  • Asynchronous leader-follower replication — a follower applies the leader's committed op log and serves reads while refusing writes; failover and consensus are explicit non-goals (ADR-0030).

0.8.0 — 2026-06-15

Added

  • Live Chroma and Postgres migration connectors, completing one-command live migration from Qdrant, Chroma, and pgvector (ADR-0029).

0.7.0 — 2026-06-15

Added

  • Multi-vector / late-interaction (ColBERT) retrieval — multivector collections ranked by MaxSim over the existing row-addressed store, reachable from every surface (ADR-0028).

0.6.0 — 2026-06-15

Added

  • Durable on-disk incremental IVF index recovered by snapshot + WAL-tail replay under the manifest's atomic swap (ADR-0025); a live Qdrant migration connector over HTTP (ADR-0027).

0.5.0 — 2026-06-15

Added

  • HNSW incremental delete (O(1) soft-delete + amortized rebuild, ADR-0026); neighbor-bounded IVF reassignment (ADR-0023); a unified secure database-open path shared by the server, MCP, and CLI.

0.4.0 — 2026-06-14

Added

  • Incremental in-place IVF index updates (SpFresh/LIRE, in-memory so the crash gate is untouched, ADR-0023); migration importers for Qdrant / Chroma / pgvector exports (ADR-0024).

0.3.0 — 2026-06-14

Added

  • Security depth — client-side payload encryption with a documented trust boundary, RBAC + scoped API keys + optional mTLS, an append-only audit log, per-collection-DEK envelope encryption with crypto-shredding, and the cockpit constellation view; cargo-fuzz targets for the wire and on-disk parsers.

0.2.0 — 2026-06-14

Added

  • Memory frugality — disk-resident graph (DiskANN/Vamana) + IVF, quantization (PQ / scalar / binary with hamming pre-filter + exact re-rank), per-collection recall/latency/memory knobs, hybrid filtered search, the TypeScript SDK, the MCP server, and the LangChain / LlamaIndex adapters.

0.1.0 — 2026-06-13

Added

  • Single-node core — encrypted collection → upsert → filtered k-NN → live TUI from one binary; storage engine (segments + WAL + crash recovery + checksums); HNSW; SIMD kernels; REST + gRPC; encryption-at-rest by default; TLS via rustls; the TUI MVP; the benchmark harness with first SIFT1M numbers; the Python SDK.