Release Notes: 1.0.0
This note describes the 1.0.0 release relative to the 1.0.0-beta baseline.
Current diff scope at the time of writing:
- 78 commits ahead of 1.0.0-beta
- 47 files changed
- 8,710 insertions and 3,686 deletions
Executive Summary
1.0.0 is not a small follow-up to 1.0.0-beta. The current branch is a substantial platform update focused on sparse retrieval, filtered search, backup operations, observability, and build/runtime ergonomics.
The largest practical change is the sparse subsystem rewrite: the earlier implementation has been replaced with a dedicated MDBX-backed inverted index with explicit on-disk version checks, configurable search batching, and better compaction behavior. Around that, the code now adds safer backup workflows, more tunable filtering, stronger operational logging, and a cleaner build/test story.
Highlights
1. Sparse search has been rebuilt
- The sparse engine moved from the older bmw.hpp path to a dedicated inverted_index implementation.
- Sparse postings are now stored as MDBX-backed blocked posting lists with per-term metadata and a superblock for format validation.
- Search batching is runtime configurable through NDD_INV_IDX_SEARCH_BATCH_SZ.
- The sparse path now supports optional float storage via NDD_INV_IDX_STORE_FLOATS, while the default path keeps compact quantized storage.
- The current implementation also documents and enforces sparse format compatibility instead of silently opening legacy data.
Why it matters:
- Better control over sparse search cost.
- Clearer recovery behavior for incompatible on-disk layouts.
- A storage layout that is easier to reason about and instrument.
2. Filtered and hybrid search are more capable
- Filtering is now documented and structured around numeric, category, and boolean paths in filter.md.
- Search requests now accept filter_params, including prefilter_threshold and boost_percentage, so filtered search can be tuned per request.
- JSON vector inserts now support meta, filter, and norm in addition to dense and sparse vector payloads.
- Filter tests were added and can be built with -DENABLE_TESTING=ON.
Why it matters:
- More predictable filtered retrieval behavior.
- Better support for hybrid dense+sparse workloads.
- A clearer path for validating filter regressions during release testing.
3. Backups are now async, user-scoped, and safer operationally
- Backup creation now runs asynchronously and returns 202 Accepted instead of blocking the request until archive creation finishes.
- Backup storage moved to per-user directories under {DATA_DIR}/backups/{username}/.
- Archive creation now stages in a temp directory and renames into place atomically.
- New backup endpoints were added for download, upload, active-status checks, and metadata inspection.
- Backup upload support exists, but the current multipart path still buffers the uploaded archive in memory before writing it to disk.
- Backup implementation details are documented in backup-system.md.
New or expanded backup API surface:
- POST /api/v1/index/{name}/backup
- GET /api/v1/backups
- GET /api/v1/backups/active
- GET /api/v1/backups/{name}/info
- GET /api/v1/backups/{name}/download
- POST /api/v1/backups/upload
- POST /api/v1/backups/{name}/restore
- DELETE /api/v1/backups/{name}
Why it matters:
- Long-running backup work no longer ties up the request thread.
- Backup lifecycle is visible to clients.
- Temp-file cleanup and atomic rename behavior reduce partial-backup risk.
4. Runtime behavior and observability improved
- Logs were standardized around LOG_INFO, LOG_WARN, and LOG_ERROR, with stable numeric codes and explicit username/index_name context in logs.md.
- MDBX timing instrumentation was added behind ND_MDBX_INSTRUMENT; sparse timing instrumentation is also available behind ND_SPARSE_INSTRUMENT.
- Server thread count is now runtime configurable through NDD_NUM_SERVER_THREADS.
- Vector cache controls were added through NDD_VECTOR_CACHE_PERCENTAGE and NDD_VECTOR_CACHE_MIN_BITS.
- The HNSW path picked up graph-backfill and cache-related work intended to improve search/load behavior on active indexes.
Why it matters:
- Operators get much better failure context in production logs.
- Performance investigations can be done without carrying permanent instrumentation overhead.
- More runtime knobs are available without recompiling.
5. Build and developer workflow are cleaner
- Sparse sources are compiled into a separate object library, which improves parallel compilation behavior.
- ENABLE_TESTING=ON now builds a dedicated ndd_filter_test target.
- run.sh was added to auto-detect the built binary and run with NDD_DATA_DIR and optional auth token.
- install.sh was expanded with clearer OS/package handling and frontend asset setup.
- README and subsystem docs were significantly expanded.
Operator-Facing Changes
| Area | What changed now | Why teams should care |
|---|---|---|
| Sparse storage | Sparse indexes now use a superblock-validated on-disk format | Older sparse data may not open without rebuild/reingest |
| Backups | Archive format is now .tar in per-user backup directories | Existing tooling that expected global .tar.gz archives needs updating |
| Create-index defaults | Default precision now resolves to int16 unless explicitly provided | Memory/latency characteristics may differ from beta defaults |
| Search tuning | filter_params.prefilter_threshold and boost_percentage are supported | Filter-heavy queries can be tuned without code changes |
| Threading | Server concurrency is runtime-configurable with NDD_NUM_SERVER_THREADS | Default runtime behavior may differ from earlier deployments |
| Cache tuning | Vector cache sizing can be adjusted by env var | Memory usage can be tuned per deployment |
| Logs | Log format is now standardized and code-based | Easier alerting, parsing, and support workflows |
Upgrade Notes
Sparse index compatibility
The current sparse implementation explicitly checks for a sparse superblock and format version. If an older sparse database exists without that metadata, the current code will reject it as incompatible rather than attempting a best-effort open.
Practical guidance:
- Plan to rebuild or re-ingest sparse indexes created before the new superblock format.
- Treat sparse on-disk compatibility as a release note item, not an implementation detail.
Backup format and layout changes
1.0.0-beta used global .tar.gz backup archives. The current code expects per-user .tar archives and adds temp staging plus active-job tracking.
Practical guidance:
- Review any scripts that assume .tar.gz.
- Review any scripts that assume a single global backup directory.
- Validate restore behavior against real beta-era archives before calling the upgrade seamless.
API behavior changes
- Backup creation is now asynchronous and returns a progress state instead of a finished archive immediately.
- Backup restore now returns 201 Created on success.
- Backup delete now returns 204.
- Download uses the backup file directly from disk and currently expects .tar.
- Search can take request-level filter tuning through filter_params.
- JSON inserts accept meta, filter, and norm.
Default and tuning changes worth checking in staging
- Default create-index precision changed from the earlier INT8 default path to int16.
- int8d and int16d legacy names are normalized to int8 and int16.
- Default filter prefilter threshold moved from 1000 to 10000.
- Default server threading now comes from runtime settings; when NDD_NUM_SERVER_THREADS is unset, the code resolves it to roughly 2 x hardware_concurrency().
Suggested Release Positioning
This release should be positioned as:
A storage and operations upgrade over 1.0.0-beta, with the headline changes being the new sparse engine, safer backup lifecycle, better filtered retrieval controls, and stronger production observability.