Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 47 additions & 4 deletions docs/en/guides/01-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1386,7 +1386,7 @@ Vector database storage configuration

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `backend` | str | VectorDB backend type: 'local' (file-based), 'http' (remote service), 'volcengine' (cloud VikingDB), 'vikingdb' (private deployment), or 'cuvs' (local storage + GPU dense search) | "local" |
| `backend` | str | VectorDB backend type: 'local' (file-based), 'http' (remote service), 'volcengine' (cloud VikingDB), 'vikingdb' (private deployment), 'qdrant' (REST), or 'cuvs' (local storage + GPU dense search) | "local" |
| `name` | str | VectorDB collection name | "context" |
| `url` | str | Remote service URL for 'http' type (e.g., 'http://localhost:5000') | null |
| `project_name` | str | Project name (alias project) | "default" |
Expand All @@ -1395,6 +1395,7 @@ Vector database storage configuration
| `sparse_weight` | float | Sparse weight for hybrid vector search, only effective when using hybrid index | 0.0 |
| `volcengine` | object | 'volcengine' type VikingDB configuration | - |
| `vikingdb` | object | 'vikingdb' type private deployment configuration | - |
| `qdrant` | object | Qdrant REST URL, API key, timeout, named vector names, and optional metadata collection name | - |
| `cuvs` | object | NVIDIA cuVS configuration for the 'cuvs' backend and the opt-in memory-aware auto mode on 'local'; see the [cuVS guide](./16-cuvs.md) | - |

Default local mode
Expand Down Expand Up @@ -1440,9 +1441,52 @@ acl_inherited_grants

Each element uses `{mask}:{principal}`: `1` means `read`, `3` means `write`, and `7` means `manage`.

Local backends add the fields to an existing collection and rebuild the scalar index during startup. Existing records are not rewritten; missing ACL fields read as `acl_enabled=false` and empty lists.
Local, cuVS, and Qdrant backends add the fields to an existing collection and rebuild/update the scalar index during startup. Existing records are not rewritten; missing ACL fields read as `acl_enabled=false` and empty lists.

For existing remote collections, including Volcengine VikingDB, provision these fields and scalar indexes before startup; OpenViking validates but does not alter the remote schema. Volcengine API-key data-plane mode also requires the context collection and configured index to exist. See [Resource Access Control (ACL)](../concepts/15-acl.md) for permission semantics.
For other existing remote collections, including Volcengine VikingDB, provision these fields and scalar indexes before startup; OpenViking validates but does not alter the remote schema. Volcengine API-key data-plane mode also requires the context collection and configured index to exist. See [Resource Access Control (ACL)](../concepts/15-acl.md) for permission semantics.

<details>
<summary><b>Qdrant REST</b></summary>

Qdrant uses the standard-library REST transport; no `qdrant-client` dependency is
required. Set `sparse_weight` to `0` for dense-only mode, or to a value in
`(0, 1]` to enable named sparse vectors and client-side weighted RRF hybrid
search:

```json
{
"storage": {
"vectordb": {
"backend": "qdrant",
"url": "http://127.0.0.1:6333",
"project": "default",
"name": "context",
"dimension": 1536,
"sparse_weight": 0.5,
"qdrant": {
"api_key": "optional-key",
"timeout_seconds": 10,
"dense_vector_name": "vector",
"sparse_vector_name": "sparse_vector"
}
}
}
}
```

OpenViking stores a metadata marker and sparse term dictionary in a deterministic
sidecar collection. Existing Qdrant collections without that marker fail closed
instead of being adopted. URI scope metadata and account/tag filters are retained;
`Contains` and server-side content grep are unsupported, so grep uses the
filesystem fallback (`USE_CONTENT_FIELD=False`).

For live coverage, set `QDRANT_URL` and optionally `QDRANT_API_KEY`, then run:

```bash
QDRANT_URL=http://127.0.0.1:6333 \
pytest --confcutdir=tests/storage -q tests/storage/test_qdrant_integration.py
```
</details>

<details>
<summary><b>openGauss</b></summary>
Expand Down Expand Up @@ -1477,7 +1521,6 @@ In the official container, the initial `omm` user may be restricted for remote l
Set `mode` to `"distributed"` for openGauss distributed deployments; OpenViking will attempt to mark metadata tables as reference tables and distribute collection tables by `id`.
</details>


## Config Files

OpenViking uses two config files:
Expand Down
211 changes: 211 additions & 0 deletions docs/superpowers/plans/2026-08-20-qdrant-integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
# Qdrant Integration Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add phase 1 dense and phase 2 sparse Qdrant support behind the current OpenViking `CollectionAdapter` contract.

**Architecture:** Keep all Qdrant-specific behavior in a REST collection implementation and a thin adapter. Store path metadata and stable IDs in the data collection, and store the OpenViking marker plus sparse term dictionary in a deterministic sidecar collection. Reuse the existing Filter AST and tenant-aware upper layer without changing its API.

**Tech Stack:** Python standard library (`urllib`, `json`, `hashlib`, `uuid`, `threading`), OpenViking `Collection`/`ICollection`, Qdrant REST API, pytest.

**Spec:** `docs/superpowers/specs/2026-08-20-qdrant-integration-design.md`

## Global Constraints

- Do not modify or reset the user's pre-existing `.gitignore` change.
- Do not reintroduce `qdrant-client` or use lossy sparse hashing.
- Do not silently drop filters or sparse vectors.
- Preserve `CollectionAdapter` and upper-layer search APIs.
- Every production behavior is introduced by a failing test first.
- Keep `USE_CONTENT_FIELD=False`.

### Task 1: Add failing pure conversion and filter tests

**Files:**
- Create: `tests/storage/test_qdrant_adapter.py`

**Interfaces:**
- Tests will import `QdrantCollectionAdapter`, `QdrantCollection`,
`compile_qdrant_filter`, `build_qdrant_payload`, `to_qdrant_point_id`,
and `SparseTermDictionary`.

- [x] **Step 1: Write failing tests**

Cover:

```python
def test_path_payload_includes_self_and_ancestors(): ...
def test_path_scope_depth_mapping_is_segment_aware(): ...
def test_multi_tag_eq_is_qdrant_must_and_in_is_match_any(): ...
def test_account_filter_is_preserved(): ...
def test_point_id_is_deterministic_and_original_id_round_trips(): ...
def test_sparse_term_collision_raises_instead_of_merging(): ...
```

- [x] **Step 2: Run the focused tests**

Run: `uv run --project . pytest -q tests/storage/test_qdrant_adapter.py`

Expected: collection/import failure because the Qdrant implementation does not
exist yet. If dependency installation is unavailable, run the pure module
through the repository's available Python environment and record the blocker.

### Task 2: Implement Qdrant REST transport and collection lifecycle

**Files:**
- Create: `openviking/storage/vectordb/collection/qdrant_rest.py`
- Create: `openviking/storage/vectordb/collection/qdrant_collection.py`
- Modify: `openviking/storage/vectordb/collection/__init__.py`

**Interfaces:**
- `QdrantRestClient.request(method, path, body=None, params=None) -> dict`
- `QdrantCollection(ICollection)`
- `QdrantCollection.create_remote_collection(schema, sparse_enabled)`
- `QdrantCollection.has_openviking_metadata() -> bool`

- [x] **Step 1: Add the minimal HTTP client**

Use `urllib.request`, JSON, optional `api-key`, bounded error bodies, and a
single configurable timeout. Keep the transport injectable for tests.

- [x] **Step 2: Implement collection existence and metadata**

Implement Qdrant collection discovery, create/delete, metadata sidecar creation,
schema marker validation, and index creation.

- [x] **Step 3: Implement data methods**

Implement `upsert_data`, `update_data`, `fetch_data`, `delete_data`,
`delete_all_data`, `aggregate_data`, and `get_meta_data` with the existing
`SearchResult`, `DataItem`, and `AggregateResult` dataclasses.

- [x] **Step 4: Run transport/lifecycle tests**

Run: `uv run --project . pytest -q tests/storage/test_qdrant_adapter.py -k 'transport or lifecycle'`

### Task 3: Implement phase 1 adapter and filter mapping

**Files:**
- Create: `openviking/storage/vectordb_adapters/qdrant_adapter.py`
- Modify: `openviking/storage/vectordb_adapters/factory.py`
- Modify: `openviking_cli/utils/config/vectordb_config.py`

**Interfaces:**
- `QdrantCollectionAdapter.from_config(config)`
- `compile_qdrant_filter(expr)`
- `build_qdrant_payload(record)`
- `to_qdrant_point_id(value)`

- [x] **Step 1: Implement deterministic IDs and path payloads**

Normalize `viking://` URIs to `/...`, include self and ancestors in
`scope_roots`, and reject malformed path scope input.

- [x] **Step 2: Implement AST mapping**

Map `And`, `Or`, `Eq`, `In`, `Range`, `TimeRange`, and `PathScope` to Qdrant
filters. Reject unsupported `Contains` with `NotImplementedError` rather than
changing its substring semantics.

- [x] **Step 3: Implement adapter factory/config wiring**

Add standard backend name `qdrant`, a nested config object for URL/auth/vector
names/timeout, and registry wiring. Preserve dotted custom class paths.

- [x] **Step 4: Run phase 1 tests**

Run: `uv run --project . pytest -q tests/storage/test_qdrant_adapter.py`

### Task 4: Implement phase 2 durable sparse term dictionary

**Files:**
- Modify: `openviking/storage/vectordb/collection/qdrant_collection.py`
- Modify: `openviking/storage/vectordb_adapters/qdrant_adapter.py`
- Modify: `tests/storage/test_qdrant_adapter.py`

**Interfaces:**
- `SparseTermDictionary.index_for(term) -> int`
- `SparseTermDictionary.encode(vector) -> dict[str, list[...]]`
- `QdrantCollection.encode_sparse_vector(vector)`

- [x] **Step 1: Add failing sparse tests**

Verify stable term indices, sidecar persistence, dense+sparse conversion, and
collision rejection.

- [x] **Step 2: Implement deterministic collision-checked Qdrant-compatible indices**

Derive a positive uint32 candidate from SHA-256 because Qdrant sparse indices
are uint32 at the REST/protobuf boundary. Persist each term/index pair in the
sidecar collection. Query existing records for the candidate index; raise a
collision error when a different term owns it.

- [x] **Step 3: Wire sparse upsert and search**

Convert OpenViking `{term: weight}` dictionaries to named Qdrant sparse vectors,
and reject sparse operations when the collection was created dense-only.

- [x] **Step 4: Run phase 2 tests**

Run: `uv run --project . pytest -q tests/storage/test_qdrant_adapter.py -k sparse`

### Task 5: Add optional live Qdrant integration coverage

**Files:**
- Create: `tests/storage/test_qdrant_integration.py`
- Modify: `tests/README.md`

- [x] **Step 1: Write environment-gated tests**

Use `QDRANT_URL` and optional `QDRANT_API_KEY`; skip cleanly without a live
server. Exercise create, upsert, path/tag filters, count, sparse search, and
drop.

- [x] **Step 2: Run the integration tests when available**

Run: `QDRANT_URL=http://127.0.0.1:6333 uv run --project . pytest -q tests/storage/test_qdrant_integration.py`

### Task 6: Verify, document, and commit

**Files:**
- Modify: `openviking/storage/vectordb_adapters/README.md`
- Modify: `docs/en/guides/01-configuration.md`
- Modify: `docs/zh/guides/01-configuration.md`

- [x] **Step 1: Add configuration and capability docs**

Document Qdrant URL/auth, vector names, dense-only versus sparse-enabled mode,
metadata sidecar behavior, and the explicit unsupported-filter policy.

- [x] **Step 2: Run final verification**

Run:

```bash
uv run --project . pytest -q tests/storage/test_qdrant_adapter.py
uv run --project . pytest -q tests/storage tests/unit
uv run --project . ruff check openviking/storage/vectordb openviking/storage/vectordb_adapters tests/storage/test_qdrant_adapter.py
git diff --check
```

- [x] **Step 3: Review the diff and commit**

```bash
git status --short
git diff --stat
git commit -am "feat: add qdrant vector backend"
```

## Verification notes (2026-08-21)

- Focused adapter coverage: `45 passed`; transport/lifecycle: `2 passed`;
sparse: `9 passed`.
- The reconstructed pre-implementation test run on `main` failed at collection
with the expected missing Qdrant module import.
- Disposable live Qdrant coverage passed: `1 passed` against a local Qdrant
container at `http://127.0.0.1:6333`; the container was removed afterward.
- The integration test path is `tests/storage/test_qdrant_integration.py`.
- The repository-wide `tests/storage tests/unit` run remains red: 19 known
failures reproduce on `main`, with additional environment/order-dependent
failures still unclassified. This is recorded as a repository-health
boundary, not a Qdrant implementation blocker.
100 changes: 100 additions & 0 deletions docs/superpowers/specs/2026-08-20-qdrant-integration-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# OpenViking Qdrant Integration Design

## Goal

Add a first-party Qdrant vector backend that preserves the current
`CollectionAdapter` contract, OpenViking path/tag/tenant semantics, and
hybrid sparse-vector correctness without reverting the removed implementation.

## Scope

### Phase 1

- Qdrant REST transport implemented with the Python standard library.
- Collection create/load/drop and index lifecycle.
- Dense vector upsert, fetch, delete, count, scalar ordering, and search.
- OpenViking filter AST mapping:
- `And`, `Or`, `Eq`, `In`, `Range`, `TimeRange`, `PathScope`.
- `Contains` is rejected unless a caller uses a supported Qdrant text field
explicitly; the adapter never silently changes substring semantics.
- Segment-aware path scopes using `uri`, `uri_depth`, and `scope_roots`.
- Exact multi-tag AND and `In` semantics for `search_tags`.
- Account isolation through the existing `_SingleAccountBackend` filter path.
- Deterministic UUID point IDs with the original OpenViking ID in payload.
- Durable OpenViking metadata marker; existing unmarked collections fail closed.
- `USE_CONTENT_FIELD = False`; grep remains filesystem-backed.

### Phase 2

- Named Qdrant sparse vector support.
- Durable sparse-term dictionary stored in a sidecar Qdrant collection.
- Stable SHA-256-derived positive uint32 term IDs (the Qdrant REST/protobuf
sparse-index limit) with collision detection and an explicit error on
collision; no lossy hash merging.
- Dense-only configurations continue to work without creating sparse metadata.
- A non-zero sparse query against a backend configured without sparse support
fails explicitly.

## Non-goals

- No revert of PR #3872.
- No Qdrant Python SDK dependency; REST is sufficient for the contract.
- No migration or implicit takeover of existing unmarked Qdrant collections.
- No server-side full-text content index in Qdrant.
- No change to upper-layer search APIs or Filter AST.
- No distributed metadata transaction protocol beyond Qdrant point-level
idempotence and collision detection.

## Data model

The data collection stores:

- Dense vector under a configurable named vector (default `vector`).
- Sparse vector under a configurable named vector (default `sparse_vector`).
- OpenViking fields from the collection schema.
- `uri_depth` as an integer payload.
- `scope_roots` as an array of exact normalized path strings.
- `_openviking_original_id` as the original string ID.

The metadata collection stores:

- A fixed OpenViking schema marker point.
- One point per sparse term containing `term` and `index`.
- The deterministic point ID is derived from the term; a matching index with a
different term is treated as a collision and raises.

## Filter mapping

Qdrant `must`/`should` clauses carry the OpenViking AST structure:

| OpenViking expression | Qdrant representation |
| --- | --- |
| `And` | `must` |
| `Or` | `should` |
| `Eq` | `match.value` |
| `In` | `match.any` for multiple values |
| `Range` / `TimeRange` | `range` |
| `PathScope(depth=0)` | exact `uri` match |
| `PathScope(depth<0)` | `scope_roots` exact element match |
| finite `PathScope` | `scope_roots` match plus `uri_depth` upper bound |

Multiple `Eq("search_tags", ...)` expressions remain separate `must` clauses,
so tag filters are AND rather than OR.

## Error handling

- HTTP failures become a backend-specific `QdrantError` carrying method, path,
status, and a bounded response body.
- Missing OpenViking metadata refuses collection loading.
- Unsupported filters and sparse requests fail before sending a lossy query.
- Invalid vector dimensions and malformed sparse values fail at conversion.
- Qdrant response shape mismatches fail loudly rather than returning empty
results.

## Testing

- Pure unit tests for path payload construction and filter compilation.
- Pure unit tests for deterministic IDs and sparse term collision handling.
- Fake-transport tests for collection CRUD, search, count, and index creation.
- Optional integration tests enabled by `QDRANT_URL`, skipped otherwise.
- Existing OpenViking unit tests remain unchanged.
Loading