Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions paradedb-demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,11 @@ Exercises:
- `vectorColumn(3)` — native pgvector `vector(3)` column with `number[]` ⇄ `'[x,y,z]'` codec.
- `paradeDbAll(keyCol)` + `paradeDbL2Distance(vecCol, query)` — vector Top-K (`@@@ pdb.all()` predicate, `<->` ORDER BY, LIMIT).
- `CREATE EXTENSION pg_search` / `vector` via the docker init scripts (`init/*.sql`).
- Automatic `CREATE INDEX ... USING bm25 (...) WITH (key_field='...')` via upstream's index-type registry.
- Automatic `CREATE INDEX ... USING paradedb (...) WITH (key_field='...')` via upstream's index-type registry.

The bm25 index covers the text columns only: released pg_search builds reject vector columns in bm25 indexes (needs branch `mvp/vector-search`, paradedb/paradedb#5685), so vector Top-K runs unaccelerated here. The gated integration test in `test/vector.integration.test.ts` exercises the vector-in-bm25 index (via `renderBm25IndexDdl`) and skips with a clear reason on released builds.
Requires a pg_search 0.25.0+ server, which registers the `paradedb` index access method; the docker image pin may lag behind, in which case `pnpm db:init` fails until the image catches up.

The ParadeDB index covers the text columns only: vector columns in ParadeDB indexes also need pg_search 0.25.0+ (paradedb/paradedb#5685), so on older builds vector Top-K runs unaccelerated. The gated integration test in `test/vector.integration.test.ts` exercises the vector-in-index DDL (via `renderParadeDbIndexDdl`) and skips with a clear reason when the server lacks support.

## Run it

Expand All @@ -35,7 +37,7 @@ pnpm start -- vector '0.1,0.9,0.1' 3
pnpm test
```

`pnpm db:init` produces the BM25 index directly from the `constraints.index([...], { type: 'bm25', options: { key_field: 'id' } })` declaration in `prisma/contract.ts`.
`pnpm db:init` produces the ParadeDB index directly from the `constraints.index([...], { type: 'paradedb', options: { key_field: 'id' } })` declaration in `prisma/contract.ts`.

Teardown:

Expand Down
10 changes: 5 additions & 5 deletions paradedb-demo/prisma/contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@ export const contract = defineContract(
Item: Item.sql(({ cols, constraints }) => ({
table: 'item',
indexes: [
// Listing `cols.embedding` here requires pg_search with vector-in-bm25
// support (branch mvp/vector-search); released builds reject vector
// columns in bm25 indexes, so the demo indexes text fields only and
// vector Top-K falls back to an unaccelerated sort.
// Listing `cols.embedding` here requires vector-in-index support
// (pg_search 0.25.0+ has it, but the docker image may lag), so the
// demo indexes text fields only and vector Top-K falls back to an
// unaccelerated sort.
constraints.index([cols.category, cols.description, cols.id, cols.rating], {
type: 'bm25',
type: 'paradedb',
options: { key_field: 'id' },
name: 'item_bm25_idx',
}),
Expand Down
4 changes: 2 additions & 2 deletions paradedb-demo/src/prisma/contract.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ import type {
} from '@prisma-next/contract/types';

export type StorageHash =
StorageHashBase<'sha256:4de46a86a05ed989a56dc452cdb4906e7822c33bb28cd0c73803592ea30ca55b'>;
StorageHashBase<'sha256:a9d70bb2a26470e9db304a036763fda7d1bc77158069e9de655e91f8126c705f'>;
export type ExecutionHash = ExecutionHashBase<string>;
export type ProfileHash =
ProfileHashBase<'sha256:9c8aa3114e84ed3b7ea2bd57526d9c2e1bf7c5292be694e9d3801f566fda7ccb'>;
Expand Down Expand Up @@ -141,7 +141,7 @@ type ContractBase = Omit<
{
readonly columns: readonly ['category', 'description', 'id', 'rating'];
readonly name: 'item_bm25_idx';
readonly type: 'bm25';
readonly type: 'paradedb';
readonly options: { readonly key_field: 'id' };
},
];
Expand Down
4 changes: 2 additions & 2 deletions paradedb-demo/src/prisma/contract.json
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@
"options": {
"key_field": "id"
},
"type": "bm25"
"type": "paradedb"
}
],
"primaryKey": {
Expand All @@ -146,7 +146,7 @@
"kind": "postgres-schema"
}
},
"storageHash": "sha256:4de46a86a05ed989a56dc452cdb4906e7822c33bb28cd0c73803592ea30ca55b"
"storageHash": "sha256:a9d70bb2a26470e9db304a036763fda7d1bc77158069e9de655e91f8126c705f"
},
"capabilities": {
"postgres": {
Expand Down
38 changes: 23 additions & 15 deletions paradedb-demo/test/vector.integration.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import 'dotenv/config';
import { renderBm25IndexDdl } from '@prisma-next/extension-paradedb/ddl';
import { renderParadeDbIndexDdl } from '@prisma-next/extension-paradedb/ddl';
import pg from 'pg';
import { beforeAll, describe, expect, it } from 'vitest';
import { loadAppConfig } from '../src/app-config';
Expand Down Expand Up @@ -33,34 +33,42 @@ describe.skipIf(SKIP)('paradedb vector integration', () => {
expect(rows[0]?.distance).toBeCloseTo(distanceL2(query, [0.2, 0.1, 0.85]), 6);
});

// Vector columns inside bm25 indexes need pg_search from branch
// `mvp/vector-search` (paradedb/paradedb#5685); released builds reject the
// DDL, so this Top-K pushdown coverage skips against current releases.
it('creates a bm25 index over the vector column when pg_search supports it', async (ctx) => {
// Vector columns inside ParadeDB indexes require pg_search 0.25.0+
// (paradedb/paradedb#5685); older builds reject the DDL, so this Top-K
// pushdown coverage skips there. A scratch table is used because a relation
// may only have one ParadeDB index and `item` already carries the
// contract-declared one.
it('creates a ParadeDB index over a vector column when pg_search supports it', async (ctx) => {
const { databaseUrl } = loadAppConfig();
const client = new pg.Client({ connectionString: databaseUrl });
await client.connect();
const ddl = renderBm25IndexDdl({
name: 'item_vector_bm25_idx',
table: 'item',
schema: 'public',
keyField: 'id',
columns: ['id', { column: 'embedding', metric: 'l2' }],
});
try {
await client.query(
'CREATE TABLE public.vector_idx_probe (id int4 PRIMARY KEY, embedding vector(3) NOT NULL)',
);
await client.query(
`INSERT INTO public.vector_idx_probe VALUES (1, '[1,0,0]'), (2, '[0.1,0.9,0.1]'), (3, '[0,0,1]')`,
);
const ddl = renderParadeDbIndexDdl({
name: 'vector_idx_probe_idx',
table: 'vector_idx_probe',
schema: 'public',
keyField: 'id',
columns: ['id', { column: 'embedding', metric: 'l2' }],
});
try {
await client.query(ddl);
} catch (error) {
ctx.skip(
`pg_search build does not support vector columns in bm25 indexes (needs branch mvp/vector-search): ${String(error)}`,
`pg_search build does not support vector columns in its indexes (requires pg_search 0.25.0+): ${String(error)}`,
);
}
const result = await client.query(
`SELECT id FROM public.item WHERE id @@@ pdb.all() ORDER BY embedding <-> '[0.1,0.9,0.1]' LIMIT 2`,
`SELECT id FROM public.vector_idx_probe WHERE id @@@ pdb.all() ORDER BY embedding <-> '[0.1,0.9,0.1]' LIMIT 2`,
);
expect(result.rows[0]?.id).toBe(2);
} finally {
await client.query('DROP INDEX IF EXISTS public.item_vector_bm25_idx');
await client.query('DROP TABLE IF EXISTS public.vector_idx_probe CASCADE');
await client.end();
}
});
Expand Down
26 changes: 14 additions & 12 deletions paradedb/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,20 @@ ParadeDB full-text and vector search extension pack for Prisma Next.

## Overview

This extension pack registers a `'bm25'` index type with the SQL family's index-type registry, so contracts can author BM25 full-text search indexes via the standard `constraints.index(...)` surface and the Postgres adapter emits `CREATE INDEX ... USING bm25 WITH (...)` DDL.
This extension pack registers a `'paradedb'` index type with the SQL family's index-type registry, so contracts can author BM25 full-text search indexes via the standard `constraints.index(...)` surface and the Postgres adapter emits `CREATE INDEX ... USING paradedb WITH (...)` DDL.

Requires pg_search 0.25.0+, which registers the `paradedb` index access method. This release renames the previously bm25-named API (`renderBm25IndexDdl`, the `'bm25'` index type, ...) to paradedb names; older servers and the old names are not supported.

The v1 surface covers the `key_field` storage parameter only. Per-field tokenizer and column configuration is deferred to expression-index support.

It also provides native pgvector `vector(n)` column support (no pgvector ORM library required): a `paradedb/vector@1` codec, a `vectorColumn(n)` column helper, the three pgvector distance operators as query operations, and the `@@@ pdb.all()` match-all predicate required for vector Top-K queries.

## Responsibilities

- **bm25 index registration**: declares a `'bm25'` entry via `defineIndexTypes()` carrying an arktype validator for the bm25 options shape
- **Index registration**: declares a `'paradedb'` entry via `defineIndexTypes()` carrying an arktype validator for the options shape
- **Vector columns**: `vectorColumn(dimensions)` maps to `vector(n)` DDL; the `paradedb/vector@1` codec serializes `number[]` ⇄ `'[1,2,3]'` in both directions
- **Vector queries**: `paradeDbL2Distance` (`<->`), `paradeDbCosineDistance` (`<=>`), `paradeDbInnerProduct` (`<#>`), and `paradeDbAll` (`@@@ pdb.all()`)
- **Vector index DDL**: `renderBm25IndexDdl(...)` renders `CREATE INDEX ... USING bm25` statements with per-column vector opclasses for raw migrations
- **Vector index DDL**: `renderParadeDbIndexDdl(...)` renders `CREATE INDEX ... USING paradedb` statements with per-column vector opclasses for raw migrations
- **Extension descriptor**: declares the `paradedb/bm25` and `paradedb/vector` capabilities for contract-level feature detection
- **Pack ref export**: ships a pure `/pack` entrypoint for TypeScript contract authoring

Expand All @@ -35,7 +37,7 @@ pnpm add @prisma-next/extension-paradedb

### Contract definition

Author bm25 indexes via the standard `constraints.index(...)` surface; the registered `'bm25'` entry narrows `options` per-`type`:
Author ParadeDB indexes via the standard `constraints.index(...)` surface; the registered `'paradedb'` entry narrows `options` per-`type`:

```typescript
import { int4Column, textColumn } from '@prisma-next/adapter-postgres/column-types';
Expand All @@ -58,8 +60,8 @@ export const contract = defineContract({
table: 'items',
indexes: [
constraints.index([cols.body], {
name: 'item_body_bm25_idx',
type: 'bm25',
name: 'item_body_search_idx',
type: 'paradedb',
options: { key_field: 'id' },
}),
],
Expand All @@ -74,7 +76,7 @@ ParadeDB BM25 indexes require a `key_field` — a unique column that identifies

## Vector search

ParadeDB indexes pgvector `vector` columns inside its own bm25 access method. This pack ships the minimal pgvector surface natively — do not install pgvector ORM packages alongside it. Vector-in-bm25 indexing requires a pg_search build from branch `mvp/vector-search` ([paradedb/paradedb#5685](https://github.com/paradedb/paradedb/issues/5685)); on released builds the queries below still run, but without Top-K index acceleration. The `vector` (pgvector) extension must be installed for the column type itself.
ParadeDB indexes pgvector `vector` columns inside its own index access method. This pack ships the minimal pgvector surface natively — do not install pgvector ORM packages alongside it. Vector-in-index support requires pg_search 0.25.0+ ([paradedb/paradedb#5685](https://github.com/paradedb/paradedb/issues/5685)); on older builds the queries below still run, but without Top-K index acceleration. The `vector` (pgvector) extension must be installed for the column type itself.

### Vector column declaration

Expand All @@ -94,18 +96,18 @@ The column emits `vector(3)` DDL and reads/writes `number[]` values.

### Index creation

Vector columns listed in a `constraints.index([...], { type: 'bm25', ... })` declaration get the bm25 AM's default opclass, `vector_l2_ops` (L2). The contract index surface cannot express per-column opclasses yet, so for the cosine or inner-product metric render the DDL for a raw migration:
Vector columns listed in a `constraints.index([...], { type: 'paradedb', ... })` declaration get the AM's default opclass, `vector_l2_ops` (L2). The contract index surface cannot express per-column opclasses yet, so for the cosine or inner-product metric render the DDL for a raw migration:

```typescript
import { renderBm25IndexDdl } from '@prisma-next/extension-paradedb/ddl';
import { renderParadeDbIndexDdl } from '@prisma-next/extension-paradedb/ddl';

renderBm25IndexDdl({
renderParadeDbIndexDdl({
name: 'item_vector_idx',
table: 'item',
keyField: 'id',
columns: ['id', 'description', { column: 'embedding', metric: 'cosine' }],
});
// CREATE INDEX "item_vector_idx" ON "item" USING bm25
// CREATE INDEX "item_vector_idx" ON "item" USING paradedb
// ("id", "description", "embedding" vector_cosine_ops) WITH (key_field = 'id')
```

Expand Down Expand Up @@ -145,7 +147,7 @@ The ORDER BY distance operator must match the index opclass metric; a mismatch d
## Not yet implemented

- Per-column / per-expression tokenizer configuration (deferred to expression-index support)
- Per-column vector opclasses in `constraints.index(...)` (use `renderBm25IndexDdl` meanwhile)
- Per-column vector opclasses in `constraints.index(...)` (use `renderParadeDbIndexDdl` meanwhile)
- `halfvec` / `sparsevec` column types
- `CREATE EXTENSION pg_search` / `CREATE EXTENSION vector` via migration planner (removed with upstream's `databaseDependencies` hook; install the extensions before `db init`)
- Aggregation and highlight functions
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
export type VectorMetric = 'l2' | 'cosine' | 'ip';

// Opclasses registered by pg_search on the bm25 access method. Same names as
// Opclasses registered by pg_search on the paradedb access method. Same names as
// pgvector's but distinct catalog objects; `vector_l2_ops` is the AM default.
export const VECTOR_OPCLASSES = {
l2: 'vector_l2_ops',
Expand All @@ -16,53 +16,53 @@ export const VECTOR_DISTANCE_OPERATORS = {
ip: '<#>',
} as const satisfies Record<VectorMetric, string>;

export interface Bm25VectorIndexColumn {
export interface ParadeDbVectorIndexColumn {
readonly column: string;
readonly metric?: VectorMetric;
}

export interface Bm25IndexDdlOptions {
export interface ParadeDbIndexDdlOptions {
readonly name: string;
readonly table: string;
readonly schema?: string;
readonly keyField: string;
readonly columns: ReadonlyArray<string | Bm25VectorIndexColumn>;
readonly columns: ReadonlyArray<string | ParadeDbVectorIndexColumn>;
}

function quoteIdentifier(identifier: string): string {
if (identifier.length === 0) {
throw new Error('renderBm25IndexDdl: identifier cannot be empty');
throw new Error('renderParadeDbIndexDdl: identifier cannot be empty');
}
if (identifier.includes('\0')) {
throw new Error('renderBm25IndexDdl: identifier cannot contain null bytes');
throw new Error('renderParadeDbIndexDdl: identifier cannot contain null bytes');
}
return `"${identifier.replace(/"/g, '""')}"`;
}

function escapeLiteral(value: string): string {
if (value.includes('\0')) {
throw new Error('renderBm25IndexDdl: literal cannot contain null bytes');
throw new Error('renderParadeDbIndexDdl: literal cannot contain null bytes');
}
return value.replace(/'/g, "''");
}

/**
* Renders `CREATE INDEX ... USING bm25 (...) WITH (key_field = '...')` DDL,
* Renders `CREATE INDEX ... USING paradedb (...) WITH (key_field = '...')` DDL,
* including per-column vector opclasses that the contract `constraints.index`
* path cannot express yet. Indexes declared through `constraints.index` get the
* default `vector_l2_ops` opclass for vector columns; use this helper (in a raw
* migration) when the index needs the cosine or inner-product metric.
*/
export function renderBm25IndexDdl(options: Bm25IndexDdlOptions): string {
export function renderParadeDbIndexDdl(options: ParadeDbIndexDdlOptions): string {
if (options.columns.length === 0) {
throw new Error('renderBm25IndexDdl: at least one column is required');
throw new Error('renderParadeDbIndexDdl: at least one column is required');
}
const columnNames = options.columns.map((entry) =>
typeof entry === 'string' ? entry : entry.column,
);
if (!columnNames.includes(options.keyField)) {
throw new Error(
`renderBm25IndexDdl: key_field "${options.keyField}" must be listed in columns`,
`renderParadeDbIndexDdl: key_field "${options.keyField}" must be listed in columns`,
);
}
const columnList = options.columns
Expand All @@ -71,7 +71,7 @@ export function renderBm25IndexDdl(options: Bm25IndexDdlOptions): string {
const opclass = VECTOR_OPCLASSES[entry.metric ?? 'l2'];
if (opclass === undefined) {
throw new Error(
`renderBm25IndexDdl: unknown vector metric ${JSON.stringify(entry.metric)}`,
`renderParadeDbIndexDdl: unknown vector metric ${JSON.stringify(entry.metric)}`,
);
}
return `${quoteIdentifier(entry.column)} ${opclass}`;
Expand All @@ -81,5 +81,5 @@ export function renderBm25IndexDdl(options: Bm25IndexDdlOptions): string {
options.schema === undefined
? quoteIdentifier(options.table)
: `${quoteIdentifier(options.schema)}.${quoteIdentifier(options.table)}`;
return `CREATE INDEX ${quoteIdentifier(options.name)} ON ${qualifiedTable} USING bm25 (${columnList}) WITH (key_field = '${escapeLiteral(options.keyField)}')`;
return `CREATE INDEX ${quoteIdentifier(options.name)} ON ${qualifiedTable} USING paradedb (${columnList}) WITH (key_field = '${escapeLiteral(options.keyField)}')`;
}
8 changes: 4 additions & 4 deletions paradedb/src/exports/ddl.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
export {
type Bm25IndexDdlOptions,
type Bm25VectorIndexColumn,
type ParadeDbIndexDdlOptions,
type ParadeDbVectorIndexColumn,
type VectorMetric,
renderBm25IndexDdl,
renderParadeDbIndexDdl,
VECTOR_DISTANCE_OPERATORS,
VECTOR_OPCLASSES,
} from '../core/bm25-index-ddl';
} from '../core/paradedb-index-ddl';
2 changes: 1 addition & 1 deletion paradedb/src/exports/index-types.ts
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
export type { Bm25IndexOptions, IndexTypes } from '../types/index-types';
export type { IndexTypes, ParadeDbIndexOptions } from '../types/index-types';
export { paradedbIndexTypes } from '../types/index-types';
4 changes: 2 additions & 2 deletions paradedb/src/types/index-types.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
import { defineIndexTypes } from '@prisma-next/sql-contract/index-types';
import { type } from 'arktype';

export const paradedbIndexTypes = defineIndexTypes().add('bm25', {
export const paradedbIndexTypes = defineIndexTypes().add('paradedb', {
options: type({
'+': 'reject',
key_field: 'string',
}),
});

export type IndexTypes = typeof paradedbIndexTypes.IndexTypes;
export type Bm25IndexOptions = IndexTypes['bm25']['options'];
export type ParadeDbIndexOptions = IndexTypes['paradedb']['options'];
Loading
Loading