Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions pgrx-examples/pbox_slice/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
.DS_Store
.idea/
/target
*.iml
**/*.rs.bk
Cargo.lock
sql/pbox_slice-*.sql
35 changes: 35 additions & 0 deletions pgrx-examples/pbox_slice/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#LICENSE Portions Copyright 2019-2021 ZomboDB, LLC.
#LICENSE
#LICENSE Portions Copyright 2021-2023 Technology Concepts & Design, Inc.
#LICENSE
#LICENSE Portions Copyright 2023-2023 PgCentral Foundation, Inc. <contact@pgcentral.org>
#LICENSE
#LICENSE All rights reserved.
#LICENSE
#LICENSE Use of this source code is governed by the MIT license that can be found in the LICENSE file.

[package]
name = "pbox_slice"
version = "0.0.0"
edition.workspace = true
rust-version.workspace = true
publish = false

[lib]
crate-type = ["cdylib"]

[features]
default = ["pg13"]
pg13 = ["pgrx/pg13", "pgrx-tests/pg13"]
pg14 = ["pgrx/pg14", "pgrx-tests/pg14"]
pg15 = ["pgrx/pg15", "pgrx-tests/pg15"]
pg16 = ["pgrx/pg16", "pgrx-tests/pg16"]
pg17 = ["pgrx/pg17", "pgrx-tests/pg17"]
pg18 = ["pgrx/pg18", "pgrx-tests/pg18"]
pg_test = []

[dependencies]
pgrx = { path = "../../pgrx" }

[dev-dependencies]
pgrx-tests = { path = "../../pgrx-tests" }
72 changes: 72 additions & 0 deletions pgrx-examples/pbox_slice/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# pbox_slice — `MemCx::alloc_varlena` shown in a real workload

A teaching extension that demonstrates `MemCx::alloc_varlena` and `PBox<RawVarlena>` in code that mirrors a real use case rather than a synthetic benchmark.

The shape is a **minimal vector-embedding store**: pack a SQL `real[]` into a 4-byte-per-dimension binary `bytea` (matching pgvector's binary convention), and provide the distance / arithmetic primitives a smallRAG or k-NN application would actually run.

| Function | What it does |
|----------|-------------|
| `embedding_pack(real[]) -> bytea` | Pack a float array as binary embedding (4 bytes/dim). NULL → 0.0. |
| `embedding_dims(bytea) -> int` | Number of dimensions (or `-1` if layout is invalid). |
| `embedding_l2_squared(bytea, bytea) -> float8` | Squared Euclidean distance — skip the sqrt because nearest-neighbour ordering only needs the monotonic transform. |
| `embedding_dot(bytea, bytea) -> float8` | Dot product. For unit-length vectors this equals cosine similarity. |
| `embedding_cosine(bytea, bytea) -> float8` | Cosine similarity. NaN on dim mismatch or zero norm. |
| `embedding_mean(bytea, bytea) -> bytea` | Element-wise mean. Useful for centroids and ensembling. |
| `embedding_normalize(bytea) -> bytea` | L2-normalize to unit length. Lets downstream cosine become a plain dot product. |
| `embedding_seq_mean(bytea, int) -> bytea` | Mean-pool a packed sequence of N embeddings. **Demonstrates child-MemCx scratch + `'mcx` lifetime protection** via `PBox::from_iter_in`. |
| `build_bytea(int) -> bytea` | Smoke-test fixture for SQL-level checks; unrelated to embeddings. |

## Example workflow

```sql
CREATE TABLE documents (
id bigserial PRIMARY KEY,
text text,
emb bytea -- packed embedding from external embedder
);

-- Insert: external embedder returns real[]; we pack to compact binary.
INSERT INTO documents (text, emb)
VALUES (
'rust is fun',
embedding_pack(ARRAY[0.13, 0.42, 0.78, /* ... */]::real[])
);

-- Verify the encoding round-trips.
SELECT embedding_dims(emb) FROM documents LIMIT 1;

-- Top-K nearest neighbours by L2 distance.
SELECT id, text,
embedding_l2_squared(emb, $query_emb) AS dist
FROM documents
ORDER BY dist
LIMIT 10;

-- Pre-normalize so cosine = dot, run faster searches downstream.
UPDATE documents SET emb = embedding_normalize(emb);
```

## Why this matters for the API surface

Every function above is a real workload an extension author would
write. They exercise the new APIs in the shapes that matter:

- **`embedding_pack`, `_mean`, `_normalize`**: build a new bytea from
scratch with exact size known up-front. No intermediate `Vec<u8>`,
no `memcpy` from Rust heap into a varlena. The output goes straight
to Postgres.
- **`embedding_l2_squared`, `_dot`, `_cosine`, `_dims`**: read existing
byteas via `&[u8]` (zero-copy in) and produce a scalar. No
allocation needed; tests confirm dimension-mismatch / zero-norm
edge cases return NaN rather than crashing.

For embeddings sized 384–1536 floats (≈1.5–6 KB each) over millions of
rows, eliminating one `Vec<u8>` allocation + memcpy per query pays off
visibly. That is the point of `MemCx::alloc_varlena`.

## Build & test

```bash
cd pgrx-examples/pbox_slice
cargo pgrx test pg18
```
5 changes: 5 additions & 0 deletions pgrx-examples/pbox_slice/pbox_slice.control
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
comment = 'pbox_slice: demonstrates MemCx slice + varlena APIs'
default_version = '@CARGO_VERSION@'
module_pathname = '$libdir/pbox_slice'
relocatable = false
superuser = false
Loading
Loading