feat: DynamoDB vector index support - #175
Draft
hicksy wants to merge 5 commits into
Draft
Conversation
Adds the vector index request and description types, validation with error messages captured from real DynamoDB, metadata storage with a schema migration on both backends, a per-index shadow table, DescribeTable reporting, and DeleteTable cleanup. SearchVectors itself is not yet implemented; this is the first slice of #174.
Contributor
Criterion Benchmark ResultsBaseline is the per-benchmark median of the last 5 stored runs, so one unusually fast or slow runner cannot skew the comparison. The range column is the spread across those runs.
Runs in the baseline
|
Empty VectorIndexes lists are rejected, fractional dimensions truncate, and oversized values reuse the captured bounds message instead of leaking serde text. Adds the captured SearchSchema rules (single HASH, no duplicate attributes, vector attribute not in AttributeDefinitions), rejects cross-family duplicate index names, reports real positions in validation errors, recreates shadow tables cleanly over orphans, and moves the request types to types.rs beside their GSI and LSI siblings.
Contributor
Criterion Benchmark Results
|
VectorIndexUpdates supports Create and Delete, with a synchronous backfill that sparse-skips values a live write would reject. Vector, GSI, and LSI names share one namespace on this path, a GSI can no longer key on a live vector attribute, billing mode cannot leave PAY_PER_REQUEST while vector indexes exist, and SearchSchema attributes survive AttributeDefinitions reconciliation. Error strings and request-model paths are pinned to captured DynamoDB behaviour.
Contributor
Criterion Benchmark Results
|
The rust-cache restore can carry a partial target/criterion from an earlier run, and Criterion's failed baseline reads then interleave errors into the bencher output, breaking the comparison parser. The comparison uses the benchmark-data branch, not Criterion's own baselines, so the run should start from a clean criterion directory.
Contributor
Criterion Benchmark ResultsBaseline is the per-benchmark median of the last 5 stored runs, so one unusually fast or slow runner cannot skew the comparison. The range column is the spread across those runs.
Runs in the baseline
|
Every write surface now keeps vector shadow tables current: PutItem, UpdateItem, DeleteItem, batch and transactional writes, PartiQL, the TTL sweep, and imports. Writes validate vector values and SearchSchema attributes with error messages captured from real DynamoDB, storing f32 index copies while the base table keeps full precision. Imports sparse-skip invalid values the way backfill does, and batch writes parse index definitions once per table.
Contributor
Criterion Benchmark ResultsBaseline is the per-benchmark median of the last 5 stored runs, so one unusually fast or slow runner cannot skew the comparison. The range column is the spread across those runs.
Runs in the baseline
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
First slice of DynamoDB vector index support (#174): CreateTable accepts
VectorIndexes, definitions persist with a schema migration on both backends, DescribeTable reports them, and DeleteTable cleans up the per-index shadow tables. Error messages are pinned to strings captured from real DynamoDB.SearchVectors, the UpdateTable path, write-path maintenance, capacity accounting, the WASM surface, and MCP parity land as further commits on this branch.Checklist
cargo fmt --checkandcargo clippy -- -D warningspass locallyCHANGELOG.mdupdated if this is a user-visible changeDynamoDB compatibility note
Vector indexes go ACTIVE immediately: CREATING in the create response, ACTIVE in DescribeTable, matching the existing GSI posture. The backfilling window real AWS reports on the UpdateTable path is therefore unreachable here; the compatibility docs entry lands with the docs commit on this branch.