feat: native vector search support (paradedb/paradedb#5685)

Author: rebasedming

CHANGELOG.md

@@ -8,6 +8,11 @@ All notable changes to this project will be documented in this file. The format
 
 - Support the `paradedb` index access method (requires pg_search 0.25.0+, unreleased). `ParadeDB::Index` and `add_paradedb_index` accept an `am:` option (`"paradedb"` or `"bm25"`), and schema dumps record `am:` when an index does not use the default.
 - `add_paradedb_index`, `remove_paradedb_index`, and `reindex_paradedb_index` are the primary names for the migration helpers (connection and migration DSL). The `bm25`-named helpers (`add_bm25_index`, `remove_bm25_index`, `reindex_bm25`) remain fully supported aliases. Schema dumps and generated migrations now emit the `paradedb`-named helpers; `schema.rb` files using either name continue to load.
+- Native pgvector `vector(n)` column support: ActiveRecord attribute type, `t.vector` / `add_column :table, :col, :vector, limit: n` migration DSL, and `schema.rb` dumping — no `neighbor` gem required.
+- Vector fields in BM25 indexes via `embedding: { metric: :l2 | :cosine | :ip }` in `ParadeDB::Index` fields and `add_paradedb_index`, emitted as `vector_l2_ops` / `vector_cosine_ops` / `vector_ip_ops` opclasses and round-tripped through the schema dumper.
+- `Model.nearest(column, vector, metric: nil)` for Top-K vector search. Orders by the pgvector distance operator (`<->`, `<=>`, `<#>`), defaults the metric from the index definition, and adds `key_field @@@ pdb.all()` when the relation has no ParadeDB predicate.
+- `l2_distance` / `cosine_distance` / `inner_product` / `vector_distance` Arel builder methods and `pdb_l2_distance` / `pdb_cosine_distance` / `pdb_inner_product` / `pdb_vector_distance` attribute predications.
+- `examples/vector_search` example. `examples/hybrid_rrf` now uses the native vector support instead of `neighbor`.
 
 ### Changed
 

README.md

@@ -47,13 +47,42 @@ The official ActiveRecord integration for [ParadeDB](https://paradedb.com) (powe
 | ParadeDB   | 0.22.0+                                          |
 | PostgreSQL | 15+ (PostgreSQL adapter with ParadeDB extension) |
 
+## Vector Search
+
+rails-paradedb natively supports pgvector `vector(n)` columns — no `neighbor` or other pgvector gem required. Declare the column, list it in the BM25 index with a distance metric, and run Top-K queries:
+
+```ruby
+class AddVectorSearchToProducts < ActiveRecord::Migration[8.1]
+  def change
+    add_column :products, :embedding, :vector, limit: 384
+
+    add_bm25_index :products,
+                   fields: { id: {}, description: {}, embedding: { metric: :l2 } },
+                   key_field: :id
+  end
+end
+```
+
+```ruby
+Product.nearest(:embedding, query_embedding).limit(10)
+```
+
+`nearest` orders by the pgvector distance operator for the chosen metric — `:l2` → `<->`, `:cosine` → `<=>`, `:ip` → `<#>` — and defaults to the metric declared in the index (`:l2` when none is declared). ParadeDB requires a `@@@` predicate alongside the vector `ORDER BY` to activate the index scan; `nearest` adds `key_field @@@ pdb.all()` automatically when the relation has no ParadeDB predicate, and composes with full-text search:
+
+```ruby
+Product.search(:description).match_all("shoes").nearest(:embedding, query_embedding).limit(10)
+```
+
+Top-K index pushdown requires a `LIMIT`, and the query metric must match the index opclass metric — a mismatch still returns correct results but silently falls back to a plain sort. Vector columns inside BM25 indexes require a ParadeDB build with bm25 vector opclasses (not yet in a released ParadeDB version); the plain `vector(n)` column type and distance expressions work on any ParadeDB or Postgres with pgvector.
+
 ## Examples
 
 - [Quickstart](examples/quickstart/quickstart.rb)
 - [Faceted Search](examples/faceted_search/faceted_search.rb)
 - [Autocomplete](examples/autocomplete/autocomplete.rb)
 - [More Like This](examples/more_like_this/more_like_this.rb)
 - [Hybrid Search (RRF)](examples/hybrid_rrf/hybrid_rrf.rb)
+- [Vector Search](examples/vector_search/vector_search.rb)
 - [RAG](examples/rag/rag.rb)
 
 ## Contributing

examples/Gemfile

@@ -1,6 +1,3 @@
 source "https://rubygems.org"
 
 eval_gemfile "../Gemfile"
-
-# Used by the hybrid search example (BM25 + pgvector).
-gem "neighbor"

examples/Gemfile.lock

@@ -67,8 +67,6 @@ GEM
     minitest (6.0.6)
       drb (~> 2.0)
       prism (~> 1.5)
-    neighbor (1.1.1)
-      activerecord (>= 7.2)
     nokogiri (1.19.4-aarch64-linux-gnu)
       racc (~> 1.4)
     nokogiri (1.19.4-arm64-darwin)
@@ -79,7 +77,6 @@ GEM
     parser (3.3.11.1)
       ast (~> 2.4.1)
       racc
-    pg (1.6.3)
     pg (1.6.3-aarch64-linux)
     pg (1.6.3-arm64-darwin)
     pg (1.6.3-x86_64-linux)
@@ -184,7 +181,6 @@ PLATFORMS
 DEPENDENCIES
   activerecord (~> 8.1)
   activesupport (~> 8.1)
-  neighbor
   pg (~> 1.5)
   railties (~> 8.1)
   rake (~> 13.4)

examples/README.md

@@ -18,9 +18,6 @@ Each example folder uses a Rails-like layout:
 BUNDLE_GEMFILE=examples/Gemfile bundle install
 ```
 
-The hybrid RRF example uses pgvector via `neighbor`, which is already included
-in `examples/Gemfile`.
-
 ### 2. Start ParadeDB
 
 ```bash
@@ -109,7 +106,7 @@ Structure:
 1. Hybrid Search with RRF (`hybrid_rrf/`)
 
 Demonstrates Reciprocal Rank Fusion (RRF) by composing a ParadeDB BM25 relation
-with a semantic relation (via `neighbor`) using CTEs.
+with a semantic relation (using the native vector distance helpers) via CTEs.
 
 ```bash
 BUNDLE_GEMFILE=examples/Gemfile bundle exec ruby examples/hybrid_rrf/setup.rb
@@ -122,6 +119,24 @@ Structure:
 - `examples/hybrid_rrf/setup.rb`
 - `examples/hybrid_rrf/hybrid_rrf.rb`
 
+1. Vector Search (`vector_search/`)
+
+Top-K vector search inside the BM25 index: `vector(n)` columns, metric
+opclasses, and `nearest` queries. Requires a ParadeDB build with bm25 vector
+opclasses (not yet in a released ParadeDB version); the setup script aborts
+with a clear message otherwise.
+
+```bash
+BUNDLE_GEMFILE=examples/Gemfile bundle exec ruby examples/vector_search/setup.rb
+BUNDLE_GEMFILE=examples/Gemfile bundle exec ruby examples/vector_search/vector_search.rb
+```
+
+Structure:
+
+- `examples/vector_search/model.rb`
+- `examples/vector_search/setup.rb`
+- `examples/vector_search/vector_search.rb`
+
 1. RAG (`rag/rag.rb`)
 
 Retrieves products with ParadeDB and sends context to OpenRouter.

examples/hybrid_rrf/hybrid_rrf.rb

@@ -24,7 +24,10 @@ def fulltext_ranked_cte(query, top_k:)
 end
 
 def semantic_ranked_cte(query_embedding, top_k:)
-  semantic_source = MockItem.nearest_neighbors(:embedding, query_embedding, distance: "cosine")
+  distance = MockItem.paradedb_arel.cosine_distance(:embedding, query_embedding)
+  semantic_source = MockItem.where.not(embedding: nil)
+                            .select(MockItem.arel_table[:id], distance.as("neighbor_distance"))
+                            .order(distance.asc)
                             .limit(top_k)
 
   MockItem.from(semantic_source, :semantic_source)
@@ -131,7 +134,7 @@ def display_results(query, rows)
   puts "=" * 80
   puts "Hybrid Search with Reciprocal Rank Fusion (single SQL query)"
   puts "=" * 80
-  puts "\nCombining ParadeDB DSL + Neighbor DSL in one CTE-based query"
+  puts "\nCombining ParadeDB BM25 + native vector distance in one CTE-based query"
 
   HybridRrfSetup.setup!
   MockItem.reset_column_information

examples/hybrid_rrf/model.rb

@@ -1,11 +1,5 @@
 # frozen_string_literal: true
 
-begin
-  require "neighbor"
-rescue LoadError
-  abort "Example requires neighbor. Run with `BUNDLE_GEMFILE=examples/Gemfile bundle exec ...`."
-end
-
 require "active_record"
 require_relative "../../lib/parade_db"
 
@@ -14,8 +8,6 @@ class MockItem < ActiveRecord::Base
 
   self.table_name = "mock_items_hybrid_rrf"
   self.primary_key = "id"
-
-  has_neighbors :embedding
 end
 
 class MockItemIndex < ParadeDB::Index

examples/vector_search/model.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require "active_record"
+require_relative "../../lib/parade_db"
+
+class VectorItem < ActiveRecord::Base
+  include ParadeDB::Model
+
+  self.table_name = "vector_items"
+  self.primary_key = "id"
+end
+
+class VectorItemIndex < ParadeDB::Index
+  self.table_name = :vector_items
+  self.key_field = :id
+  self.index_name = :vector_items_bm25_idx
+  self.fields = {
+    id: nil,
+    description: nil,
+    category: { tokenizer: ParadeDB::Tokenizer.literal() },
+    embedding: { metric: :l2 }
+  }
+end

examples/vector_search/setup.rb

@@ -0,0 +1,102 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "logger"
+require "rails"
+require "active_record"
+require_relative "../../lib/parade_db"
+
+class VectorSearchExampleApp < Rails::Application
+  config.root = File.expand_path("../..", __dir__)
+  config.eager_load = false
+  config.logger = Logger.new(nil)
+  config.secret_key_base = "paradedb_examples_secret_key_base"
+end
+
+VectorSearchExampleApp.initialize!
+
+require_relative "model"
+
+module VectorSearchSetup
+  module_function
+
+  SEED_ITEMS = [
+    { description: "Sleek running shoes", category: "Footwear", embedding: [1.0, 0.1, 0.0] },
+    { description: "Trail running shoes with grip", category: "Footwear", embedding: [0.9, 0.2, 0.1] },
+    { description: "White leather sneakers", category: "Footwear", embedding: [0.7, 0.4, 0.2] },
+    { description: "Innovative wireless earbuds", category: "Electronics", embedding: [0.0, 1.0, 0.1] },
+    { description: "Over-ear noise cancelling headphones", category: "Electronics", embedding: [0.1, 0.9, 0.2] },
+    { description: "Portable bluetooth speaker", category: "Electronics", embedding: [0.2, 0.8, 0.4] },
+    { description: "Insulated camping tent", category: "Outdoor", embedding: [0.1, 0.2, 1.0] },
+    { description: "Lightweight hiking backpack", category: "Outdoor", embedding: [0.3, 0.1, 0.9] }
+  ].freeze
+
+  def database_url
+    return ENV["DATABASE_URL"] if ENV["DATABASE_URL"]
+
+    host = ENV.fetch("PGHOST", "localhost")
+    port = ENV.fetch("PGPORT", "5432")
+    user = ENV.fetch("PGUSER", "postgres")
+    password = ENV.fetch("PGPASSWORD", "postgres")
+    database = ENV.fetch("PGDATABASE", "postgres")
+
+    "postgresql://#{user}:#{password}@#{host}:#{port}/#{database}"
+  end
+
+  def connect!
+    return if ActiveRecord::Base.connected?
+
+    ActiveRecord::Base.establish_connection(database_url)
+    ActiveRecord::Base.logger = nil
+  end
+
+  def vector_search_supported?
+    ActiveRecord::Base.connection.select_value(<<~SQL)
+      SELECT 1
+      FROM pg_opclass oc
+      JOIN pg_am am ON am.oid = oc.opcmethod
+      WHERE am.amname = 'bm25' AND oc.opcname = 'vector_l2_ops'
+      LIMIT 1
+    SQL
+  end
+
+  def setup!
+    connect!
+
+    conn = ActiveRecord::Base.connection
+    conn.execute("CREATE EXTENSION IF NOT EXISTS pg_search;")
+    conn.execute("CREATE EXTENSION IF NOT EXISTS vector;")
+
+    unless vector_search_supported?
+      abort "This example requires a ParadeDB build with bm25 vector opclasses " \
+            "(unreleased; see paradedb/paradedb branch mvp/vector-search)."
+    end
+
+    conn.execute("DROP TABLE IF EXISTS vector_items CASCADE;")
+    ActiveRecord::Schema.define do
+      suppress_messages do
+        create_table :vector_items, force: true do |t|
+          t.text :description
+          t.text :category
+          t.vector :embedding, limit: 3
+        end
+      end
+    end
+
+    VectorItem.reset_column_information
+    SEED_ITEMS.each { |attrs| VectorItem.create!(attrs) }
+
+    conn.create_paradedb_index(VectorItemIndex, if_not_exists: true)
+    VectorItem.count
+  end
+end
+
+if $PROGRAM_NAME == __FILE__
+  puts "=" * 60
+  puts "Vector Search Setup - Creating vector_items Table"
+  puts "=" * 60
+
+  count = VectorSearchSetup.setup!
+  puts "+ Seeded #{count} items with vector(3) embeddings"
+  puts "\nSetup complete! Run: BUNDLE_GEMFILE=examples/Gemfile bundle exec ruby examples/vector_search/vector_search.rb"
+end

examples/vector_search/vector_search.rb

@@ -0,0 +1,51 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative "setup"
+
+QUERY_EMBEDDING = [1.0, 0.0, 0.0].freeze
+
+def demo_top_k
+  puts "\n--- Top-K Nearest Neighbors (L2, from the index metric) ---"
+  results = VectorItem.nearest(:embedding, QUERY_EMBEDDING).limit(3)
+  puts(results.map { |item| "  - #{item.description}" })
+end
+
+def demo_metric_override
+  puts "\n--- Metric Override (cosine; falls back to a plain sort without a matching opclass) ---"
+  results = VectorItem.nearest(:embedding, QUERY_EMBEDDING, metric: :cosine).limit(3)
+  puts(results.map { |item| "  - #{item.description}" })
+end
+
+def demo_filtered_vector_search
+  puts "\n--- Vector Search + Full-Text Filter: category 'Footwear' ---"
+  results = VectorItem.search(:category)
+                      .term("Footwear")
+                      .nearest(:embedding, QUERY_EMBEDDING)
+                      .limit(3)
+  puts(results.map { |item| "  - #{item.description}" })
+end
+
+def demo_distance_projection
+  puts "\n--- Distance Projection ---"
+  distance = VectorItem.paradedb_arel.l2_distance(:embedding, QUERY_EMBEDDING)
+  results = VectorItem.nearest(:embedding, QUERY_EMBEDDING)
+                      .select(VectorItem.arel_table[Arel.star], distance.as("distance"))
+                      .limit(3)
+  puts(results.map { |item| "  - #{item.description} (distance: #{item.distance.round(3)})" })
+end
+
+if $PROGRAM_NAME == __FILE__
+  puts "=" * 60
+  puts "rails-paradedb Vector Search Example"
+  puts "=" * 60
+
+  VectorSearchSetup.setup!
+
+  demo_top_k
+  demo_metric_override
+  demo_filtered_vector_search
+  demo_distance_projection
+
+  puts "\nDone!"
+end