Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ All notable changes to this project will be documented in this file. The format

## [Unreleased]

### Added

- Native vector search support: `vector(n)` column mapping for `float[]`
properties (`HasVectorType`), vector fields with metric operator classes in
ParadeDB indexes (`HasField(..., VectorMetric)`), and the `L2Distance`,
`CosineDistance`, and `InnerProduct` distance functions.

### Changed

- **Breaking**: `HasBm25Index` and `Bm25IndexBuilder<TEntity>` are renamed to
Expand Down
4 changes: 3 additions & 1 deletion NUGET-README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ParadeDB for Entity Framework Core

The official [Entity Framework Core](https://learn.microsoft.com/en-us/ef/core/) integration for [ParadeDB](https://paradedb.com), including first-class support for managing BM25 indexes and running queries using the full ParadeDB API. Follow the [getting started guide](https://docs.paradedb.com/documentation/getting-started/environment#entity-framework-core) to begin.
The official [Entity Framework Core](https://learn.microsoft.com/en-us/ef/core/) integration for [ParadeDB](https://paradedb.com), including first-class support for managing BM25 indexes and running queries using the full ParadeDB API. The integration covers both full-text and vector search — vector search indexes pgvector `vector` columns directly, with no separate pgvector ORM plugin required. Follow the [getting started guide](https://docs.paradedb.com/documentation/getting-started/environment#entity-framework-core) to begin.

## Requirements & Compatibility

Expand All @@ -10,6 +10,7 @@ The official [Entity Framework Core](https://learn.microsoft.com/en-us/ef/core/)
| EF Core | 8.0+ |
| ParadeDB | 0.25.0+ |
| PostgreSQL | 15+ (with ParadeDB extension) |
| pgvector | Required for vector search |

## Examples

Expand All @@ -18,6 +19,7 @@ The official [Entity Framework Core](https://learn.microsoft.com/en-us/ef/core/)
- [Autocomplete](examples/Autocomplete/Program.cs)
- [More Like This](examples/MoreLikeThis/Program.cs)
- [Hybrid Search (RRF)](examples/HybridRrf/Program.cs)
- [Vector Search](examples/VectorSearch/Program.cs)
- [RAG](examples/Rag/Program.cs)

## Contributing
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@

## ParadeDB for Entity Framework Core

The official [Entity Framework Core](https://learn.microsoft.com/en-us/ef/core/) integration for [ParadeDB](https://paradedb.com) (powered by the [`pg_search`](https://github.com/paradedb/paradedb) Postgres extension), including first-class support for managing BM25 indexes and running queries using the full ParadeDB API. Follow the [getting started guide](https://docs.paradedb.com/documentation/getting-started/environment#entity-framework-core) to begin.
The official [Entity Framework Core](https://learn.microsoft.com/en-us/ef/core/) integration for [ParadeDB](https://paradedb.com) (powered by the [`pg_search`](https://github.com/paradedb/paradedb) Postgres extension), including first-class support for managing BM25 indexes and running queries using the full ParadeDB API. The integration covers both full-text and vector search — vector search indexes pgvector `vector` columns directly, with no separate pgvector ORM plugin required. Follow the [getting started guide](https://docs.paradedb.com/documentation/getting-started/environment#entity-framework-core) to begin.

## Requirements & Compatibility

Expand All @@ -46,6 +46,7 @@ The official [Entity Framework Core](https://learn.microsoft.com/en-us/ef/core/)
| EF Core | 8.0+ |
| ParadeDB | 0.25.0+ |
| PostgreSQL | 15+ (with ParadeDB extension) |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add pgvector as a requirement in these tables?

| pgvector | Required for vector search |

## Examples

Expand All @@ -54,6 +55,7 @@ The official [Entity Framework Core](https://learn.microsoft.com/en-us/ef/core/)
- [Autocomplete](examples/Autocomplete/Program.cs)
- [More Like This](examples/MoreLikeThis/Program.cs)
- [Hybrid Search (RRF)](examples/HybridRrf/Program.cs)
- [Vector Search](examples/VectorSearch/Program.cs)
- [RAG](examples/Rag/Program.cs)

## Contributing
Expand Down
4 changes: 0 additions & 4 deletions examples/Directory.Packages.props
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
<Project>
<Import Project="..\Directory.Packages.props" />

<ItemGroup>
<PackageVersion Include="Pgvector.EntityFrameworkCore" Version="0.3.0" />
</ItemGroup>
</Project>
3 changes: 1 addition & 2 deletions examples/Examples.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,13 @@
</PropertyGroup>

<Target Name="RequireExample" BeforeTargets="CoreCompile" Condition="'$(Example)' == ''">
<Error Text="Set the Example property to one of: Quickstart, FacetedSearch, Autocomplete, MoreLikeThis, HybridRrf, Rag." />
<Error Text="Set the Example property to one of: Quickstart, FacetedSearch, Autocomplete, MoreLikeThis, HybridRrf, VectorSearch, Rag." />
</Target>

<ItemGroup>
<PackageReference Include="EFCore.NamingConventions" />
<PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" />
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" />
<PackageReference Include="Pgvector.EntityFrameworkCore" />
</ItemGroup>

<ItemGroup>
Expand Down
3 changes: 1 addition & 2 deletions examples/HybridRrf/Data/MockItemWithEmbedding.cs
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
using Pgvector;
using Shared;

namespace HybridRrf.Data;

public class MockItemWithEmbedding : MockItem
{
public Vector? Embedding { get; set; }
public float[]? Embedding { get; set; }
}
39 changes: 39 additions & 0 deletions examples/HybridRrf/EmbeddingLoader.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
using HybridRrf.Data;
using Microsoft.EntityFrameworkCore;

namespace HybridRrf;

public static class EmbeddingLoader
{
public static async Task LoadAsync(AppDbContext db)
{
var csvPath = Path.Combine(
AppContext.BaseDirectory,
"HybridRrf",
"mock_items_embeddings.csv"
);
var lines = await File.ReadAllLinesAsync(csvPath);

var embeddings = new Dictionary<int, float[]>();

for (var i = 1; i < lines.Length; i++)
{
var parts = lines[i].Split(',', 3);
embeddings[int.Parse(parts[0])] = parts[2]
.Trim('"', '[', ']')
.Split(',', StringSplitOptions.TrimEntries)
.Select(float.Parse)
.ToArray();
}

var ids = embeddings.Keys.ToArray();
var items = await db.MockItems.Where(x => ids.Contains(x.Id)).ToListAsync();
foreach (var item in items)
{
item.Embedding = embeddings[item.Id];
}

await db.SaveChangesAsync();
Console.WriteLine($"Loaded {items.Count} embeddings");
}
}
47 changes: 5 additions & 42 deletions examples/HybridRrf/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,10 @@
using HybridRrf.Data;
using Microsoft.EntityFrameworkCore;
using ParadeDB.EntityFrameworkCore.Extensions;
using Pgvector;
using Pgvector.EntityFrameworkCore;
using Shared;

var options = new DbContextOptionsBuilder<AppDbContext>()
.UseNpgsql(
ExampleSetup.ConnectionString,
o =>
{
o.UseParadeDb();
o.UseVector();
}
)
.UseNpgsql(ExampleSetup.ConnectionString, o => o.UseParadeDb())
.UseSnakeCaseNamingConvention()
.Options;

Expand All @@ -27,7 +18,7 @@
Console.WriteLine("RRF formula: score = sum(1 / (k + rank)) across all rankings");

await ExampleSetup.SetupHybridAsync(dbContext);
await LoadEmbeddingsAsync(dbContext);
await EmbeddingLoader.LoadAsync(dbContext);

await Demo(dbContext, "running shoes", QueryEmbeddings.Values);
await Demo(dbContext, "footwear for exercise", QueryEmbeddings.Values);
Expand All @@ -40,14 +31,14 @@

static async Task Demo(AppDbContext db, string query, Dictionary<string, float[]> queryEmbeddings)
{
var results = await HybridSearch(db, query, new Vector(queryEmbeddings[query]));
var results = await HybridSearch(db, query, queryEmbeddings[query]);
DisplayResults(query, results);
}

static async Task<List<(string Description, double RrfScore)>> HybridSearch(
AppDbContext db,
string query,
Vector queryEmbedding,
float[] queryEmbedding,
int topK = 20,
int rrfK = 60,
int limit = 5
Expand All @@ -71,7 +62,7 @@ static async Task Demo(AppDbContext db, string query, Dictionary<string, float[]
{
x.Id,
x.Description,
Distance = x.Embedding!.CosineDistance(queryEmbedding),
Distance = EF.Functions.CosineDistance(x.Embedding, queryEmbedding),
})
.OrderBy(x => x.Distance)
.Take(topK)
Expand Down Expand Up @@ -101,31 +92,3 @@ static void DisplayResults(string query, List<(string Description, double RrfSco
Console.WriteLine($" {i + 1}. {desc, -60} (RRF: {results[i].RrfScore:F4})");
}
}

static async Task LoadEmbeddingsAsync(AppDbContext db)
{
var csvPath = Path.Combine(AppContext.BaseDirectory, "HybridRrf", "mock_items_embeddings.csv");
var lines = await File.ReadAllLinesAsync(csvPath);

var embeddings = new Dictionary<int, float[]>();

for (var i = 1; i < lines.Length; i++)
{
var parts = lines[i].Split(',', 3);
embeddings[int.Parse(parts[0])] = parts[2]
.Trim('"', '[', ']')
.Split(',', StringSplitOptions.TrimEntries)
.Select(float.Parse)
.ToArray();
}

var ids = embeddings.Keys.ToArray();
var items = await db.MockItems.Where(x => ids.Contains(x.Id)).ToListAsync();
foreach (var item in items)
{
item.Embedding = new Vector(embeddings[item.Id]);
}

await db.SaveChangesAsync();
Console.WriteLine($"Loaded {items.Count} embeddings");
}
9 changes: 9 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,15 @@ ranking.
dotnet run --project examples/Examples.csproj -p:Example=HybridRrf
```

## Vector Search (`examples/VectorSearch`)

Runs Top-K nearest-neighbor queries over a pgvector column with ParadeDB
vector support, backed by a paradedb index.

```bash
dotnet run --project examples/Examples.csproj -p:Example=VectorSearch
```

## RAG: Retrieval-Augmented Generation (`examples/Rag`)

Builds a small QA flow that retrieves relevant context and sends it to an LLM
Expand Down
15 changes: 11 additions & 4 deletions examples/Shared/ExampleSetup.cs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ await db.Database.ExecuteSqlRawAsync(
"CALL paradedb.create_bm25_test_table(schema_name => 'public', table_name => 'mock_items')"
);
await db.Database.ExecuteSqlRawAsync("DROP INDEX IF EXISTS mock_items_bm25_idx");
// The VectorSearch example may have left its own index behind, and newer
// pg_search versions allow only one ParadeDB index per table
await db.Database.ExecuteSqlRawAsync("DROP INDEX IF EXISTS mock_items_vector_idx");
Comment on lines 48 to +51

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should update all examples to use one index name between them. Maybe search_idx so it's not bm25 or vector specific. Then we don't need to have a separate drop command

await db.Database.ExecuteSqlRawAsync(
"""
CREATE INDEX mock_items_bm25_idx ON mock_items
Expand Down Expand Up @@ -102,14 +105,18 @@ public static async Task SetupHybridAsync(DbContext db)
{
await SetupMockItemsAsync(db);
await db.Database.ExecuteSqlRawAsync("CREATE EXTENSION IF NOT EXISTS vector");
// Newer pg_search versions create mock_items with an embedding column of a
// different dimension; the examples use 384-dimensional embeddings
await db.Database.ExecuteSqlRawAsync(
"""
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'mock_items' AND column_name = 'embedding'
) THEN
IF (
SELECT atttypmod FROM pg_attribute
WHERE attrelid = 'mock_items'::regclass
AND attname = 'embedding' AND NOT attisdropped
) IS DISTINCT FROM 384 THEN
ALTER TABLE mock_items DROP COLUMN IF EXISTS embedding;
ALTER TABLE mock_items ADD COLUMN embedding vector(384);
END IF;
END $$;
Expand Down
65 changes: 65 additions & 0 deletions examples/VectorSearch/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
using HybridRrf;
using HybridRrf.Data;
using Microsoft.EntityFrameworkCore;
using ParadeDB.EntityFrameworkCore.Extensions;
using Shared;

var options = new DbContextOptionsBuilder<AppDbContext>()
.UseNpgsql(ExampleSetup.ConnectionString, o => o.UseParadeDb())
.UseSnakeCaseNamingConvention()
.Options;

await using var dbContext = new AppDbContext(options);

Console.WriteLine(new string('=', 70));
Console.WriteLine("Vector Search");
Console.WriteLine(new string('=', 70));
Console.WriteLine("\nTop-K nearest-neighbor search over a pgvector column");

await ExampleSetup.SetupHybridAsync(dbContext);
await EmbeddingLoader.LoadAsync(dbContext);

// Newer pg_search versions allow only one ParadeDB index per table
await dbContext.Database.ExecuteSqlRawAsync("DROP INDEX IF EXISTS mock_items_bm25_idx");
await dbContext.Database.ExecuteSqlRawAsync("DROP INDEX IF EXISTS mock_items_vector_idx");
await dbContext.Database.ExecuteSqlRawAsync(
"""
CREATE INDEX mock_items_vector_idx ON mock_items
USING paradedb (id, embedding vector_cosine_ops)
WITH (key_field='id');
"""
);
Console.WriteLine("Created a paradedb index over the embedding column (cosine metric)");

await Demo(dbContext, "running shoes");
await Demo(dbContext, "wireless earbuds");
return;

static async Task Demo(AppDbContext db, string query)
{
var queryEmbedding = QueryEmbeddings.Values[query];

// The @@@ predicate (EF.Functions.All) and the LIMIT are required for the
// ParadeDB index to serve the query; the ORDER BY metric must match the
// index opclass (cosine here)
var results = await db
.MockItems.Where(x => EF.Functions.All(x.Id))
.Select(x => new
{
x.Description,
Distance = EF.Functions.CosineDistance(x.Embedding, queryEmbedding),
})
.OrderBy(x => x.Distance)
.Take(5)
.ToListAsync();

Console.WriteLine($"\n{new string('=', 70)}");
Console.WriteLine($"Query: '{query}'");
Console.WriteLine(new string('=', 70));

for (var i = 0; i < results.Count; i++)
{
var desc = results[i].Description[..Math.Min(60, results[i].Description.Length)];
Console.WriteLine($" {i + 1}. {desc, -60} (distance: {results[i].Distance:F4})");
}
}
1 change: 1 addition & 0 deletions scripts/run_examples.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ examples=(
Autocomplete
MoreLikeThis
HybridRrf
VectorSearch

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the other ORMs I see you put VectorSearch below Rag. I think it looks better above, but that's a nit

Rag
)

Expand Down
20 changes: 20 additions & 0 deletions src/Extensions/ParadeDbDbContextOptionsBuilderExtensions.cs
Original file line number Diff line number Diff line change
@@ -1,17 +1,37 @@
using Microsoft.EntityFrameworkCore.Infrastructure;
using Npgsql.EntityFrameworkCore.PostgreSQL.Infrastructure;
using ParadeDB.EntityFrameworkCore.Internal;
#if NET8_0
using Npgsql;
using ParadeDB.EntityFrameworkCore.Internal.Storage;
#endif

// ReSharper disable SuspiciousTypeConversion.Global

namespace ParadeDB.EntityFrameworkCore.Extensions;

public static class ParadeDbDbContextOptionsBuilderExtensions
{
#if NET8_0
private static int _vectorResolverRegistered;
#endif

public static NpgsqlDbContextOptionsBuilder UseParadeDb(
this NpgsqlDbContextOptionsBuilder optionsBuilder
)
{
#if NET8_0
// Npgsql.EntityFrameworkCore.PostgreSQL 8.x has no INpgsqlDataSourceConfigurationPlugin,
// so the vector resolver is registered globally instead
if (Interlocked.Exchange(ref _vectorResolverRegistered, 1) == 0)
{
#pragma warning disable CS0618
NpgsqlConnection.GlobalTypeMapper.AddTypeInfoResolverFactory(
new PdbVectorTypeInfoResolverFactory()
);
#pragma warning restore CS0618
}
#endif
var coreOptionsBuilder = (
(IRelationalDbContextOptionsBuilderInfrastructure)optionsBuilder
).OptionsBuilder;
Expand Down
Loading
Loading