Cosmos Full Text Search support

[maumar]

• Adding model building API to configure property as full-text search enabled, as well as setup the index for it,
• Adding model validation (e.g. FTS index not matching FTS property),
• Adding EF.Functions stubs and translations for FullTextContains, FullTextContainsAll, FullTextContainsAny, FullTextScore and RRF (for hybrid),
• Adding logic in SelectExpression to produce ORDER BY RANK when necessary,
• Adding validation when attempting to mix with ORDER BY RANK with regular ORDER BY,
• Rewrite OFFSET/LIMIT from parameter to constant when ORDER BY RANK is present.
• Adding model building support for default language on the container level

Also fixed / added support for vector search on owned types (since it shares logic with FTS) and added some tests.

outstanding work:

• support for FTS Container building using Azure.ResourceManager.CosmosDb (currently blocked on updated package being released)

Fixes #35476
Fixes #35851
Fixes #35852
Fixes #35853
Fixes #35867

[Copilot]

Pull Request Overview

This PR adds support for full-text search in Cosmos by introducing new full-text index types, translators, model validations, and extensions.

• Introduces full-text indexing support in container creation and client wrappers.
• Adds a new CosmosFullTextSearchTranslator and corresponding DbFunctions extensions.
• Updates model validation and Fluent API extensions to include full-text search configuration.

Reviewed Changes

Copilot reviewed 23 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs	Adds support for full-text indexing in container creation, but the new full-text index assignment may override vector index settings.
src/EFCore.Cosmos/Query/Internal/Translators/CosmosVectorSearchTranslator.cs	Modifies method checking logic that may prematurely reject valid method calls by changing && to
src/EFCore.Cosmos/Query/Internal/Translators/CosmosFullTextSearchTranslator.cs	Implements a new translator for full-text search functions.
src/EFCore.Cosmos/Query/Internal/Expressions/SqlFunctionExpression.cs	Introduces an extra constructor parameter to flag scoring functions.
src/EFCore.Cosmos/Query/Internal/Expressions/SelectExpression.cs	Adds a RankOrdering property and methods to apply rank-based ordering.
src/EFCore.Cosmos/Query/Internal/CosmosQueryableMethodTranslatingExpressionVisitor.cs	Adjusts ordering logic to support scoring functions and rank ordering.
src/EFCore.Cosmos/Query/Internal/CosmosQuerySqlGenerator.cs	Updates SQL generation to include rank ordering handling.
src/EFCore.Cosmos/Metadata/Internal/CosmosAnnotationNames.cs	Defines new annotations for full-text search configuration.
src/EFCore.Cosmos/Infrastructure/Internal/CosmosModelValidator.cs	Validates full-text index configuration, ensuring only single-property indexes with language annotation are defined.
src/EFCore.Cosmos/Extensions/CosmosPropertyExtensions.cs, CosmosPropertyBuilderExtensions.cs, CosmosIndexExtensions.cs, CosmosIndexBuilderExtensions.cs, CosmosDbFunctionsExtensions.cs	Adds Fluent API and extension methods for configuring full-text search on properties and indexes.

Files not reviewed (3)

• Directory.Packages.props: Language not supported
• src/EFCore.Cosmos/EFCore.Cosmos.csproj: Language not supported
• src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported

Comments suppressed due to low confidence (2)

src/EFCore.Cosmos/Query/Internal/Translators/CosmosVectorSearchTranslator.cs:33

• Changing the logical operator from '&&' to '||' alters the intended logic and causes the translator to return null for methods declared on CosmosDbFunctionsExtensions that are not named 'VectorDistance'. Revisit the condition to ensure it only returns null when both conditions are unmet.

if (method.DeclaringType != typeof(CosmosDbFunctionsExtensions) || method.Name != nameof(CosmosDbFunctionsExtensions.VectorDistance))

src/EFCore.Cosmos/Extensions/CosmosPropertyExtensions.cs:137

• [nitpick] The Experimental attribute for full-text search methods is using 'CosmosVectorSearchExperimental' instead of the more appropriate 'CosmosFullTextSearchExperimental'. This may cause confusion regarding the feature's experiment status.

[Experimental(EFDiagnostics.CosmosVectorSearchExperimental)]

[roji]

@maumar just to note that I'm holding back from reviewing since this is still in draft (you seemed to maybe want people to take a look in yesterday's meeting). Let me know if you want a review etc.

[maumar]

@roji feel free to hold off on reviewing a draft. I'm pretty close to marking it for review I think

[roji]

Ended up doing a quick review even though the PR is still in draft... Here are some comments.

[roji]

/cc @AndriySvyryd in case you want to take a quick look for the metadata side and Cosmos container building part.

[roji]

@maumar see batch of comments.

I think we should maybe discuss the FTS property vs. index question in design...

[maumar]

@roji @AndriySvyryd I updated the PR addressing the Property/Index API and fixing a few small errors. You can review just the second commit - all the changes are there.

• I put the HasDefaultFullTextLanguage setting on the entity type builder, rather than creating a container builder just for it. We already have some container-level settings (e.g. TTL) and we defined them on entity type builder, so going for consistency here.
• Added tests to make sure values are propagated correctly (if property specifies language, use that, otherwise use default, otherwise use en-US). These tests need to be adjusted once cosmos supports more languages, for now they all just throw Cosmos exception complaining about not supported language. I manually verified that we wire-up things correctly when creating the container.
• Added test for using full text property without the index and removed validation preventing this scenario.
• Conditionally skipping the tests when running on live service due to Azure.ResourceManager.CosmosDb not supporting FTS.

[maumar]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).