[release/9.0-staging] Cosmos Full Text Search support (query part) #35909
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #35476 Fixes #35853 (need to fix this one, otherwise vector translator will try to translate full text methods and fail) Description This PR enables full-text search queries using EF Core 9 when targeting Azure Cosmos Db. This is one of the flagship new features for Cosmos and the aim here is to help with it's adoption. This is very limited port of the full feature we are adding in EF 10 Preview 4. We are only adding querying capabilities: function stubs for FullTextContains, FullTextContainsAll, FullTextContainsAny, FullTextScore and RRF as well as logic translating these signatures to built-in Cosmos functions. No model building or data manipulation - containers need to be created outside EF (using Cosmos SDK or in the Data Explorer). Customer impact Customers will be able to use the upcoming full text search capabilities when working with Azure Cosmos Db without the need to upgrade to EF 10 preview. How found Partner team ask. Regression No Testing Extensively tested on EF 10, manual testing on EF9. End-to-end testing is not possible because we can't create containers programmatically (no support for it inside EF Core itself, and the Cosmos SDK which supports it is currently only available in beta, so we can't take dependency on it). Instead, we created containers and data using EF 10, ported all the query tests from EF 10 and ran them using the EF9 bits. Risk Low. Code here is purely additive and actually localized to only a handful of places in the code: validation in ApplyOrdering/AppendOrdering, FTS method translator, parameter inliner and sql generator. Feature is marked as experimental and quirks have been added.
There was a problem hiding this comment.
Copilot reviewed 11 out of 14 changed files in this pull request and generated no comments.
Files not reviewed (3)
- src/EFCore.Cosmos/EFCore.Cosmos.csproj: Language not supported
- src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported
- src/EFCore.Cosmos/Properties/CosmosStrings.resx: Language not supported
Comments suppressed due to low confidence (1)
src/EFCore.Cosmos/Query/Internal/Translators/CosmosVectorSearchTranslator.cs:44
- The conditional check in the legacy branch uses '&&' instead of '||', which may lead to unexpected behavior when only one of the conditions fails. Consider revising the condition to use '||' for consistency with the new behavior branch.
else if (method.DeclaringType != typeof(CosmosDbFunctionsExtensions) && method.Name != nameof(CosmosDbFunctionsExtensions.VectorDistance))
AndriySvyryd
approved these changes
Apr 9, 2025
|
@AndriySvyryd new version up |
maumar
force-pushed
the
cosmos_fts_90
branch
2 times, most recently
from
a5035dc to
fad7f74
Compare
change TFScore signature to accept params, fixed SqlFunctionExpression to use the new ctor internally, added mandatory property overrides for FragmentExpression (vector search) fixed parameter inliner to not match parameter of type string, but only string[] when processing ContainsAll/Any
AndriySvyryd
approved these changes
Apr 14, 2025
maumar
added a commit
that referenced
this pull request
Apr 14, 2025
…rch experience Continuation of #35909 Bumping version of Cosmos SDK improves user experience out of the box for some scenarios (namely hybrid search). Separated this from the main PR as it required separate approval.
maumar
added a commit
that referenced
this pull request
Apr 15, 2025
… hybrid search experience out of the box bumping cosmos sdk version to improve hybrid search experience out of the box - this is a dry run, will incorporate these changes to the proper 9.0 PR once/if we have green light for the bump Continuation of #35909 Description As part of adding full text search support we realized that some scenarios (specifically hybrid search and some instances of ContainsAll) don't work with the Cosmos SDK that is currently referenced by EFCore.Cosmos 9.x package. The issues can be solved by bumping SDK version to 3.48 Customer impact Customers trying full text search will experience errors for some scenarios - cryptic exception (Syntax error near '-' for hybrid, and invalid results for some ContainsAll queries). Workaround is to manually upgrade Cosmos SDK dependency. How found Internal testing by the partner team. Regression No Testing All current Cosmos tests pass. Performed smoke test to validate that breaking change added in the SDK (requiring explicit Newtonsoft.Json dependency) doesn't have impact on the customer app. Risk Low. This can only affect customers using EFCore.Cosmos, one breaking change introduced in the SDK is benign from the perspective of customers using EFCore to communicate with Cosmos (EFCore package absorbs the breaking change).
maumar
added a commit
that referenced
this pull request
Apr 15, 2025
… hybrid search experience out of the box (#35954) bumping cosmos sdk version to improve hybrid search experience out of the box - this is a dry run, will incorporate these changes to the proper 9.0 PR once/if we have green light for the bump Continuation of #35909 Description As part of adding full text search support we realized that some scenarios (specifically hybrid search and some instances of ContainsAll) don't work with the Cosmos SDK that is currently referenced by EFCore.Cosmos 9.x package. The issues can be solved by bumping SDK version to 3.48 Customer impact Customers trying full text search will experience errors for some scenarios - cryptic exception (Syntax error near '-' for hybrid, and invalid results for some ContainsAll queries). Workaround is to manually upgrade Cosmos SDK dependency. How found Internal testing by the partner team. Regression No Testing All current Cosmos tests pass. Performed smoke test to validate that breaking change added in the SDK (requiring explicit Newtonsoft.Json dependency) doesn't have impact on the customer app. Risk Low. This can only affect customers using EFCore.Cosmos, one breaking change introduced in the SDK is benign from the perspective of customers using EFCore to communicate with Cosmos (EFCore package absorbs the breaking change).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Limited port of #35868
Fixes #35476
Fixes #35853 (need to fix this one, otherwise vector translator will try to translate full text methods and fail)
Description
This PR enables full-text search queries using EF Core 9 when targeting Azure Cosmos Db. This is one of the flagship new features for Cosmos and the aim here is to help with it's adoption. This is very limited port of the full feature we are adding in EF 10 Preview 4. We are only adding querying capabilities: function stubs for FullTextContains, FullTextContainsAll, FullTextContainsAny, FullTextScore and RRF as well as logic translating these signatures to built-in Cosmos functions. No model building or data manipulation - containers need to be created outside EF (using Cosmos SDK or in the Data Explorer).
Customer impact
Customers will be able to use the upcoming full text search capabilities when working with Azure Cosmos Db without the need to upgrade to EF 10 preview.
How found
Partner team ask.
Regression
No
Testing
Extensively tested on EF 10, manual testing on EF9. End-to-end testing is not possible because we can't create containers programmatically (no support for it inside EF Core itself, and the Cosmos SDK which supports it is currently only available in beta, so we can't take dependency on it). Instead, we created containers and data using EF 10, ported all the query tests from EF 10 and ran them using the EF9 bits.
Risk
Low. Code here is purely additive and actually localized to only a handful of places in the code: validation in ApplyOrdering/AppendOrdering, FTS method translator, parameter inliner and sql generator. Feature is marked as experimental and quirks have been added.