[ENH] For local: use subquery for FTS, unions for int & float metadata expr, is true -> is not null

[jairad26]

Description of changes

Move FTS to subquery approach for improved performance, and use unions in the case of int and float metadata, and replace is true uses with is not null to utilize db indexes
before (0.6.3):

after:

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[jairad26]

• [ENH] For local: use subquery for FTS, unions for int & float metadata expr, is true -> is not null #4556 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Performance Improvements for Local Chroma Queries

This PR implements several query optimization techniques for local Chroma databases. It moves full-text search (FTS) to a subquery approach, implements union queries for int and float metadata to better utilize indexes, and replaces 'is true' conditions with 'is not null' to take advantage of database indexing. These changes result in significant performance improvements as shown in the before/after benchmarks provided in the PR description.

Key Changes:
• Replaced direct FTS queries with more efficient subqueries
• Implemented union-based approach for int/float metadata queries
• Changed 'is true' conditions to 'is not null' for better index utilization
• Removed redundant left joins in the query construction

Affected Areas:
• rust/segment/src/sqlite_metadata.rs

This summary was automatically generated by @propel-code-bot

[HammadB]

Suggest adding a few coverage tests for common edge cases

• Metadata key null for one record, exists for another
• FTS match but no metadata match

etc