feat: Allow tokenizers to accept arbitrary options

[isaacvando]

Ticket(s) Closed

Updates the tokenizers to accept explicit positional arguments (like int minGram, etc) and a dictionary of arbitrary options. This matches what we have in other ORMs and prevents us from having to wrap all of the many options each tokenizer can accept. Provides less type-safety than an explicit wrapping, but it makes the maintenance burden much lighter and it removes the possibility of a user ever needing an option that we hadn't exposed yet. The old API would have needed to get much larger to support all of the options (like alias) too.

What

Why

How

Tests

[codecov]

Codecov Report

❌ Patch coverage is 90.24390% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.25%. Comparing base (0959d50) to head (df9d7c6).

Files with missing lines	Patch %	Lines
src/ParadeDB.EntityFrameworkCore/Tokenizer.cs	88.57%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #49      +/-   ##
==========================================
- Coverage   91.72%   87.25%   -4.47%     
==========================================
  Files          30       29       -1     
  Lines         423      361      -62     
  Branches       37       37              
==========================================
- Hits          388      315      -73     
- Misses         27       34       +7     
- Partials        8       12       +4     

Flag	Coverage Δ
net10.0	87.25% <90.24%> (-4.47%)	⬇️
net8.0	87.25% <90.24%> (-4.47%)	⬇️
net9.0	87.25% <90.24%> (-4.47%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...Query/ParadeDbEvaluatableExpressionFilterPlugin.cs	95.65% <100.00%> (+1.53%)	⬆️
src/ParadeDB.EntityFrameworkCore/Tokenizer.cs	91.11% <88.57%> (-8.89%)	⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.