feat: Introduce function-based API for tokenizers

[isaacvando]

Ticket(s) Closed

• Closes Schema compat failure for ParadeDB v0.23.0 #109

What

• Adds support for the icu and edge_ngram tokenizers.
• Introduces new function-based API for tokenizers in indexing and querying:

Match("running shoes", operator="AND", tokenizer=Tokenizer.ngram(3,3, options={"positions": True}))

Why

How

Tests

I added a test to explicitly exercise all of the tokenizers.

[philippemnoel]

As discussed in Slack, we want to support the edge_ngram tokenizer and add tests for all tokenizers

[codecov]

Codecov Report

❌ Patch coverage is 87.50000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.94%. Comparing base (ce16f23) to head (3bf6d3c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
paradedb/search.py	88.11%	8 Missing and 4 partials ⚠️
paradedb/indexes.py	85.71%	1 Missing and 1 partial ⚠️
paradedb/api.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
- Coverage   91.55%   90.94%   -0.61%     
==========================================
  Files          11       11              
  Lines        1397     1370      -27     
  Branches      303      287      -16     
==========================================
- Hits         1279     1246      -33     
- Misses         68       76       +8     
+ Partials       50       48       -2     

Flag	Coverage Δ
dj4.2	90.94% <87.50%> (-0.61%)	⬇️
dj5.2	90.94% <87.50%> (-0.61%)	⬇️
dj6.0	90.94% <87.50%> (-0.61%)	⬇️
django-paradedb	90.94% <87.50%> (-0.61%)	⬇️
pg15	90.94% <87.50%> (-0.61%)	⬇️
pg16	90.94% <87.50%> (-0.61%)	⬇️
pg17	90.94% <87.50%> (-0.61%)	⬇️
pg18	90.94% <87.50%> (-0.61%)	⬇️
py3.10	90.94% <87.50%> (-0.61%)	⬇️
py3.11	90.94% <87.50%> (-0.61%)	⬇️
py3.12	90.94% <87.50%> (-0.61%)	⬇️
py3.13	90.94% <87.50%> (-0.61%)	⬇️
py3.14	90.94% <87.50%> (-0.61%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
paradedb/__init__.py	85.00% <ø> (ø)
paradedb/api.py	74.19% <80.00%> (ø)
paradedb/indexes.py	82.75% <85.71%> (-5.83%)	⬇️
paradedb/search.py	93.58% <88.11%> (-0.25%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[philippemnoel]

Match("running shoes", operator="AND", tokenizer=Tokenizer.ngram(3,3, positions=true))

I quite like this new proposed API. Would want @rebasedming to sign off on it, but I'd be in favour. Would we also make this change to the other ORMs, or is this Django-specific?

[isaacvando]

Cool! One thought I had is that it would probably be better to use a dictionary for named args so that we don't have to try to keep all of them in sync; I expect that would be painful. So it would look something like this instead:

Match("running shoes", operator="AND", tokenizer=Tokenizer.ngram(3,3, {"positions": "true"}))

I realized after proposing this that SQLAlchemy has already had a similar API for specifying tokenizers for index creation specifically. But it isn't used in querying. So I went ahead and standardized use of that in my other PR.

I haven't looked closely at that part in rails, but I'm sure we could do something similar.

[philippemnoel]

Cool! One thought I had is that it would probably be better to use a dictionary for named args so that we don't have to try to keep all of them in sync; I expect that would be painful. So it would look something like this instead:

Match("running shoes", operator="AND", tokenizer=Tokenizer.ngram(3,3, {"positions": "true"}))

I realized after proposing this that SQLAlchemy has already had a similar API for specifying tokenizers for index creation specifically. But it isn't used in querying. So I went ahead and standardized use of that in my other PR.

I haven't looked closely at that part in rails, but I'm sure we could do something similar.

Even then, this looks better to me than the current situation. However, would it really be that painful? Tokenizers don't change much. We sometimes expose new fields from existing tokenizers, but rarely add new tokenizers now. I don't think it'd be that painful. Perhaps I'm understimating the issue or missing something?

[isaacvando]

Even then, this looks better to me than the current situation. However, would it really be that painful? Tokenizers don't change much. We sometimes expose new fields from existing tokenizers, but rarely add new tokenizers now. I don't think it'd be that painful. Perhaps I'm understimating the issue or missing something?

If they did change often, I could see it being hard to keep all the changes in sync as we don't have automated detection for that kind of change right now. It would be annoying for users to try to use a particular field and then find we hadn't added it yet. But, if they don't change very often then it seems fine to expose all of the arguments.

It looks like each tokenizer has ~15 named args. Do we want to expose all of them in the function signatures?

select 'hello world'::pdb.simple('invalid=arg')::text[];
ERROR:  Invalid option: 'invalid'. Allowed options: alias, alpha_num_only, ascii_folding, b, columnar, fieldnorms, k1, lowercase, normalizer, remove_long, remove_short, stemmer, stopwords, stopwords_language, trim.

Do all of these look up to date to you? I tried using stopwords but I couldn't get it to work and I don't see it being used anywhere in the codebase with the new syntax.

Seeing that each tokenizer has so many arguments is another reason I was initially hesitant to add them all.

[philippemnoel]

Even then, this looks better to me than the current situation. However, would it really be that painful? Tokenizers don't change much. We sometimes expose new fields from existing tokenizers, but rarely add new tokenizers now. I don't think it'd be that painful. Perhaps I'm understimating the issue or missing something?

If they did change often, I could see it being hard to keep all the changes in sync as we don't have automated detection for that kind of change right now. It would be annoying for users to try to use a particular field and then find we hadn't added it yet. But, if they don't change very often then it seems fine to expose all of the arguments.

It looks like each tokenizer has ~15 named args. Do we want to expose all of them in the function signatures?

select 'hello world'::pdb.simple('invalid=arg')::text[];
ERROR:  Invalid option: 'invalid'. Allowed options: alias, alpha_num_only, ascii_folding, b, columnar, fieldnorms, k1, lowercase, normalizer, remove_long, remove_short, stemmer, stopwords, stopwords_language, trim.

Do all of these look up to date to you? I tried using stopwords but I couldn't get it to work and I don't see it being used anywhere in the codebase with the new syntax.

Seeing that each tokenizer has so many arguments is another reason I was initially hesitant to add them all.

Oh man yeah that might be a lot then... I'll defer to Ming on this

[isaacvando]

Thinking about this more, accepting a dictionary of named arguments instead of wrapping all of them seems like the way to go here to me.

[philippemnoel]

Thinking about this more, accepting a dictionary of named arguments instead of wrapping all of them seems like the way to go here to me.

Sounds good to me.