Add __like and __ilike filters (#421)

[simoncampion]

Adds support for __like and __ilike filters (#421)

Description

Adds support for filtering by arbitrary LIKE patterns. For example:

await User.filter(name__like='J_hn%')

I made three potentially contentious implementation decisions:

1. The caller needs to escape % and _ with backslashes in the pattern string if they desire to match those characters literally. If they are not escaped, they are treated as SQL wildcards. The caller does not need to escape \ to match it literally. As far as I can see, this is consistent with how __contains works, for example. It also escapes \ on behalf of the caller.
2. I implemented __ilike using UPPER + LIKE. Therefore, it works with flavors of SQL that do not support the ILIKE operator, such as MySQL.
3. The LIKE operator, and therefore __like, at least in the current implementation, is case-sensitive or case-insensitive depending on the flavor of SQL that is used. If desired, this might be changed, so that __like is guaranteed to be case-sensitive independently of the underlying database. I did not try to enforce case-sensitivity because LIKE queries on the database and __like filters in Tortoise would yield different results, which might be confusing.

Motivation and Context

Fixes #421
(The issue is assigned to someone else but has not been worked on for over a year. I hope it's fine that I opened a PR addressing it.)

How Has This Been Tested?

I added tests in test_filters.py. Someone with a good understanding of SQL injections should take a look at the tests and let me know if there are further tests I should add.

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added the changelog accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

The two tests tests/test_default.py::TestDefault::test_default and tests/fields/test_time.py::TestDatetimeFields::test_update fail when running make test_postgres locally on my machine. However, that is the case on the develop branch as well and seems unrelated to the changes I made.

This is my first contribution to Tortoise ORM. Please let me know if there are more elegant ways to implement LIKE filters using existing functionality in the code base that I might not be familiar with.

[long2ice]

__startswith, __endswith is not enough?

[simoncampion]

__like allows for more powerful filters than __startswith and __endswith (see the linked issue for an example). The question is whether users will value these more powerful filters sufficiently to justify adding support for them. My sense is that they will, but there's room for reasonable disagreement about this.

(FWIW, @abondar and @grigi seem to have thought the feature was worth adding.)

[MLBZ521]

I'd vote for this feature -- I could utilize it in my project.

[MLBZ521]

I figured out how to use and combined Q expressions to get a somewhat similar functionality for now.

For anyone else that may stumble across this, here's what I did:

core.send.py

from tortoise.expressions import Q
import core
[...]

async def policy_list(filter_values: str, username: str):

	user_object = await core.user.get({"username": username})

	q_expression = Q()

	for filter_value in filter_values.split(" "):
		q_expression &= Q(name__icontains=filter_value)

	if not user_object.full_admin:
		sites = user_object.site_access.split(", ")
		q_expression &= Q(site__in=sites)

	policies_object = await core.policy.get(q_expression)

core.policy.py

from tortoise.expressions import Q
import core
[...]

async def get(policy_filter: dict | Q | None = None):

	if not policy_filter:
		return await models.Policies.all()
	elif isinstance(policy_filter, dict):
		results = await models.Policy_Out.from_queryset(models.Policies.filter(**policy_filter))
	elif isinstance(policy_filter, Q):
		results = await models.Policy_Out.from_queryset(models.Policies.filter(policy_filter))

	return results[0] if len(results) == 1 else results

[waketzheng]

@simoncampion Please fix conflicts.

[simoncampion]

@waketzheng Sorry for the late response. I'm currently traveling, but I'll be able to take a look at the end of this month.

[simoncampion]

@waketzheng It looks like you've already fixed the conflicts. Let me know if there's anything else you need me to do.

Aside: It seems to me that this commit you added to the PR (and possibly other commits---I haven't checked all of them) aren't necessary to resolve merge conflicts and unrelated to the functionality added by the PR. Am I missing something? Or did you not mean to add these commits to the PR?

[waketzheng]

@waketzheng It looks like you've already fixed the conflicts. Let me know if there's anything else you need me to do.

Aside: It seems to me that this commit you added to the PR (and possibly other commits---I haven't checked all of them) aren't necessary to resolve merge conflicts and unrelated to the functionality added by the PR. Am I missing something? Or did you not mean to add these commits to the PR?

TODO List

• sqlite
• postgres
• mysql
• mssql
• oracle
• refactor: reduce duplicated code

mssql/oracle does not pass unittest yet, cloud you help to do that?

[simoncampion]

@waketzheng I see that you forced __like to be case-sensitive for MySQL with this change. To clarify, would you prefer that __like is always case-sensitive for all database types, even if the equivalent LIKE statement when executed against the database wouldn't be case-sensitive?

Also see my note in the PR description:

The LIKE operator, and therefore __like, at least in the current implementation, is case-sensitive or case-insensitive depending on the flavor of SQL that is used. If desired, this might be changed, so that __like is guaranteed to be case-sensitive independently of the underlying database. I did not try to enforce case-sensitivity because LIKE queries on the database and __like filters in Tortoise would yield different results, which might be confusing.

[simoncampion]

I fixed the failing mssql tests.

I spent some time trying to get oracle tests to run on my local machine in a dockerized setup, as I'd prefer not to install dependencies for those tests on my machine. Unfortunately, I haven't yet managed to get them to run. Is there documentation on how to run the oracle tests somewhere? I didn't find anything here.

[codspeed-hq]

CodSpeed Performance Report

Merging #954 will not alter performance

_{Comparing simoncampion:add-like-and-ilike (df48fc1) with develop (4b4e4a5)}

Summary

✅ 16 untouched benchmarks

[waketzheng]

@waketzheng I see that you forced __like to be case-sensitive for MySQL with this change. To clarify, would you prefer that __like is always case-sensitive for all database types, even if the equivalent LIKE statement when executed against the database wouldn't be case-sensitive?

Also see my note in the PR description:

The LIKE operator, and therefore __like, at least in the current implementation, is case-sensitive or case-insensitive depending on the flavor of SQL that is used. If desired, this might be changed, so that __like is guaranteed to be case-sensitive independently of the underlying database. I did not try to enforce case-sensitivity because LIKE queries on the database and __like filters in Tortoise would yield different results, which might be confusing.

For me, most of the time, I don't care the SQL statement and db type. I just want __like to give me the case-sensitive result and __ilike return case-insensitive result.
May be we can add description in document to dispel confusion.