Skip to content

Commit 91686e9

Browse files
pomegranitedregisb
pomegranited
and
regisb
authored
feat: add Meilisearch-compatible search engine with tests (#164)
The goal of this change is to introduce a search engine that is compatible with the edx-search API but that uses Meilisearch instead of Elasticsearch. That way, we can replace one by the other across edx-platform by simply changing a single SEARCH_ENGINE django setting. There are a couple of differences between Meilisearch and Elasticsearch: 1. Filterable attributes must be defined explicitly. 2. No support for datetime objects, which must be converted to timestamps (with an extra field to store the timezone). 3. No special characters allowed in the primary key values, such that we must hash course IDs before we can use them as primary key values. Note that this PR does not introduce any breaking change. This is an opt-in engine that anyone is free to use. There is some setup work for every search feature: see the engine module documentation for more information. See the corresponding conversation here: openedx/frontend-app-authoring#1334 (comment) * fix: `make test` command Unit test command was failing because manage.py was pointing to an incorrect module by default. * feat: add convenient "make compile-requirements" command This command makes it possible to compile requirements without upgrading them. Note that the `make upgrade` command still works with this change. --------- Co-authored-by: Régis Behmo <[email protected]>
1 parent 8acaa21 commit 91686e9

File tree

12 files changed

+1079
-145
lines changed

12 files changed

+1079
-145
lines changed

‎Makefile‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -34,23 +34,26 @@ test_with_es: clean test.start_elasticsearch
3434
coverage run --source='.' manage.py test
3535
make test.stop_elasticsearch
3636

37-
upgrade: export CUSTOM_COMPILE_COMMAND=make upgrade
38-
upgrade: ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
37+
compile-requirements: export CUSTOM_COMPILE_COMMAND=make upgrade
38+
compile-requirements: ## Re-compile *.in requirements to *.txt (without upgrading)
3939
pip install -qr requirements/pip-tools.txt
4040
# Make sure to compile files after any other files they include!
4141
pip-compile --rebuild --allow-unsafe --rebuild -o requirements/pip.txt requirements/pip.in
42-
pip-compile --rebuild --upgrade -o requirements/pip-tools.txt requirements/pip-tools.in
42+
pip-compile --rebuild ${COMPILE_OPTS} -o requirements/pip-tools.txt requirements/pip-tools.in
4343
pip install -qr requirements/pip.txt
4444
pip install -qr requirements/pip-tools.txt
45-
pip-compile --rebuild --upgrade -o requirements/base.txt requirements/base.in
46-
pip-compile --rebuild --upgrade -o requirements/testing.txt requirements/testing.in
47-
pip-compile --rebuild --upgrade -o requirements/quality.txt requirements/quality.in
48-
pip-compile --rebuild --upgrade -o requirements/ci.txt requirements/ci.in
49-
pip-compile --rebuild --upgrade -o requirements/dev.txt requirements/dev.in
45+
pip-compile --rebuild ${COMPILE_OPTS} -o requirements/base.txt requirements/base.in
46+
pip-compile --rebuild ${COMPILE_OPTS} -o requirements/testing.txt requirements/testing.in
47+
pip-compile --rebuild ${COMPILE_OPTS} -o requirements/quality.txt requirements/quality.in
48+
pip-compile --rebuild ${COMPILE_OPTS} -o requirements/ci.txt requirements/ci.in
49+
pip-compile --rebuild ${COMPILE_OPTS} -o requirements/dev.txt requirements/dev.in
5050
# Let tox control the Django version for tests
5151
sed '/^[dD]jango==/d' requirements/testing.txt > requirements/testing.tmp
5252
mv requirements/testing.tmp requirements/testing.txt
5353

54+
upgrade: ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
55+
$(MAKE) compile-requirements COMPILE_OPTS="--upgrade"
56+
5457
test: test_with_es ## run tests and generate coverage report
5558

5659
install-local: ## installs your local edx-search into the LMS and CMS python virtualenvs

‎edxsearch/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
""" Container module for testing / demoing search """
22

3-
__version__ = '4.0.0'
3+
__version__ = '4.1.0'

‎edxsearch/settings.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@
5353
'django.contrib.sessions',
5454
'django.contrib.messages',
5555
'django.contrib.staticfiles',
56+
'eventtracking.django',
5657
'waffle',
5758
)
5859

@@ -99,3 +100,31 @@
99100
# https://docs.djangoproject.com/en/1.6/howto/static-files/
100101

101102
STATIC_URL = '/static/'
103+
104+
# EVENT TRACKING #################################
105+
106+
TRACK_MAX_EVENT = 50000
107+
108+
TRACKING_BACKENDS = {
109+
'logger': {
110+
'ENGINE': 'track.backends.logger.LoggerBackend',
111+
'OPTIONS': {
112+
'name': 'tracking'
113+
}
114+
}
115+
}
116+
117+
# We're already logging events, and we don't want to capture user
118+
# names/passwords. Heartbeat events are likely not interesting.
119+
TRACKING_IGNORE_URL_PATTERNS = [r'^/event', r'^/login', r'^/heartbeat']
120+
121+
EVENT_TRACKING_ENABLED = True
122+
EVENT_TRACKING_BACKENDS = {
123+
'logger': {
124+
'ENGINE': 'eventtracking.backends.logger.LoggerBackend',
125+
'OPTIONS': {
126+
'name': 'tracking',
127+
'max_event_size': TRACK_MAX_EVENT,
128+
}
129+
}
130+
}

‎requirements/base.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,4 @@ Django # Web application framework
1515
elasticsearch>=7.8.0,<8.0.0
1616
edx-toggles
1717
event-tracking
18+
meilisearch

‎requirements/base.txt‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,26 @@
66
#
77
amqp==5.2.0
88
# via kombu
9+
annotated-types==0.7.0
10+
# via pydantic
911
asgiref==3.8.1
1012
# via django
1113
attrs==23.2.0
1214
# via openedx-events
1315
billiard==4.2.0
1416
# via celery
17+
camel-converter[pydantic]==4.0.1
18+
# via meilisearch
1519
celery==5.4.0
1620
# via event-tracking
1721
certifi==2024.7.4
18-
# via elasticsearch
22+
# via
23+
# elasticsearch
24+
# requests
1925
cffi==1.16.0
2026
# via pynacl
27+
charset-normalizer==3.4.0
28+
# via requests
2129
click==8.1.7
2230
# via
2331
# celery
@@ -77,12 +85,16 @@ event-tracking==2.4.0
7785
# -r requirements/base.in
7886
fastavro==1.9.5
7987
# via openedx-events
88+
idna==3.10
89+
# via requests
8090
jinja2==3.1.4
8191
# via code-annotations
8292
kombu==5.3.7
8393
# via celery
8494
markupsafe==2.1.5
8595
# via jinja2
96+
meilisearch==0.31.5
97+
# via -r requirements/base.in
8698
newrelic==9.12.0
8799
# via edx-django-utils
88100
openedx-events==9.11.0
@@ -95,6 +107,10 @@ psutil==6.0.0
95107
# via edx-django-utils
96108
pycparser==2.22
97109
# via cffi
110+
pydantic==2.9.2
111+
# via camel-converter
112+
pydantic-core==2.23.4
113+
# via pydantic
98114
pymongo==3.13.0
99115
# via
100116
# edx-opaque-keys
@@ -109,6 +125,8 @@ pytz==2024.1
109125
# via event-tracking
110126
pyyaml==6.0.1
111127
# via code-annotations
128+
requests==2.32.3
129+
# via meilisearch
112130
six==1.16.0
113131
# via
114132
# edx-ccx-keys
@@ -124,11 +142,16 @@ stevedore==5.2.0
124142
text-unidecode==1.3
125143
# via python-slugify
126144
typing-extensions==4.12.2
127-
# via edx-opaque-keys
145+
# via
146+
# edx-opaque-keys
147+
# pydantic
148+
# pydantic-core
128149
tzdata==2024.1
129150
# via celery
130151
urllib3==1.26.19
131-
# via elasticsearch
152+
# via
153+
# elasticsearch
154+
# requests
132155
vine==5.1.0
133156
# via
134157
# amqp

‎requirements/dev.txt‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,11 @@ amqp==5.2.0
99
# -r requirements/quality.txt
1010
# -r requirements/testing.txt
1111
# kombu
12+
annotated-types==0.7.0
13+
# via
14+
# -r requirements/quality.txt
15+
# -r requirements/testing.txt
16+
# pydantic
1217
asgiref==3.8.1
1318
# via
1419
# -r requirements/quality.txt
@@ -37,6 +42,11 @@ cachetools==5.4.0
3742
# via
3843
# -r requirements/ci.txt
3944
# tox
45+
camel-converter[pydantic]==4.0.1
46+
# via
47+
# -r requirements/quality.txt
48+
# -r requirements/testing.txt
49+
# meilisearch
4050
celery==5.4.0
4151
# via
4252
# -r requirements/quality.txt
@@ -47,6 +57,7 @@ certifi==2024.7.4
4757
# -r requirements/quality.txt
4858
# -r requirements/testing.txt
4959
# elasticsearch
60+
# requests
5061
cffi==1.16.0
5162
# via
5263
# -r requirements/quality.txt
@@ -56,6 +67,11 @@ chardet==5.2.0
5667
# via
5768
# -r requirements/ci.txt
5869
# tox
70+
charset-normalizer==3.4.0
71+
# via
72+
# -r requirements/quality.txt
73+
# -r requirements/testing.txt
74+
# requests
5975
click==8.1.7
6076
# via
6177
# -r requirements/pip-tools.txt
@@ -185,6 +201,11 @@ filelock==3.15.4
185201
# -r requirements/ci.txt
186202
# tox
187203
# virtualenv
204+
idna==3.10
205+
# via
206+
# -r requirements/quality.txt
207+
# -r requirements/testing.txt
208+
# requests
188209
iniconfig==2.0.0
189210
# via
190211
# -r requirements/quality.txt
@@ -213,6 +234,10 @@ mccabe==0.7.0
213234
# via
214235
# -r requirements/quality.txt
215236
# pylint
237+
meilisearch==0.31.5
238+
# via
239+
# -r requirements/quality.txt
240+
# -r requirements/testing.txt
216241
mock==5.1.0
217242
# via
218243
# -r requirements/quality.txt
@@ -275,6 +300,16 @@ pycparser==2.22
275300
# -r requirements/quality.txt
276301
# -r requirements/testing.txt
277302
# cffi
303+
pydantic==2.9.2
304+
# via
305+
# -r requirements/quality.txt
306+
# -r requirements/testing.txt
307+
# camel-converter
308+
pydantic-core==2.23.4
309+
# via
310+
# -r requirements/quality.txt
311+
# -r requirements/testing.txt
312+
# pydantic
278313
pylint==3.2.6
279314
# via
280315
# -r requirements/quality.txt
@@ -344,6 +379,11 @@ pyyaml==6.0.1
344379
# -r requirements/quality.txt
345380
# -r requirements/testing.txt
346381
# code-annotations
382+
requests==2.32.3
383+
# via
384+
# -r requirements/quality.txt
385+
# -r requirements/testing.txt
386+
# meilisearch
347387
six==1.16.0
348388
# via
349389
# -r requirements/quality.txt
@@ -380,6 +420,8 @@ typing-extensions==4.12.2
380420
# -r requirements/quality.txt
381421
# -r requirements/testing.txt
382422
# edx-opaque-keys
423+
# pydantic
424+
# pydantic-core
383425
tzdata==2024.1
384426
# via
385427
# -r requirements/quality.txt
@@ -390,6 +432,7 @@ urllib3==1.26.19
390432
# -r requirements/quality.txt
391433
# -r requirements/testing.txt
392434
# elasticsearch
435+
# requests
393436
vine==5.1.0
394437
# via
395438
# -r requirements/quality.txt

‎requirements/quality.txt‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ amqp==5.2.0
88
# via
99
# -r requirements/testing.txt
1010
# kombu
11+
annotated-types==0.7.0
12+
# via
13+
# -r requirements/testing.txt
14+
# pydantic
1115
asgiref==3.8.1
1216
# via
1317
# -r requirements/testing.txt
@@ -24,6 +28,10 @@ billiard==4.2.0
2428
# via
2529
# -r requirements/testing.txt
2630
# celery
31+
camel-converter[pydantic]==4.0.1
32+
# via
33+
# -r requirements/testing.txt
34+
# meilisearch
2735
celery==5.4.0
2836
# via
2937
# -r requirements/testing.txt
@@ -32,10 +40,15 @@ certifi==2024.7.4
3240
# via
3341
# -r requirements/testing.txt
3442
# elasticsearch
43+
# requests
3544
cffi==1.16.0
3645
# via
3746
# -r requirements/testing.txt
3847
# pynacl
48+
charset-normalizer==3.4.0
49+
# via
50+
# -r requirements/testing.txt
51+
# requests
3952
click==8.1.7
4053
# via
4154
# -r requirements/testing.txt
@@ -130,6 +143,10 @@ fastavro==1.9.5
130143
# via
131144
# -r requirements/testing.txt
132145
# openedx-events
146+
idna==3.10
147+
# via
148+
# -r requirements/testing.txt
149+
# requests
133150
iniconfig==2.0.0
134151
# via
135152
# -r requirements/testing.txt
@@ -150,6 +167,8 @@ markupsafe==2.1.5
150167
# jinja2
151168
mccabe==0.7.0
152169
# via pylint
170+
meilisearch==0.31.5
171+
# via -r requirements/testing.txt
153172
mock==5.1.0
154173
# via -r requirements/testing.txt
155174
newrelic==9.12.0
@@ -188,6 +207,14 @@ pycparser==2.22
188207
# via
189208
# -r requirements/testing.txt
190209
# cffi
210+
pydantic==2.9.2
211+
# via
212+
# -r requirements/testing.txt
213+
# camel-converter
214+
pydantic-core==2.23.4
215+
# via
216+
# -r requirements/testing.txt
217+
# pydantic
191218
pylint==3.2.6
192219
# via
193220
# edx-lint
@@ -233,6 +260,10 @@ pyyaml==6.0.1
233260
# via
234261
# -r requirements/testing.txt
235262
# code-annotations
263+
requests==2.32.3
264+
# via
265+
# -r requirements/testing.txt
266+
# meilisearch
236267
six==1.16.0
237268
# via
238269
# -r requirements/testing.txt
@@ -260,6 +291,8 @@ typing-extensions==4.12.2
260291
# via
261292
# -r requirements/testing.txt
262293
# edx-opaque-keys
294+
# pydantic
295+
# pydantic-core
263296
tzdata==2024.1
264297
# via
265298
# -r requirements/testing.txt
@@ -268,6 +301,7 @@ urllib3==1.26.19
268301
# via
269302
# -r requirements/testing.txt
270303
# elasticsearch
304+
# requests
271305
vine==5.1.0
272306
# via
273307
# -r requirements/testing.txt

0 commit comments

Comments
 (0)