Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 20 additions & 12 deletions .github/workflows/update-dev-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,8 @@ jobs:
- name: Setup Node.js
uses: actions/setup-node@v5
with:
node-version: '24'
cache: 'npm'
node-version: "24"
cache: "npm"

- name: Transpile markdown files to MDX
run: |
Expand All @@ -151,38 +151,46 @@ jobs:
npx fumadocs-transpiler@latest --input docs/product-opener --output "content/docs/Product-Opener/(docs)" --verbose
echo "Product Opener documentation transpiled successfully"
fi

# Uncomment and modify these sections when other repositories are enabled
# echo "Transpiling Robotoff documentation..."
# if [ -d "docs/robotoff" ]; then
# npx fumadocs-transpiler@latest --input docs/robotoff --output "content/docs/Robotoff/(docs)" --verbose
# echo "Robotoff documentation transpiled successfully"
# fi

# echo "Transpiling Infrastructure documentation..."
# if [ -d "docs/infrastructure" ]; then
# npx fumadocs-transpiler@latest --input docs/infrastructure --output "content/docs/infrastructure/(docs)" --verbose
# echo "Infrastructure documentation transpiled successfully"
# fi

# echo "Transpiling Open Prices documentation..."
# if [ -d "docs/open-prices" ]; then
# npx fumadocs-transpiler@latest --input docs/open-prices --output "content/docs/Open-prices/(docs)" --verbose
# echo "Open Prices documentation transpiled successfully"
# fi

# echo "Transpiling Knowledge Panels documentation..."
# if [ -d "docs/knowledge-panels" ]; then
# npx fumadocs-transpiler@latest --input docs/knowledge-panels --output "content/docs/Knowledge-Panel/(docs)" --verbose
# echo "Knowledge Panels documentation transpiled successfully"
# fi

# echo "Transpiling Search-a-licious documentation..."
# if [ -d "docs/search-a-licious" ]; then
# npx fumadocs-transpiler@latest --input docs/search-a-licious --output "content/docs/search-a-licious/(docs)" --verbose
# echo "Search-a-licious documentation transpiled successfully"
# fi

- name: Fix malformed code blocks
run: |
echo "Fixing malformed code block titles..."
# The Fumadoc .md to .mdx transpiler add titles but without adding a language
# which then makes Shiki fail, so we remove the title when there isn't a language
find content/docs -name "*.mdx" -exec sed -i 's/^``` title=\".*$/```/g' {} \;
echo "Code block titles fixed"
Comment on lines 186 to 192
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can open a bug in the transpiler also, so that it's fix upstream.

cc: @JagjeevanAK


- name: Copy documentation assets
run: |
echo "Copying Product Opener assets..."
Expand All @@ -191,36 +199,36 @@ jobs:
cp -r docs/product-opener/assets/* "content/docs/Product-Opener/(docs)/assets/"
echo "Product Opener assets copied successfully"
fi

# Uncomment and modify these sections when other repositories are enabled
# echo "Copying Robotoff assets..."
# if [ -d "docs/robotoff/assets" ]; then
# mkdir -p "content/docs/Robotoff/(docs)/assets"
# cp -r docs/robotoff/assets/* "content/docs/Robotoff/(docs)/assets/"
# echo "Robotoff assets copied successfully"
# fi

# echo "Copying Infrastructure assets..."
# if [ -d "docs/infrastructure/assets" ]; then
# mkdir -p "content/docs/infrastructure/(docs)/assets"
# cp -r docs/infrastructure/assets/* "content/docs/infrastructure/(docs)/assets/"
# echo "Infrastructure assets copied successfully"
# fi

# echo "Copying Open Prices assets..."
# if [ -d "docs/open-prices/assets" ]; then
# mkdir -p "content/docs/Open-prices/(docs)/assets"
# cp -r docs/open-prices/assets/* "content/docs/Open-prices/(docs)/assets/"
# echo "Open Prices assets copied successfully"
# fi

# echo "Copying Knowledge Panels assets..."
# if [ -d "docs/knowledge-panels/assets" ]; then
# mkdir -p "content/docs/Knowledge-Panel/(docs)/assets"
# cp -r docs/knowledge-panels/assets/* "content/docs/Knowledge-Panel/(docs)/assets/"
# echo "Knowledge Panels assets copied successfully"
# fi

# echo "Copying Search-a-licious assets..."
# if [ -d "docs/search-a-licious/assets" ]; then
# mkdir -p "content/docs/search-a-licious/(docs)/assets"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ for (key, value) in images_to_update:

// Now, create a button with this text.
// e.g., "Refresh front picture", "Take packaging picture"
``` title="Client-Side Implementation Logic"
```

#### Optional: Displaying the Image Age

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Here are the different messages to use according to the state:

```sh title="Data completion flow"
if "en:categories-completed" in states_tags AND nutrition_grade=Null
``` title="Data completion flow"
```

```sh title="Data completion flow"
"We could not compute an Nutri-Score for this product. It might be that the category is an exception. If you believe this is an error, you can email [email protected]"
Expand All @@ -72,7 +72,7 @@ if "en:categories-completed" in states_tags AND nutrition_grade=Null

```sh title="Data completion flow"
if "en:categories-completed" in states_tags AND "en:nutrition-facts-to-be-completed" in states_tags
``` title="Data completion flow"
```

- Prompt: "Add nutrition facts to compute the Nutri-Score"

Expand All @@ -99,7 +99,7 @@ if "en:categories-to-be-completed" in states_tags AND "en:nutrition-facts-comple

```sh title="Data completion flow"
if "en:categories-to-be-completed" in states_tags AND "en:nutrition-facts-to-be-completed" in states_tags
``` title="Data completion flow"
```

- Prompt: "Add nutrition facts and a category to compute the Nutri-Score"

Expand All @@ -126,7 +126,7 @@ if "en:nutrition-photo-to-be-selected" in states_tags OR "en:photos-to-be-upload
```sh title="Data completion flow"
msgctxt "nutrition_grade_fr_fiber_warning"
msgid "Warning: the amount of fiber is not specified, their possible positive contribution to the grade could not be taken into account."
``` title="Data completion flow"
```

##### a message if fruit/nuts are missing

Expand All @@ -140,7 +140,7 @@ msgid "Warning: the amount of fruits, vegetables and nuts is not specified, thei
```sh title="Data completion flow"
msgctxt "nutrition_grade_fr_fruits_vegetables_nuts_estimate_warning"
msgid "Warning: the amount of fruits, vegetables and nuts is not specified on the label, it was estimated from the list of ingredients: %d%"
``` title="Data completion flow"
```

##### a message if fruits/nuts is an estimate from category

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ The dynamic test grouping system replaces hardcoded test groups with an intellig

The system uses a **greedy bin-packing algorithm**:

``` title="Algorithm Details"
```
1. Discover all test files automatically
2. Load historical timing data (if available)
3. Sort tests by execution time (descending)
Expand All @@ -52,7 +52,7 @@ The system uses a **greedy bin-packing algorithm**:

## File Structure

``` title="File Structure"
```
.test_groups_cache/ # Cache directory (git-ignored)
├── unit_groups.json # Cached unit test groups
├── unit_groups.mk # Makefile format for unit groups
Expand Down Expand Up @@ -150,7 +150,7 @@ To change, update:

The system generates statistics showing:

``` title="Performance Reports"
```
# Group Statistics:
# Group 1: 12 tests, 8m 45s
# Group 2: 10 tests, 9m 12s
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ off@off:/srv/off$ (off) ./scripts/minion_producers.pl minion job
132132 inactive pro.openfoodfacts.org update_export_status_for_csv_file
132131 failed openfoodfacts.org import_csv_file
132130 finished pro.openfoodfacts.org export_csv_file
``` title="Checking the status of Minion tasks (jobs)"
```

## Debugging failed import of pro platform data on public platform

Expand Down Expand Up @@ -50,7 +50,7 @@ In the off container, there is little useful information in the logs:

``` bash
import_csv_file_task - job: 132131 started - args: {"comment":"Import from producers platform","csv_file":"/srv/off-pro/export_files/org-systeme-u/export.1741792171.exported.csv","global_values":{"data_sources":"Producers, Producer - systeme-u"},"query":{"owner":"org-systeme-u","states_tags":"en:to-be-exported","data_quality_errors_producers_tags.0":{"$exists":false},"code":"3256221408515"},"export_job_id":132130,"source_id":"org-systeme-u","manufacturer":1,"org_id":"systeme-u","export_id":1741792171,"include_images_paths":1,"user_id":"org-systeme-u","source_name":"systeme-u","include_obsolete_products":1,"exported_t":1741792171,"owner_id":"org-systeme-u"}
``` title="Debugging failed import of pro platform data on public platform"
```

To get more data (debug level) in minion_log4perl.log:

Expand Down Expand Up @@ -83,7 +83,7 @@ Last lines in minion_log4perl.log:
[24193] /srv/off/lib/ProductOpener/TaxonomiesEnhancer.pm 257 ProductOpener.TaxonomiesEnhancer {} check_ingredients_between_languages > detect_missing_stop_words_before_list - first ingredient in ingredients1 (fr:ble-dur-precuit-concasse) is unknown (is_in_taxonomy => 1) or first ingredient in ingredients2 is known (is_in_taxonomy => 1)
[24193] /srv/off/lib/ProductOpener/TaxonomiesEnhancer.pm 356 ProductOpener.TaxonomiesEnhancer {} check_ingredients_between_languages > detect_missing_stop_words_after_list - start, lang1: fr, lang2: en
[60104] scripts/minion_producers.pl 87 main {minion_backend => [..] minion producers workers stopped
``` title="Debugging failed import of pro platform data on public platform"
```

Unfortunately there's no clearer error message.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ An explanation of the setup can be found at [explain-pro-dev-setup.md](explain-p

``` sh
docker compose exec minion /opt/product-opener/scripts/minion_producers.pl minion job
``` title="Working with Product Import/Export and Interacting with the Public Platform:"
```
(add --help to see all options), or refer to https://docs.mojolicious.org/Minion/Command/minion/job

- You may also inspect the database by running:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This page describes how to test and debug your changes once you have set up the

``` sh
make log
``` title="Tail Docker Compose logs"
```

You will get logs from nginx, mongodb, postgres, etc.

Expand All @@ -39,7 +39,7 @@ It will `tail -f` all the files present in the `logs/` directory:
You can also simply run:
``` sh
tail -f <FILEPATH>
``` title="Tail other logs"
```
to check a specific log.

One of the most important is `log4perl.log`.
Expand All @@ -62,7 +62,7 @@ Run the following to open a bash shell within the `backend` container:

``` sh
docker compose exec backend bash
``` title="Opening a shell in a Docker container"
```

You should see `root@<CONTAINER_ID>:/#` (opened root shell): you are now within the Docker container and can begin typing some commands!

Expand All @@ -81,7 +81,7 @@ Navigate to your specific directory using `cd` and run

``` sh
mkdir directory-name
``` title="Creating directory"
```

### Running minion jobs

Expand All @@ -97,7 +97,7 @@ The above command will show the status of minion jobs. Run the following command

``` sh
./minion_producers.pl minion worker -m production -q pro.openfoodfacts.org
``` title="Running minion jobs"
```

### Restarting Apache

Expand Down Expand Up @@ -141,7 +141,7 @@ Most are useful!

### Live reload
To automate the live reload on code changes, you can install the Python package `when-changed`:
``` title="Live reload"
```
pip3 install when-changed
when-changed -r docker/ docker-compose.yml .env -c "make restart" # restart backend container on compose changes
when-changed -r lib/ -r docker/ docker-compose.yml -c "docker compose backend restart" # restart Apache on code changes
Expand All @@ -155,7 +155,7 @@ An alternative to `when-changed` is `inotifywait`.

``` sh
docker compose exec mongodb mongo
``` title="Run queries on MongoDB database"
```

The above command will open a MongoDB shell, allowing you to use all the `mongo`
commands to interact with the database:
Expand Down Expand Up @@ -256,7 +256,7 @@ To switch between configurations, set `ENV_FILE` before running `make` commands,
ENV_FILE=.env.off-pro make up # starts the OFF Producer's Platform containers.
ENV_FILE=.env.obf make up # starts the OBF containers.
ENV_FILE=.env.opff make up # starts the OPFF containers.
``` title="different .env file"
```

or export it to keep it for a while:

Expand Down Expand Up @@ -304,7 +304,7 @@ PRODUCT_OPENER_DOMAIN=openfoodfacts.localhost
PRODUCT_OPENER_PORT=80
PRODUCT_OPENER_FLAVOR=openfoodfacts
PRODUCT_OPENER_FLAVOR_SHORT=off
``` title="different .env file"
```
to
``` sh
COMPOSE_PROJECT_NAME=po_obf
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ You can then use the following script from a backend bash shell to update produc

``` sh
./update_all_products.pl --fields categories --compute-ecoscore
``` title="Review and fix any changed Categories"
```

The process will set the `en:ecoscore_grade_changed` and `en:ecoscore_changed` misc_tags, which can be queried to analyse the results. For example, the following script generates a CSV file that summaries all the categories where the grade has changed:

Expand Down
2 changes: 1 addition & 1 deletion content/docs/Product-Opener/(docs)/dev/how-to-use-repl.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Just run

``` sh
docker compose run --rm backend re.pl
``` title="Launch Repl"
```

If you want to access external services (like mongodb), do not forget to start them.

Expand Down