Merge remote-tracking branch 'origin/v2' into feat/agent-framework-integration

Author: JCHAVEROT

.gitignore

@@ -114,6 +114,7 @@ venv.bak/
 # Milvus DB
 db/
 *.db
+*.db.lock
 
 # Project files
 tmp/

README.md

@@ -66,6 +66,8 @@ brew install cairo pango gdk-pixbuf libffi
 uv pip install weasyprint
 ```
 
+You can also run MMORE on Windows by following our [Windows setup notes](docs/source/getting_started/windows.md).
+
 #### Step 1 – Install MMORE
 
 Dependencies are split by pipeline stage. Install only what you need:
@@ -103,6 +105,22 @@ uv pip install "mmore[process,cpu]"
 
 > :warning: **Check the instructions for contributors directly at [`docs/for_devs.md`](./docs/for_devs.md)**
 
+### Interactive TUI
+
+Prefer a guided experience over editing YAML by hand? Install the `tui` extra and launch the interactive Terminal UI:
+
+```bash
+uv sync --extra tui
+mmore tui
+```
+
+From the launcher you can:
+
+- run any stage (process / postprocess / index / rag / chat) interactively,
+- chain the full pipeline (process → postprocess → index → chat),
+- generate stage YAML configs through a guided wizard,
+- pick from existing example configs without leaving the terminal.
+
 ### Minimal Example
 
 You can use our predefined CLI commands to execute parts of the pipeline. Note that you might need to prepend `python -m` to the command if the package does not properly create bash aliases.

docs/source/core_features/llm_as_a_judge.md

@@ -0,0 +1,72 @@
+# LLM as a judge
+
+Add a `judge:` block to your RAG config to check retrieval quality before generation. When chunks are not good enough, the judge can trigger a corrective action such as re-search, sub-questions, or web context and then merge everything into one deduplicated list.
+
+## How it works
+
+1. Retrieve chunks from the index (Milvus + optional BGE rerank).
+2. Evaluate them in a loop (at most `max_corrective_steps` corrective actions, default `1`):
+  - **`PROCEED` without calling the judge LLM** when index metrics meet `metric_thresholds` — retrieval is already considered good enough.
+  - Otherwise, call `judge.llm` and run the chosen corrective action.
+  - Repeat on the merged chunks until the judge says `PROCEED` or the step budget is exhausted.
+3. Generate the answer from the final context.
+
+Disallowed decisions are coerced to a fallback action (`RE_RETRIEVE`, `ADD_QUESTIONS`, or `PROCEED`). Invalid JSON defaults to `PROCEED`.
+
+## Decisions
+
+
+| Decision        | What it does                                                    |
+| --------------- | --------------------------------------------------------------- |
+| `PROCEED`       | Chunks are good enough; continue to the answer LLM              |
+| `RE_RETRIEVE`   | Search the index again (reformulated query and/or more results) |
+| `ADD_QUESTIONS` | Up to 3 extra searches from sub-questions, then merge           |
+| `ADD_CONTEXT`   | DuckDuckGo web snippets, then merge                             |
+
+
+## Configuration
+
+`examples/rag/config_judge.yaml` is a standalone config — it does not load on top of `config.yaml`.
+
+```bash
+python3 -m mmore rag --config-file examples/rag/config_judge.yaml
+```
+
+Or copy the `judge:` block into your own config.
+
+Key settings under `rag.judge`:
+
+- `metric_thresholds` — index minimums (`min_mean_similarity`, `min_max_rerank_score`, `min_num_docs`, …)
+- `max_corrective_steps` — how many corrective actions after the first retrieval
+- `allow_re_retrieve` / `allow_add_questions` / `allow_add_context` — which corrective actions the judge may choose (see below)
+- `system_prompt` / `user_prompt` — judge prompts; user prompt supports `{query}`, `{metrics}`, `{chunks}`, `{allowed_actions}`, and correction-step placeholders
+
+### Using one corrective action
+
+In your RAG config, under `**rag.judge**`, set `allow_re_retrieve`, `allow_add_questions`, and `allow_add_context` so **only one** corrective action is `true` (the others `false`). `PROCEED` is always available in `{allowed_actions}`.
+
+When `**metric_thresholds` are met**, the pipeline `**PROCEEDS` immediately** without calling the judge LLM: index retrieval is already of high quality (similarity, rerank scores, enough documents).
+
+When **thresholds fail**, the judge LLM is invoked. With a single corrective action enabled, it **systematically chooses that action** and fills the matching payload (`extra_questions`, `web_query`, or `retrieve_params`). Use a query suited to that action (multi-part question → `ADD_QUESTIONS`; missing corpus fact → `ADD_CONTEXT`; weak or mis-phrased retrieval → `RE_RETRIEVE`). Adjust `system_prompt` / `user_prompt` under `rag.judge` if needed.
+
+
+| Goal                            | `rag.judge` `allow_`* settings                                                      |
+| ------------------------------- | ----------------------------------------------------------------------------------- |
+| Sub-questions (`ADD_QUESTIONS`) | `allow_add_questions: true`, `allow_add_context: false`, `allow_re_retrieve: false` |
+| Web context (`ADD_CONTEXT`)     | `allow_add_context: true`, `allow_add_questions: false`, `allow_re_retrieve: false` |
+| Re-retrieval (`RE_RETRIEVE`)    | `allow_re_retrieve: true`, `allow_add_questions: false`, `allow_add_context: false` |
+
+
+Examples: `examples/rag/demo/config_add_questions.yaml`, `config_judge_add_questions.yaml`.
+
+For `ADD_CONTEXT`, install web search support:
+
+```bash
+pip install "mmore[rag,websearch]"
+```
+
+## See also
+
+- [RAG](../getting_started/rag.md)
+- [Websearch](websearch.md)
+

docs/source/developer_documentation/for_devs.md

@@ -31,6 +31,7 @@ This guide will help you set up your development environment and contribute to t
     - [Writing tests](#writing-tests)
   - [🔀 Pull Request Process](#-pull-request-process)
     - [PR checklist](#pr-checklist)
+  - [🖥️ Interactive TUI](#️-interactive-tui)
   - [💡 Development tips](#-development-tips)
     - [Working with `uv`](#working-with-uv)
   - [❓ Questions](#-questions)
@@ -256,6 +257,25 @@ def test_something_on_gpu():
 - [ ] Examples are provided for new features
 - [ ] Commit messages are clear and descriptive
 
+## 🖥️ Interactive TUI
+
+MMORE ships with a Terminal UI that wraps the CLI commands behind guided menus and config wizards. Useful for trying the pipeline without writing YAML by hand.
+
+Launch it from a project working directory:
+
+```bash
+mmore tui
+```
+
+From the main menu you can:
+
+- **Run a single command** — pick any stage (`process`, `postprocess`, `index`, `retrieve`, `rag`, `ragcli`, `websearch`), then either select an existing YAML, generate one through a guided wizard, or type a path manually. Generated configs are written to `./tui-configs/` and validated against the stage's dataclass before running.
+- **Run full pipeline** — chains `process → postprocess → index` using existing configs.
+- **Build a full pipeline config (guided wizard)** — walks through the three stages in order, wiring the postprocess output JSONL into the index config automatically.
+- **Chat with indexed documents** — shortcut to `ragcli`.
+
+Stages whose extras are missing are disabled in the menu with an install hint (e.g. `uv sync --extra rag --extra cpu`). Press `Ctrl-C` inside any sub-flow to cancel back to the main menu; press it again at the main menu to quit.
+
 ## 💡 Development tips
 
 ### Working with `uv`

docs/source/developer_documentation/retriever_api_specs.yaml

@@ -265,6 +265,10 @@ paths:
                     type: string
                   chunkId:
                     type: string
+                  filename:
+                    type: string
+                    nullable: true
+                    description: Original filename of the source document.
                   content:
                     type: string
                   metadata:

docs/source/getting_started/installation.md

@@ -216,3 +216,4 @@ For a manual non-Docker setup, use either the standard installation or the `uv`
 - [Quickstart](quickstart.md)
 - [Process](process.md)
 - [uv workflow](../advanced_usage/uv.md)
+- [Running on Windows](windows.md) — what differs on Windows and how to fix it

docs/source/getting_started/windows.md

@@ -0,0 +1,128 @@
+# 🪟 Running MMORE on Windows
+
+## Overview
+
+MMORE was developed and tested mainly on Linux. It runs on Windows too, but a few things behave differently. This page lists those differences and the fix for each one.
+
+If you work on Linux or macOS, you can skip this page.
+
+## 1. Install the prerequisites
+
+Unlike most Linux distributions, Windows does not ship Python, Git, or FFmpeg.
+Install them first with
+[winget](https://learn.microsoft.com/windows/package-manager/winget/):
+
+```powershell
+winget install Python.Python.3.11
+winget install Git.Git
+winget install astral-sh.uv
+winget install Gyan.FFmpeg
+```
+
+Then clone the repo and install MMORE into a virtual environment:
+
+```powershell
+git clone https://github.com/swiss-ai/mmore.git
+cd mmore
+uv venv
+.venv\Scripts\activate
+uv pip install -e ".[all,cu126]"
+```
+
+Use `cu126` for an NVIDIA GPU, or `cpu` otherwise. See the
+[README](https://github.com/swiss-ai/mmore#step-1--install-mmore) for the full
+list of extras.
+
+## 2. `milvus-lite` is not available on Windows
+
+Every example config whose `db.uri` is `./proc_demo.db` relies on `milvus-lite`
+(`examples/index/config.yaml`, `examples/retriever_api/config.yaml`,
+`examples/rag/config.yaml`, `examples/rag/config_api.yaml`). There is no Windows
+build of `milvus-lite`, so any of them fails with:
+
+```
+ModuleNotFoundError: No module named 'milvus_lite'
+```
+
+### Fix: run Milvus in Docker
+
+This repo ships no Compose file, so download the official Milvus standalone one
+matching your installed `pymilvus` version (see the
+[Milvus install docs](https://milvus.io/docs/install_standalone-docker-compose.md)):
+
+```powershell
+# Download the Milvus docker compose file from GitHub
+Invoke-WebRequest `
+  -Uri "https://github.com/milvus-io/milvus/releases/download/v2.6.6/milvus-standalone-docker-compose.yml" `
+  -OutFile "milvus-docker-compose.yml"
+# Start Milvus containers
+docker compose -f milvus-docker-compose.yml up -d
+```
+
+Wait about a minute, then check `docker ps` shows the three containers
+(`etcd`, `minio`, `milvus-standalone`) as `(healthy)`.
+
+### Create the database
+
+MMORE does not create the database automatically when connecting to a remote Milvus. Run this once:
+
+```powershell
+python -c "from pymilvus import connections, db; connections.connect(uri='http://127.0.0.1:19530'); db.create_database('my_db')"
+```
+
+### Point the configs at the Docker instance
+
+The `db` block lives at a different level depending on the config. Change
+`uri: ./proc_demo.db` to `uri: http://127.0.0.1:19530` in each one you use.
+
+`examples/retriever_api/config.yaml` (and `examples/rag/config*.yaml`) — `db`
+is at the root:
+
+```yaml
+db:
+  uri: http://127.0.0.1:19530
+  name: my_db
+```
+
+`examples/index/config.yaml` — `db` is nested under `indexer`:
+
+```yaml
+indexer:
+  db:
+    uri: http://127.0.0.1:19530
+    name: my_db
+```
+
+### Check that the setup works
+
+Once Milvus is running, confirm the connection:
+
+```powershell
+python -c "from pymilvus import MilvusClient; c = MilvusClient(uri='http://127.0.0.1:19530', db_name='my_db'); print(c.list_collections())"
+```
+
+This returns a list of collections (empty before you index anything).
+
+## 3. Surya OCR can crash the process on large PDFs
+
+When processing large PDFs, the surya-based OCR may crash with:
+
+```
+Process finished with exit code 0xC0000005
+```
+
+This is a hard crash inside a native dependency. On Windows, use the fast processors instead, which rely on PyMuPDF rather than surya.
+
+In your `process` config, `use_fast_processors` goes under `dispatcher_config`:
+
+```yaml
+dispatcher_config:
+  use_fast_processors: true
+```
+
+You lose some accuracy on heavily scanned PDFs, but the pipeline no longer crashes.
+
+## See also
+
+- [Installation](installation.md)
+- [Quickstart](quickstart.md)

docs/source/index.md

@@ -41,6 +41,7 @@ getting_started/architecture
 getting_started/process
 getting_started/indexing
 getting_started/rag
+getting_started/windows
 ```
 
 ```{toctree}
@@ -50,6 +51,7 @@ getting_started/rag
 core_features/colpali
 core_features/websearch
 core_features/evaluation
+core_features/llm_as_a_judge
 ```
 
 ```{toctree}
@@ -75,6 +77,7 @@ developer_documentation/index_api
 Here is a quick overview of the main pages:
 
 - [Installation](getting_started/installation.md): set up MMORE and prepare your environment
+- [Running on Windows](getting_started/windows.md): what differs on Windows and how to fix it
 - [Quickstart](getting_started/quickstart.md): run a first minimal workflow end to end
 - [Architecture](getting_started/architecture.md): understand the main system components and how they interact
 - [Processing pipeline](getting_started/process.md): understand how documents are ingested and transformed
@@ -83,6 +86,7 @@ Here is a quick overview of the main pages:
 - [ColPali](core_features/colpali.md): multimodal retrieval-related documentation
 - [Websearch](core_features/websearch.md): web search integration and related workflows
 - [Evaluation](core_features/evaluation.md): assess system performance
+- [LLM as a judge](core_features/llm_as_a_judge.md): corrective retrieval with an LLM judge
 - [Distributed processing](advanced_usage/distributed_processing.md): scale processing across larger workloads
 - [Profiler](advanced_usage/profiler.md): profile and analyze performance
 - [uv](advanced_usage/uv.md): environment and dependency workflow