Add websearch

docs/websearch.md

@@ -0,0 +1,97 @@
+# WebSearch Integration in RAG Pipeline
+
+## Implementation
+
+### Overview
+The WebSearch integration uses the DuckDuckGo search API through a wrapper provided by the LangChain library. The implementation combines:
+
+- **DuckDuckGo Search:** Retrieves external information through concise subqueries generated by a LLm
+- **LLM Integration:** Summarizes and integrates retrieved web snippets with RAG results to provide a final answer
+
+## Customization
+
+Based on the implementation of the `RAG` module, the `Websearch` module enables the creation of a modular Websearch inference pipeline for your indexed multimodal documents, using two inference modes:
+ 1. **API**: Creates a server hosting the pipeline
+ 2. **Local**: Runs the inference locally
+
+You can customize various parts of the pipeline by defining [an inference Websearch configuration file](/examples/websearchRAG/config_api.yaml).
+
+Users can adjust the pipeline according to their [requirements](/examples/websearchRAG/config.yaml) through the following parameters:
+
+- `use_rag`: Enables or disables RAG retrieval.
+- `use_summary`: Activates summarization of retrieved web snippets.
+- `n_loops`: Defines the number of search iterations to refine results.
+- `n_subqueries`: Specifies the number of subqueries generated for each input query.
+
+### Workflow
+
+0. **RAG pipeline:**
+    - Launch the RAG pipeline if `use_rag` is enabled and the set the output as being the current knowledge
+1. **Input Query Processing:**
+   - The pipeline processes the user query and generates subqueries for web searches in order to complete the current knowledge.
+2. **WebSearch Execution:**
+   - DuckDuckGo searches are performed for each subquery
+3. **Summarization:**
+   - Retrieved web snippets are summarized using an LLM if `use_summary` is enabled.
+4. **Integration with RAG (if use_rag):**
+   - WebSearch results are combined with RAG outputs to generate the current knowledge
+5. **Start again:**
+    - We loop again from step 1 with the updated current knowledge
+
+## Minimal Example
+
+Here is a example to create a Websearch pipeline hosted through [LangServe](https://python.langchain.com/docs/langserve/) servers.
+
+1. Create your RAG Inference config file based on the [local example](/examples/websearch/config.yaml) or the [API example](/examples/websearch/config_api.yaml).
+
+2. Start your Websearch pipeline using the `run_websearch.py` script and your config file
+    ```bash
+    python3 -m mmore websearch --config_file /path/to/config.yaml
+    ```
+
+3. In API mode, query the server like any other LangServe server:
+    ```bash
+    curl --location --request POST http://localhost:8000/websearch/ \
+    -H 'Content-Type: application/json' \
+    -d ' {
+        "query": {
+        "input": "When was Barack Obama born?",
+        "collection_name": "my_docs"
+        },
+        "use_rag": true,
+        "use_summary": true
+        }
+    }'
+    ```
+    In the API mode, it is necessary to provide the `use_rag` and `use_summary`parameters in the query, the number of `n_loops` and `n_subqueries` are defined in the config file.
+
+    In local mode, the pipeline is run directly with the input data specified in the configuration file and the result is saved at the specified path.
+
+    For both mode, if we want to use the RAG pipeline, it is necessay to provide the path to the rag configuration file.
+
+## Results and Outputs
+
+### Output Format
+The pipeline provides outputs in the following structure:
+
+- **Short Answer:** Concise response derived from combined RAG and WebSearch results.
+- **Detailed Answer:** Expanded response with context from both sources.
+- **Sources:** A list of URLs and their respective titles in the format:
+  ```json
+    {"URL": ["Title 1", "Title 2", ...]}
+  ```
+
+### Example Output
+```json
+{
+    "query": "What are the latest advancements in AI?",
+    "rag_informations": "Pre-existing knowledge from RAG.",
+    "rag_summary": "Summary of RAG knowledge.",
+    "web_summary": "Summarized web search information.",
+    "short_answer": "AI advancements include new models and real-time applications.",
+    "detailed_answer": "Recent AI developments include breakthroughs in large language models and innovative real-time applications, supported by diverse resources from RAG and web search.",
+    "sources": {
+        "https://example1.com" : ["AI Breakthrough; Latest in AI"],
+        "https://example2.com" : ["Advancements in AI; AI Trends"],
+    }
+}

examples/websearchRAG/config.yaml

@@ -0,0 +1,14 @@
+websearch:
+  use_rag: true
+  rag_config_path: examples/rag/config.yaml
+  use_summary: true
+  n_subqueries : 2
+  input_file: examples/rag/output.json
+  input_queries: examples/rag/queries.jsonl
+  output_file: examples/websearchRAG/enhanced_results_trial.json
+  n_loops: 2
+  max_searches: 5
+  mode: local
+  llm_config:
+    llm_name: OpenMeditron/meditron3-8b
+    max_new_tokens: 1200

examples/websearchRAG/config_api.yaml

@@ -0,0 +1,15 @@
+websearch:
+  rag_config_path: examples/rag/config_api.yaml
+  n_subqueries : 3
+  n_loops: 2
+  max_searches: 10
+  mode: api
+  llm_config:
+    llm_name: OpenMeditron/meditron3-8b
+    max_new_tokens: 1200
+
+# Mode Config
+mode_args:
+  endpoint: '/rag'
+  port: 8000
+  host: 'localhost'