Add websearch

[laetitia-wilhelm]

Websearch on RAG output

Input:
Output file from the RAG model

Output:

• Original query
• Summary of the RAG output
• Brief answer to the query
• Detailed answer

---

How to Run

python -m mmore websearch --config-file examples/websearch/config.yaml

---

Key Parameters

• n_loops: Number of search loops
• max_searches: Maximum number of sources retrieved per web search

---

Pipeline Overview

1. Load Input Data

   • Extract the original query and the initial answer generated by the RAG model.

2. Generate Initial Summary

   • Summarize the RAG answer with respect to the original query.

3. Iterative Search and Analysis (Repeated n_loops times)

   • Generate Search Query:
     Formulate a refined search query by combining the original query, current knowledge, and previous findings (if any).
   • Perform Web Search:
     Retrieve relevant web results using DuckDuckGo.
   • Analyze Search Results:
     Use the language model (LLM) to integrate new web information with existing knowledge, updating the summary accordingly.
   • Update the current knowledge and previous analysis for the next iteration.

4. Save Final Results

   • Store the final combined summary derived from both web search and RAG output.

[fabnemEPFL]

Great work so far, sounds promising. Some changes needed.
I have to go so there will be a follow-up review later today

[fabnemEPFL]

Additional comments

[fabnemEPFL]

I will make soon the changes related to the few additional comments I added

[fabnemEPFL]

Make it a field of type LLMConfig