Perform iterative deep research web searches using SearXNG and BeautifulSoup for content scraping

Author: author: Jeremy Petzold (Inspired by the work of Teodor Cucu's research-openwebui )

Version: 0.5.0

License: MIT

Overview

This project implements an asynchronous deep research tool that performs iterative internet searches using a SearXNG-compatible endpoint. It processes search results by scraping the content of the returned pages with BeautifulSoup, refines the search query if too few results are obtained, and finally generates a comprehensive Markdown report. Throughout the search process, status updates and citation events are sent to a frontend via a callback function.

The origional code base for this Deep Research tool was based on the work of Teodor Cucu's research-openwebui. Multiple enhancments have been made to improve the research and output.

• Enhanced the iteration search adjustments
• cleaned up some issues with performance and memory
• improved the report generation

To-do:

• Update the search query changes between iterations to utilize an LLM to determine what to ask based on gaps from the previous itteration.
• modularize the code base into multiple OWUI tools to be chained together: Search, analysis, report generation

#Pre-requisits

• Have a local instance of SearXNG set up and running
• For better privacy, run SearXNG and the computer running Open WebUI behind a trusted VPN service such as Mullvad.

How-To-Install/Use :

1. Install the Tool:

   • In Open WebUI, go to workspace -> tools
   • Create a new tool and call it Research
   • Copy the code from Deepresearch.py and paste it into the tool's code entry
   • Click Save

2. Set SRXNG URL: Before leaving Tools, click the gear next to the research tool and enter the SearXNG instance URL.

3. Install the Prompt (REQUIRED):

   • In Open WebUI, go to workspace -> Prompt
   • Create a new prompt named research
   • Copy the text from prompt.txt and paste it into the prompt text entry
   • Click Save

4. Create the Deep Research Model:

   • In Open WebUI, go to Workspace -> Models
   • Create a new model named Deep Research
   • Set the base model to your prefered installed local model
   • For System Prompt, enter /research to call the prompt test from step 3
   • Under the Tool's section of the form, check the Research tool
   • Under the Tool's capabilities section, uncheck Web Search
   • Click Save

5. Open a New Chat and Activate Research:
   Open Open WebUI, create a new chat, and select the Deep Research model.

6. Observe the Process:
   Type in your research topic and submit. The research begins when you see status updates like "Processing X results from Itteration Y...".
   Each iteration will display how many pages were fetched.

7. Completion and Citations:
   Once complete, you should see final report along with citation events listing the sources.