Feature/search providers refactor

[pivotdude]

This pull request introduces a major enhancement to the application's search capabilities by integrating several new, powerful search and scraping providers. It also includes a significant refactoring of the MetaSearchAgent to create a more modular, extensible, and maintainable architecture.

Users can now choose from a variety of search backends, including Exa, Tavily, Firecrawl, and Jina AI, in addition to the existing SearXNG. This allows for more flexible and specialized information retrieval depending on the user's needs.

The core search logic has been decoupled from the specific provider implementations, making it much easier to add new providers in the future.

---

Key Changes

New Search Providers Added:

• Integrated support for Exa, Tavily, Firecrawl, and Jina AI as new search options.
• Added necessary dependencies to package.json: @mendable/firecrawl-js, @tavily/core, and exa-js.
• Created dedicated provider implementation files under src/lib/search/searchProviders/providers/.

Modular Search Provider Framework:

• Introduced a BaseSearchProviderManager that uses a factory pattern to dynamically instantiate the selected search provider.

Defined a common SearchProvider interface and specific option types for each service in src/lib/search/searchProviders/types.ts.

MetaSearchAgent Refactoring:

• The monolithic metaSearchAgent.ts file has been deleted.

Its logic has been broken down into smaller, single-responsibility classes for better separation of concerns:

• ConfigManager.ts: Manages configuration for search handlers.
• SearchRetriever.ts: Handles query generation and fetching initial search results.
• DocumentRanker.ts: Reranks documents for relevance, specifically when using providers that require it (like searxng).
• AnswerGenerator.ts: Constructs the final answer based on the retrieved and ranked context.
• StreamHandler.ts: Manages the event stream for real-time responses.

Configuration and Settings UI:

• Updated sample.config.toml with new sections for API keys and a global search PROVIDER setting.
• The Settings page (/settings) now features input fields for the new API keys and a dropdown to select the active search provider.

API and Logic Updates:

• API routes (/api/chat and /api/search) now use the dynamic getSearchHandler function instead of a static searchHandlers object.
• Added isValidFocusMode validation to ensure robustness in the API routes.

Configuration Required
To use the new search providers, update your config.toml with the following sections and provide your API keys:

Ini, TOML

[MODELS.EXA]
API_KEY = ""

[MODELS.TAVILY]
API_KEY = ""

[MODELS.FIRECRAWL]
API_KEY = ""

[MODELS.JINA]
API_KEY = ""

[SEARCH]
PROVIDER = "searxng" # Options: "searxng", "exa", "tavily", "firecrawl", "jina-ai"
LANGUAGE = "en"
COUNT = 20

[SEARCH.PROVIDERS]
  # Provider specific settings. Uncomment and configure as needed.

  # [SEARCH.PROVIDERS.exa]
  # USE_AUTOPROMPT = false

  # [SEARCH.PROVIDERS.tavily]
  # SEARCH_DEPTH = "basic"

  # [SEARCH.PROVIDERS.firecrawl]
  # ONLY_MAIN_CONTENT = true

[ItzCrazyKns]

Hey, thank you very much for the PR!. Let us resolve the conflicts and get this merged