Skip to content

Voice Agent: add private SearXNG web-search tool to Pipecat #122

Description

@alucero270

Goal

Allow the Pantheon Pipecat voice agent to perform web searches when current or unknown information is needed, without granting the LLM unrestricted network access.

Current validated state

  • Pipecat runs host-native on Prometheus.
  • Private SearXNG is already deployed in Docker and routed through https://searxng.home.arpa.
  • The SearXNG JSON endpoint returned HTTP 200 with results during the 2026-06-20 validation; measured search time was about 1.2 seconds.
  • gemma-4-12b-it-qat-co produced a valid OpenAI-compatible web_search function call through llama-swap with finish reason tool_calls and a structured query argument.
  • Installed Pipecat 1.2.1 supports FunctionSchema, ToolsSchema, LLMContext tools, and llm.register_function.

Proposed implementation

  1. Give the host-native Pipecat process a private, stable SearXNG target.
    • Preferred: loopback-only bind such as 127.0.0.1:8088:8080.
    • Alternative: use the Traefik route only after the internal certificate is trusted by the Pipecat Python environment.
    • Do not expose SearXNG publicly or bind it to all host interfaces.
  2. Add a bounded web_search(query) function to the Git-backed voice-agent bot.
  3. Register the function with Pipecat and expose a FunctionSchema through ToolsSchema.
  4. Query SearXNG /search with format=json, a fixed result limit, timeout, and safe-search setting.
  5. Return only compact result objects containing title, URL, and a truncated snippet.
  6. Update the system prompt so search is used only for current, uncertain, or explicitly requested information.
  7. Preserve source URLs in the conversation/tool result for attribution.

Security constraints

  • The model must not choose the search endpoint or arbitrary request URL.
  • Search only in the first iteration; do not add arbitrary page fetching yet.
  • Limit query length, result count, response size, and request duration.
  • Treat search snippets and webpages as untrusted content and never as instructions.
  • Block private, loopback, link-local, and metadata-address access before adding any future fetch_url tool.
  • Do not commit SearXNG secrets, .env, transcripts, or user queries.
  • Keep SearXNG private under existing firewall and Traefik policy.

Acceptance criteria

  • Ordinary conversational turns do not invoke search.
  • A current-information prompt causes Gemma to call web_search automatically.
  • Pipecat receives 3-5 compact SearXNG results and produces a spoken answer.
  • Search timeout/failure returns a natural, nonfatal response.
  • Search adds measured latency and does not break interruption handling.
  • Source titles and URLs remain visible in the conversation event/history.
  • No arbitrary URL-fetching or public SearXNG exposure is introduced.
  • Component and manual WebRTC tests are documented with rollback instructions.

Follow-up

A separate fetch_url tool may be evaluated later with SSRF defenses, content-size limits, MIME restrictions, and explicit redirect validation.

Related: #72, #115, #116

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions