Document HTTP(s)/MITM proxy configuration for scrape

[quackerd]

Problem Description

Could the README include a short section documenting how to route scrape traffic through an explicit HTTP(s) forward/MITM proxy?

This already works with the current Bun based implementation. I am not requesting a new proxy integration or browser dependency — just documentation for the existing runtime behavior.

For example, I currently use trawl's MITM proxy with the MCP configured like this:

environment:
  HTTP_PROXY: "http://trawl:8192"
  HTTPS_PROXY: "http://trawl:8192"
  NO_PROXY: "degoog.example.com,localhost,127.0.0.1"
  NODE_EXTRA_CA_CERTS: "/certs/ca.crt"
volumes:
  - ./certs/ca.crt:/certs/ca.crt:ro

HTTP_PROXY / HTTPS_PROXY route Bun's outgoing fetches through the proxy, while NO_PROXY can be used to keep degoog itself and local/internal endpoints out of that path.

For MITM proxies, the proxy's CA cert also needs to be trusted using NODE_EXTRA_CA_CERTS.

Motivation

The README currently describes scrape as static only. However, an external forward proxy can transparently add capabilities, such as JS rendering, anti anti-bot, challenge/captcha solving (flaresolvrr, byparr, trawl), without requiring any changes to the MCP itself.

Suggested Documentation

Something along the lines of:

## Using an HTTP(S)/MITM proxy

degoog MCP uses Bun, which respects the standard HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables. For example:

`HTTP_PROXY`: "http://proxy:8080"
`HTTPS_PROXY`: "http://proxy:8080"
`NO_PROXY`: "degoog,localhost,127.0.0.1"

If the proxy intercepts HTTPS connections, mount and trust its CA certificate using `NODE_EXTRA_CA_CERTS`.

Potential Multi-backend Fetch Support

It would be also very nice to explicitly support different fetch backends: native static fetch, flaresolvrr backends (which is the same as degoog's flaresolvrr extension), MITM/HTTP Proxies (already covered by env vars), even 4play. That would enable users to freely pick the content fetcher based on their needs.

[fccview]

That's actually an angle I didn't think about at all, frankly.

Imagine my plan was to allow the mcp to talk back to degoog transports and use them as scrape utilities, but this is waaaaay smarter, good idea.

I think it's worth adding a few individual markdown files as documentation in a dedicated docs folder, given how extensive the whole thing is already, I'll get Claude on it with my documentation skill I have for degoog, should be good enough to handle this too <3

p.s. i'll add this to the documentation as well of course!

[quackerd]

mcp-searxng currently only allows MITM/HTTP forward proxy too. I think it would be a great feature to support custom flaresolvrr-compatible fetch endpoints in the future.

[fccview]

mcp-searxng currently only allows MITM/HTTP forward proxy too. I think it would be a great feature to support custom flaresolvrr-compatible fetch endpoints in the future.

Yeah, given this already probes the degoog api it's already technically capable of probing degoog extensions via api, my plan was to allow scrape to go through any configured third party degoog transport extension (e.g. add degoog-org-official-extensions-flaresolverr as a transport handler into the yml setting and the mcp magically uses it as transport instead of basic fetch)

Everything is already there up for grub, it should really be quite easy to setup :)

[quackerd]

Yeah that's basically a design decision. If you care about "architectural purity" and wants to make degoog search only -- you could alternatively share the extension code via git submodules or other means, and enabling configuring individual plugins via config files/env vars.

But if you want this specifically to be tightly coupled with degoog then just let degoog expose extra scrape APIs. Both make sense.

[fccview]

Yeah that's basically a design decision. If you care about "architectural purity" and wants to make degoog search only -- you could alternatively share the extension code via git submodules or other means, and enabling configuring individual plugins via config files/env vars.

But if you want this specifically to be tightly coupled with degoog then just let degoog expose extra scrape APIs. Both make sense.

I did NOT think of that, very valid, somebody may want to use this with something else that's not degoog and they should be able to do it without me forcing degoog upon them, very valid.
I will add a transport section to the settings and add json/html type transports configurable from there, letting users just decide what they want to do. I'll also document the currently already working http proxies ones as that's 0 overhead on my end lol