Skip to content

Commit db21b97

Browse files
ihor-sokoliukclaudecodex
committed
feat(proxy): add SEARCH_USER_AGENT override for SearXNG-instance requests (FEAT-050)
Introduces getSearchUserAgent() (SEARCH_USER_AGENT || USER_AGENT) and applies it to all three instance-side fetches — searxng_web_search, /config discovery, and suggestions — so User-Agent config mirrors the proxy model: a global default (USER_AGENT) plus per-group overrides (SEARCH_USER_AGENT for instance traffic, URL_READER_USER_AGENT for reads). web_url_read is unchanged. Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Codex <noreply@openai.com>
1 parent 9721156 commit db21b97

7 files changed

Lines changed: 141 additions & 9 deletions

File tree

‎CONFIGURATION.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,8 +70,11 @@ Self-hosting SearXNG with JSON output enabled remains the recommended setup. The
7070

7171
| Variable | Required | Default | Description |
7272
|---|---|---|---|
73-
| `USER_AGENT` | No | — | Global User-Agent header for all outgoing requests (e.g. `MyBot/1.0`) |
74-
| `URL_READER_USER_AGENT` | No | — | User-Agent for `web_url_read` only — overrides `USER_AGENT` for URL reads |
73+
| `USER_AGENT` | No | — | Global default User-Agent header for outgoing requests (e.g. `MyBot/1.0`) |
74+
| `SEARCH_USER_AGENT` | No | `USER_AGENT` | User-Agent for SearXNG instance requests: `searxng_web_search`, `/config` capability discovery, and search suggestions |
75+
| `URL_READER_USER_AGENT` | No | `USER_AGENT` | User-Agent for `web_url_read` only |
76+
77+
`SEARCH_USER_AGENT` and `URL_READER_USER_AGENT` are per-group overrides. When unset, both fall back to `USER_AGENT`. If neither the group override nor `USER_AGENT` is set, no User-Agent header is added by `mcp-searxng`.
7578

7679
## Proxy
7780

@@ -205,6 +208,7 @@ Complete MCP client configuration with every variable. Mix and match as needed
205208
"AUTH_USERNAME": "your_username",
206209
"AUTH_PASSWORD": "your_password",
207210
"USER_AGENT": "MyBot/1.0",
211+
"SEARCH_USER_AGENT": "MySearchBot/1.0",
208212
"URL_READER_USER_AGENT": "Mozilla/5.0 (compatible; MyBot/1.0)",
209213
"SEARCH_HTTP_PROXY": "http://search-proxy.company.com:8080",
210214
"SEARCH_HTTPS_PROXY": "http://search-proxy.company.com:8080",

‎__tests__/unit/instance-info.test.ts‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -444,6 +444,7 @@ async function runTests() {
444444
await testFunction('config request includes User-Agent header when USER_AGENT is set', async () => {
445445
clearInstanceInfoCacheForTests();
446446
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
447+
envManager.delete('SEARCH_USER_AGENT');
447448
envManager.set('USER_AGENT', 'MyBot/1.0');
448449
const mockServer = createMockServer();
449450
const { mockFetch, getCapturedOptions } = createCapturingMockFetch();
@@ -461,9 +462,31 @@ async function runTests() {
461462
envManager.restore();
462463
}, results);
463464

465+
await testFunction('config request uses SEARCH_USER_AGENT over USER_AGENT', async () => {
466+
clearInstanceInfoCacheForTests();
467+
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
468+
envManager.set('SEARCH_USER_AGENT', 'SearchBot/2.0');
469+
envManager.set('USER_AGENT', 'GlobalBot/1.0');
470+
const mockServer = createMockServer();
471+
const { mockFetch, getCapturedOptions } = createCapturingMockFetch();
472+
fetchMocker.mock(async (url, options) => {
473+
await mockFetch(url, options);
474+
return createMockFetch({ json: makeConfig() })(url, options);
475+
});
476+
477+
await fetchInstanceInfo(mockServer as any);
478+
479+
const headers = getCapturedOptions()?.headers as Record<string, string>;
480+
assert.equal(headers?.['user-agent'], 'SearchBot/2.0');
481+
482+
fetchMocker.restore();
483+
envManager.restore();
484+
}, results);
485+
464486
await testFunction('config request omits User-Agent header when USER_AGENT is unset', async () => {
465487
clearInstanceInfoCacheForTests();
466488
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
489+
envManager.delete('SEARCH_USER_AGENT');
467490
envManager.delete('USER_AGENT');
468491
const mockServer = createMockServer();
469492
const { mockFetch, getCapturedOptions } = createCapturingMockFetch();

‎__tests__/unit/search.test.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -828,6 +828,7 @@ async function runTests() {
828828

829829
await testFunction('User-Agent header added when USER_AGENT env var is set', async () => {
830830
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
831+
envManager.delete('SEARCH_USER_AGENT');
831832
envManager.set('USER_AGENT', 'MyCustomBot/1.0');
832833

833834
const mockServer = createMockServer();
@@ -852,8 +853,36 @@ async function runTests() {
852853
envManager.restore();
853854
}, results);
854855

856+
await testFunction('SEARCH_USER_AGENT overrides USER_AGENT for search requests', async () => {
857+
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
858+
envManager.set('SEARCH_USER_AGENT', 'SearchBot/2.0');
859+
envManager.set('USER_AGENT', 'GlobalBot/1.0');
860+
861+
const mockServer = createMockServer();
862+
const { mockFetch, getCapturedOptions } = createCapturingMockFetch();
863+
864+
fetchMocker.mock(async (url, options) => {
865+
await mockFetch(url, options);
866+
throw new Error('MOCK_STOP');
867+
});
868+
869+
try {
870+
await performWebSearch(mockServer as any, 'test query');
871+
} catch {
872+
// expected
873+
}
874+
875+
const options = getCapturedOptions();
876+
const headers = options?.headers as Record<string, string>;
877+
assert.equal(headers?.['User-Agent'], 'SearchBot/2.0');
878+
879+
fetchMocker.restore();
880+
envManager.restore();
881+
}, results);
882+
855883
await testFunction('User-Agent header absent when USER_AGENT env var not set', async () => {
856884
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
885+
envManager.delete('SEARCH_USER_AGENT');
857886
envManager.delete('USER_AGENT');
858887

859888
const mockServer = createMockServer();

‎__tests__/unit/suggestions.test.ts‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,7 @@ async function runTests() {
172172

173173
await testFunction('autocompleter request includes User-Agent header when USER_AGENT is set', async () => {
174174
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
175+
envManager.delete('SEARCH_USER_AGENT');
175176
envManager.set('USER_AGENT', 'MyBot/1.0');
176177
const mockServer = createMockServer();
177178
const { mockFetch, getCapturedOptions } = createCapturingMockFetch();
@@ -190,8 +191,30 @@ async function runTests() {
190191
envManager.restore();
191192
}, results);
192193

194+
await testFunction('autocompleter request uses SEARCH_USER_AGENT over USER_AGENT', async () => {
195+
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
196+
envManager.set('SEARCH_USER_AGENT', 'SearchBot/2.0');
197+
envManager.set('USER_AGENT', 'GlobalBot/1.0');
198+
const mockServer = createMockServer();
199+
const { mockFetch, getCapturedOptions } = createCapturingMockFetch();
200+
201+
fetchMocker.mock(async (url, options) => {
202+
await mockFetch(url, options);
203+
return createMockFetch({ json: ['type', ['typescript']] })(url, options);
204+
});
205+
206+
await performSearchSuggestions(mockServer as any, 'type');
207+
208+
const headers = getCapturedOptions()?.headers as Record<string, string>;
209+
assert.equal(headers?.['user-agent'], 'SearchBot/2.0');
210+
211+
fetchMocker.restore();
212+
envManager.restore();
213+
}, results);
214+
193215
await testFunction('autocompleter request omits User-Agent header when USER_AGENT is unset', async () => {
194216
envManager.set('SEARXNG_URL', 'https://test-searx.example.com');
217+
envManager.delete('SEARCH_USER_AGENT');
195218
envManager.delete('USER_AGENT');
196219
const mockServer = createMockServer();
197220
const { mockFetch, getCapturedOptions } = createCapturingMockFetch();

‎__tests__/unit/url-reader.test.ts‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,52 @@ async function runTests() {
284284
}
285285
}, results);
286286

287+
await testFunction('SEARCH_USER_AGENT does not override USER_AGENT for URL reads', async () => {
288+
envManager.set('SEARCH_USER_AGENT', 'SearchBot/2.0');
289+
envManager.set('USER_AGENT', 'GlobalBot/1.0');
290+
envManager.delete('URL_READER_USER_AGENT');
291+
const mockServer = createMockServer();
292+
const seenUserAgents: string[] = [];
293+
const { url, close } = await startHttpServer((req, res) => {
294+
seenUserAgents.push(req.headers['user-agent'] ?? '');
295+
res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
296+
res.end('<html><body><h1>Readable</h1></body></html>');
297+
});
298+
299+
try {
300+
await fetchAndConvertToMarkdown(mockServer as any, url);
301+
302+
assert.ok(seenUserAgents.length > 0, 'Expected the URL reader to make a request');
303+
assert.ok(seenUserAgents.every((userAgent) => userAgent === 'GlobalBot/1.0'));
304+
} finally {
305+
await close();
306+
envManager.restore();
307+
}
308+
}, results);
309+
310+
await testFunction('URL_READER_USER_AGENT still overrides USER_AGENT when SEARCH_USER_AGENT is set', async () => {
311+
envManager.set('SEARCH_USER_AGENT', 'SearchBot/2.0');
312+
envManager.set('USER_AGENT', 'GlobalBot/1.0');
313+
envManager.set('URL_READER_USER_AGENT', 'ReaderBot/3.0');
314+
const mockServer = createMockServer();
315+
const seenUserAgents: string[] = [];
316+
const { url, close } = await startHttpServer((req, res) => {
317+
seenUserAgents.push(req.headers['user-agent'] ?? '');
318+
res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
319+
res.end('<html><body><h1>Readable</h1></body></html>');
320+
});
321+
322+
try {
323+
await fetchAndConvertToMarkdown(mockServer as any, url);
324+
325+
assert.ok(seenUserAgents.length > 0, 'Expected the URL reader to make a request');
326+
assert.ok(seenUserAgents.every((userAgent) => userAgent === 'ReaderBot/3.0'));
327+
} finally {
328+
await close();
329+
envManager.restore();
330+
}
331+
}, results);
332+
287333
// ── HEAD content-length preflight ─────────────────────────────────────────
288334

289335
await testFunction('HEAD preflight returns Content-Length when present', async () => {

‎src/proxy.ts‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -286,16 +286,23 @@ export function createDefaultAgent(): Agent | undefined {
286286
return _defaultAgent;
287287
}
288288

289+
/**
290+
* Resolve the User-Agent for SearXNG-instance requests.
291+
*/
292+
export function getSearchUserAgent(): string | undefined {
293+
return process.env.SEARCH_USER_AGENT || process.env.USER_AGENT;
294+
}
295+
289296
/**
290297
* Apply the shared SearXNG-instance request configuration — the SEARCH-group
291-
* proxy dispatcher and the global `USER_AGENT` header — to an outgoing request.
298+
* proxy dispatcher and resolved SEARCH-group User-Agent header — to an outgoing request.
292299
*
293-
* Used by the two instance-side fetches that don't build their own auth headers:
300+
* Used by the instance-side fetches that don't build their own auth headers:
294301
* the `/config` fetch (`instance-info.ts`) and the autocompleter fetch
295302
* (`suggestions.ts`), so both route through the same proxy and present a
296303
* consistent User-Agent identity without duplicating the wiring. `searxng_web_search`
297-
* still builds its own options in `search.ts` (Basic-Auth handling is interleaved
298-
* there); folding it in here is tracked under FEAT-050.
304+
* builds its own options in `search.ts` because Basic Auth handling is interleaved
305+
* there, but uses the same `getSearchUserAgent()` resolver.
299306
*
300307
* The User-Agent is merged through a `Headers` instance, so any already-set
301308
* `headers` — whether a plain object, a `Headers` instance, or a tuple array —
@@ -312,7 +319,7 @@ export function applySearchRequestConfig(
312319
(requestOptions as any).dispatcher = dispatcher;
313320
}
314321

315-
const userAgent = process.env.USER_AGENT;
322+
const userAgent = getSearchUserAgent();
316323
if (userAgent) {
317324
// Normalize via Headers so any HeadersInit shape (plain object, Headers
318325
// instance, or tuple array) merges without dropping already-set entries,

‎src/search.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
22
import { parse } from "node-html-parser";
33
import { SearXNGWeb } from "./types.js";
44
import { getKnownCategories, getKnownEngines } from "./instance-info.js";
5-
import { createProxyAgent, createDefaultAgent, ProxyType } from "./proxy.js";
5+
import { createProxyAgent, createDefaultAgent, getSearchUserAgent, ProxyType } from "./proxy.js";
66
import { logMessage } from "./logging.js";
77
import {
88
getHealthySearxngInstances,
@@ -449,7 +449,7 @@ function buildSearchRequestOptions(url: URL): RequestInit {
449449
};
450450
}
451451

452-
const userAgent = process.env.USER_AGENT;
452+
const userAgent = getSearchUserAgent();
453453
if (userAgent) {
454454
requestOptions.headers = {
455455
...requestOptions.headers,

0 commit comments

Comments
 (0)