feat(agents): file navigation, web browsing, scratchpad tools, and write security guardrails (#495)

## Summary

Before: monolithic ChatAgent with a 13K-token system prompt and 22 tools
caused 95s TTFT on local models. Write operations had zero security
checks. Users had to manually find files, download web content, and do
data analysis outside GAIA.

After: ChatAgent is split into 5 focused agents (chat, doc, file, data,
web) with lean prompt profiles, plus a centralized write-guardrail layer
and 3 new tool groups. TTFT drops from 95s to 0.12s (chat) / 3-10s
(doc). Eval pass rate: 87-89% judged.

### Agent split

| Agent | Profile | Prompt size | Tools | Purpose |
|-------|---------|-------------|-------|---------|
| `chat` | conversation only | ~2K tokens | none | Fast greetings,
general chat |
| `doc` | RAG + file search | ~5K tokens | RAG, file search | Document
Q&A with hallucination prevention |
| `file` | filesystem ops | ~4K tokens | browse, tree, find, read,
bookmark | File navigation and discovery |
| `data` | scratchpad + CSV | ~3K tokens | create_table, insert, query,
list, drop | Multi-document structured analysis |
| `web` | browser tools | ~2K tokens | fetch_page, search_web,
download_file | Web research and content extraction |

Each has a `-lite` variant using a ~4B model for low-memory hardware.
Per-scenario `agent_type` field in eval YAML routes scenarios to the
right agent.

### Security hardening

- **Write guardrails** (`src/gaia/security.py`): blocked directories
(incl. `/var/log`, `/var/lib`, `/var/spool`, `/opt`), sensitive file
protection, size limits, overwrite prompts, timestamped backups,
rotating audit log (10 MB x 3), symlink resolution
- **SSRF prevention** (`src/gaia/web/client.py`): `PinnedIPAdapter`
closes DNS-rebinding TOCTOU window, monotonic rate limiter, per-hop
redirect validation, blocked ports, private-IP rejection
- **SQL injection defense** (`src/gaia/scratchpad/service.py`): column
DDL validation, `PRAGMA`/`VACUUM`/`REINDEX` blocked in queries, VACUUM
on clear_all, per-call OOM guards
- **Download guardrail** (`browser_tools.py`): post-download
sensitive-filename check (`.env`, `credentials.json`, etc.) — deletes
and blocks if matched
- **Per-instance tool registry**: `_snapshot_tools()` prevents tool
leakage across agent instances in the same process

### Follow-up issues

- #972 — Trim pre-existing system prompt bloat (~6K tokens)
- #955 — CodeAgent write tools missing blocklist guardrails

## Test plan

- [x] ~500 PR-specific unit tests pass (11 new test files, ~8K LOC)
- [x] Full unit suite passes, lint clean
- [x] Agent eval: 87-89% judged pass rate, 100% on
personality/RAG/adversarial/web scenarios
- [x] All 10 critical CI checks pass
- [x] 2 remaining CodeQL alerts documented as false positives (EMR
dashboard)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ovtcharov <kovtchar@amd.com>

Author: 3 people

.github/workflows/test_unit.yml

@@ -58,11 +58,11 @@ jobs:
           # pyfakefs is required by tests/unit/installer/test_uninstall_command.py
           # which uses the `fs` fixture to build a fake filesystem for testing
           # tiered uninstall logic cross-platform without touching the real FS.
-          #
+          # pytest-mock is required by the browser/filesystem tool tests.
           # keyring + httpx + respx are required by tests/unit/connections/
           # (issue #915). The in-memory keyring backend in tests/conftest.py
           # avoids the SecretService daemon prerequisite on Linux runners.
-          uv pip install --system pytest pytest-cov pytest-asyncio pyfakefs \
+          uv pip install --system pytest pytest-cov pytest-asyncio pytest-mock pyfakefs \
                                   keyring httpx respx
           uv pip install --system -e ".[api]"
 
@@ -140,6 +140,17 @@ jobs:
           echo "  - ASR: Automatic speech recognition utilities"
           echo "  - TTS: Text-to-speech utilities"
           echo "  - InitCommand: gaia init profiles and installer logic"
+          echo "  - FileSystemIndex: Persistent file index with FTS5 search"
+          echo "  - FileSystemToolsMixin: browse_directory, tree, file_info, find_files, read_file, bookmark tools"
+          echo "  - ScratchpadService: SQLite working memory for data analysis"
+          echo "  - ScratchpadToolsMixin: create_table, insert_data, query_data, list_tables, drop_table tools"
+          echo "  - BrowserTools: WebClient SSRF prevention, HTML extraction, downloads"
+          echo "  - WebClient Edge Cases: parse_html fallback, extract_text, tables, links, download redirects"
+          echo "  - Categorizer: auto_categorize, category map completeness, extension uniqueness"
+          echo "  - ChatAgent Integration: filesystem, scratchpad, browser init/config/cleanup"
+          echo "  - File Write Guardrails: blocked dirs, sensitive files, size limits, backup, audit"
+          echo "  - Security Edge Cases: symlinks, audit logging, TOCTOU, prompt_overwrite"
+          echo "  - Service Edge Cases: DB corruption rebuild, shared DB, row limits, transaction atomicity"
           echo ""
           echo "Integration Tests:"
           echo "  - DatabaseMixin + Agent: Full agent lifecycle with database"

docs/docs.json

@@ -287,7 +287,9 @@
               "spec/prisma-tools-mixin",
               "spec/typescript-tools-mixin",
               "spec/external-tools-mixin",
-              "spec/web-tools-mixin"
+              "spec/web-tools-mixin",
+              "spec/browser-tools",
+              "spec/file-system-agent"
             ]
           },
           {

docs/server.js

@@ -271,8 +271,35 @@ const loginLimiter = rateLimit({
   legacyHeaders: false,
 });
 
+// General per-IP rate limiter for all auth endpoints (not just /login).
+// Defined here so it can be applied to every auth route below, closing the
+// "missing rate-limiting" CodeQL alert on /auth/logout and
+// /auth/login-error which would otherwise accept unlimited requests.
+const rateLimitStore = new Map();
+const RATE_LIMIT_WINDOW = 60 * 1000; // 1 minute
+const RATE_LIMIT_MAX = 100; // max requests per window per IP
+
+function rateLimiter(req, res, next) {
+  const ip = req.ip || req.connection.remoteAddress;
+  const now = Date.now();
+  const record = rateLimitStore.get(ip) || { count: 0, resetAt: now + RATE_LIMIT_WINDOW };
+
+  if (now > record.resetAt) {
+    record.count = 0;
+    record.resetAt = now + RATE_LIMIT_WINDOW;
+  }
+
+  record.count++;
+  rateLimitStore.set(ip, record);
+
+  if (record.count > RATE_LIMIT_MAX) {
+    return res.status(429).send('Too Many Requests');
+  }
+  next();
+}
+
 // Login handler
-app.post('/auth/login', loginLimiter, (req, res) => {
+app.post('/auth/login', loginLimiter, rateLimiter, (req, res) => {
   const { code, nonce } = req.body;
 
   if (code === ACCESS_CODE) {
@@ -285,15 +312,29 @@ app.post('/auth/login', loginLimiter, (req, res) => {
       maxAge: COOKIE_MAX_AGE,
       sameSite: 'lax'
     });
-    // Retrieve redirect URL from server-side storage and validate with url.parse()
+    // Server-side redirect target. Instead of validating the user-supplied
+    // pathname and forwarding it (which CodeQL's
+    // js/server-side-unvalidated-url-redirection analyzer can't prove safe),
+    // we maintain an explicit allowlist of post-login destinations and
+    // round-trip the incoming pathname through it. Anything that doesn't
+    // exactly match a known-safe path falls back to '/'.
+    const ALLOWED_POST_LOGIN_PATHS = new Set([
+      '/',
+      '/index.html',
+    ]);
     const target = consumeRedirect(nonce);
     const parsed = url.parse(target || '');
-    // Only redirect to relative paths (no host/protocol) to prevent open redirects
-    if (!parsed.host && !parsed.protocol && parsed.pathname) {
-      res.redirect(303, parsed.pathname);
-    } else {
-      res.redirect(303, '/');
-    }
+    const pathname = parsed.pathname || '/';
+    // Block open-redirects and traversal before the allowlist check.
+    const structurallySafe =
+      !parsed.host &&
+      !parsed.protocol &&
+      pathname.startsWith('/') &&
+      !pathname.startsWith('//') &&
+      !pathname.split('/').includes('..');
+    const resolvedPath =
+      structurallySafe && ALLOWED_POST_LOGIN_PATHS.has(pathname) ? pathname : '/';
+    res.redirect(303, resolvedPath);
   } else {
     // Retrieve the original redirect URL and re-store with a new nonce for retry
     const originalRedirect = consumeRedirect(nonce);
@@ -303,7 +344,7 @@ app.post('/auth/login', loginLimiter, (req, res) => {
 });
 
 // Login error handler (uses nonce to retrieve redirect URL)
-app.get('/auth/login-error', (req, res) => {
+app.get('/auth/login-error', rateLimiter, (req, res) => {
   // Retrieve redirect URL from server-side storage and re-store for the form
   const originalRedirect = consumeRedirect(req.query.nonce);
   const newNonce = storeRedirect(originalRedirect);
@@ -312,11 +353,14 @@ app.get('/auth/login-error', (req, res) => {
 });
 
 // Logout handler
-app.get('/auth/logout', (req, res) => {
+app.get('/auth/logout', rateLimiter, (req, res) => {
   res.clearCookie(COOKIE_NAME);
   res.redirect('/');
 });
 
+// Apply rate limiter before auth middleware for every other route
+app.use(rateLimiter);
+
 // Apply auth middleware
 app.use(authMiddleware);