feat(agents): file navigation, web browsing, scratchpad tools, and write security guardrails

[kovtcharov]

Summary

Before: monolithic ChatAgent with a 13K-token system prompt and 22 tools caused 95s TTFT on local models. Write operations had zero security checks. Users had to manually find files, download web content, and do data analysis outside GAIA.

After: ChatAgent is split into 5 focused agents (chat, doc, file, data, web) with lean prompt profiles, plus a centralized write-guardrail layer and 3 new tool groups. TTFT drops from 95s to 0.12s (chat) / 3-10s (doc). Eval pass rate: 87-89% judged.

Agent split

Agent	Profile	Prompt size	Tools	Purpose
chat	conversation only	~2K tokens	none	Fast greetings, general chat
doc	RAG + file search	~5K tokens	RAG, file search	Document Q&A with hallucination prevention
file	filesystem ops	~4K tokens	browse, tree, find, read, bookmark	File navigation and discovery
data	scratchpad + CSV	~3K tokens	create_table, insert, query, list, drop	Multi-document structured analysis
web	browser tools	~2K tokens	fetch_page, search_web, download_file	Web research and content extraction

Each has a -lite variant using a ~4B model for low-memory hardware. Per-scenario agent_type field in eval YAML routes scenarios to the right agent.

Security hardening

• Write guardrails (src/gaia/security.py): blocked directories (incl. /var/log, /var/lib, /var/spool, /opt), sensitive file protection, size limits, overwrite prompts, timestamped backups, rotating audit log (10 MB x 3), symlink resolution
• SSRF prevention (src/gaia/web/client.py): PinnedIPAdapter closes DNS-rebinding TOCTOU window, monotonic rate limiter, per-hop redirect validation, blocked ports, private-IP rejection
• SQL injection defense (src/gaia/scratchpad/service.py): column DDL validation, PRAGMA/VACUUM/REINDEX blocked in queries, VACUUM on clear_all, per-call OOM guards
• Download guardrail (browser_tools.py): post-download sensitive-filename check (.env, credentials.json, etc.) — deletes and blocks if matched
• Per-instance tool registry: _snapshot_tools() prevents tool leakage across agent instances in the same process

Follow-up issues

• perf(chat): system prompt is ~6K tokens — trim for faster TTFT on local models #972 — Trim pre-existing system prompt bloat (~6K tokens)
• fix(security): CodeAgent write tools missing blocklist/guardrails #955 — CodeAgent write tools missing blocklist guardrails

Test plan

• ~500 PR-specific unit tests pass (11 new test files, ~8K LOC)
• Full unit suite passes, lint clean
• Agent eval: 87-89% judged pass rate, 100% on personality/RAG/adversarial/web scenarios
• All 10 critical CI checks pass
• 2 remaining CodeQL alerts documented as false positives (EMR dashboard)

[kovtcharov]

@claude review this PR.

[github-actions]

Claude finished @kovtcharov's task —— View job

---

PR Review Complete

• Read PR diff and changed files
• Review security module (src/gaia/security.py)
• Review filesystem tools (src/gaia/agents/tools/filesystem_tools.py)
• Review browser tools and web client
• Review scratchpad service
• Review ChatAgent integration
• Review CodeAgent file_io.py security fixes
• Check CodeQL security alerts
• Provide comprehensive review feedback

---

Overall Assessment: ✅ Approve with Minor Suggestions

This is a well-designed, security-conscious PR that adds significant functionality to the ChatAgent. The write security guardrails, SSRF prevention, and audit logging are particularly well-implemented. Below are my detailed findings.

---

🟢 Strengths

Security Module (src/gaia/security.py)

• Excellent blocked directory coverage for both Windows and Unix systems
• Sensitive file protection (.env, keys, certificates) is comprehensive
• Symlink resolution via os.path.realpath() prevents TOCTOU attacks
• Fail-closed design: blocks writes when path validation fails
• Audit logging provides valuable forensics capability

Web Client (src/gaia/web/client.py)

• Robust SSRF prevention: validates scheme, port, and resolved IP addresses
• Manual redirect following with validation at each hop prevents redirect-based SSRF
• Per-domain rate limiting prevents accidental DoS
• Content-Length validation before download prevents memory exhaustion
• Path traversal protection in download() via save_path.resolve() check

Scratchpad Service (src/gaia/scratchpad/service.py)

• Good SQL injection prevention via _sanitize_name()
• Read-only query enforcement (SELECT only) is correctly implemented
• Resource limits (100 tables, 1M rows) prevent runaway resource usage
• WAL mode enables concurrent access

CodeAgent File I/O (src/gaia/agents/code/tools/file_io.py)

• Fixed missing PathValidator checks on write_file and edit_file tools
• Automatic backup creation before overwrites
• Audit logging integrated

---

🟡 Minor Issues (Non-blocking)

1. CodeQL Alert: URL Substring Sanitization in Tests
tests/unit/test_browser_tools.py:155-162

The CodeQL alerts about URL substring sanitization are false positives in this context. These are unit tests validating rate limiter domain tracking, not security-sensitive URL validation. The test code:

self.client._rate_limit_wait("example.com")
assert "example.com" in self.client._domain_last_request

is simply checking that domains are correctly stored in the rate limit tracking dictionary. No action needed.

2. CodeQL Alert: DOM Text as HTML
src/gaia/apps/jira/webui/public/js/modules/chat-ui.js:51

The sanitizeHTML() function at line 49-64 already addresses this by removing dangerous elements (script, iframe, etc.) and stripping event handlers. However, consider adding data: and vbscript: to the URL scheme check at line 58:

// Current check:
if (attr.name === 'href' && attr.value.trimStart().toLowerCase().startsWith('javascript:'))

// Suggested enhancement:
const dangerousSchemes = ['javascript:', 'data:', 'vbscript:'];
if (attr.name === 'href' && dangerousSchemes.some(s => attr.value.trimStart().toLowerCase().startsWith(s)))

3. CodeQL Alert: Polynomial Regex
src/gaia/agents/emr/dashboard/server.py:76-84

The regex patterns in sanitize_error_response() could theoretically cause ReDoS with crafted input. However, these are applied to error messages (internal content), not user input. The risk is low since error messages come from Python exceptions. If you want to harden this, consider using simpler patterns or adding a length limit before regex processing.

4. Potential Enhancement: Browser Tools Path Validation
src/gaia/agents/tools/browser_tools.py:259-266

The download_file tool checks is_path_allowed but doesn't call validate_write() to check blocked directories. Consider:

# After path validation:
is_blocked, reason = mixin._path_validator.is_write_blocked(resolved_dir)
if is_blocked:
    return f"Error: {reason}"

---

🔴 Security Alerts from CodeQL (Already Addressed or Pre-existing)

The following alerts appear to be pre-existing issues or already addressed in this PR:

Alert	File	Status
Server-side URL redirect	docs/server.js:295	✅ Fixed in this PR - Added // prefix check
Clear-text password logging	jira/agent.py:654	⚠️ Pre-existing - Only logs URL path, not credentials
Uncontrolled path expression	emr/dashboard/server.py:1659	✅ Fixed in this PR - Added path validation
CDN integrity check	eval/webapp/public/index.html:9-10	🟡 Pre-existing - Consider adding SRI hashes

---

📋 Test Coverage

The test coverage is excellent:

• 11 new test files with ~8000 lines of tests
• Comprehensive security edge case testing (symlinks, TOCTOU, path traversal)
• SSRF prevention tests
• SQL injection prevention tests

---

Summary

This PR significantly improves GAIA's security posture by:

1. Adding comprehensive write guardrails
2. Implementing robust SSRF prevention for web browsing
3. Fixing missing PathValidator checks in CodeAgent tools
4. Adding audit logging for all file mutations

The minor suggestions above are non-blocking. The code is ready to merge.

---

[itomek]

Approving. The PR has had multiple rounds of automated review with substantive iteration from the author — most recently the system-prompt gating fix, the edit_file size-cap enforcement, the bookmarks per-instance fix, and the dev-server rate-limiter eviction. The remaining items from the latest bot pass (inline import sys, EMR comment polish) are appropriate for follow-up rather than blocking this merge.

Net: the security work (write guardrails, SSRF defense-in-depth, SQL injection layering, FTS5 self-healing index) is solid, test coverage is real, and the feature surface is gated correctly behind config flags. Ready to land.

---

Generated by Claude Code

[kovtcharov-amd]

Posting some results from eval testing below:

[kovtcharov-amd]

@claude review this PR.

[github-actions]

PR Review — #495

Strong PR overall. The PathValidator write guardrails (src/gaia/security.py), SQL-injection defenses in ScratchpadService, and the macOS /private symlink handling are all well-thought-out, and the test coverage (~8000 lines across 11 files) is genuinely impressive. A few items worth tightening before merge:

🟡 Important

1. SSRF — DNS rebinding bypass in WebClient._validate_host_ip (src/gaia/web/client.py:132-156)

validate_url() resolves the hostname via socket.getaddrinfo(), then requests.Session.request() performs a second independent DNS lookup at fetch time. A hostile DNS server returning a public IP on the first query and a private IP (e.g., 169.254.169.254 AWS metadata, 127.0.0.1) on the second bypasses every SSRF check.

The standard fix is to resolve once, validate the IP, then connect to the IP directly with the original Host header preserved — or pin socket.getaddrinfo results via a custom requests.adapters.HTTPAdapter. Worth fixing now since this surface also handles user-controllable redirect URLs.

2. download_file skips the sensitive-filename guardrail (src/gaia/agents/tools/browser_tools.py:259-275, src/gaia/web/client.py:485-609)

The download path checks is_path_allowed and is_write_blocked on the directory but never passes the final resolved file path through validate_write. So an agent that downloads to ~/Downloads/credentials.json or ~/Downloads/id_rsa succeeds, even though write_file blocks the same destination. Suggested fix: after _sanitize_filename, run the resolved save_path through path_validator.is_write_blocked (which already covers SENSITIVE_FILE_NAMES + SENSITIVE_EXTENSIONS).

3. Blocked-directories list misses common Linux write targets (src/gaia/security.py:107-128)

Only /var/run is in the Unix blocklist — but /var/log, /var/lib, /var/spool, and /opt are equally problematic for an agent to scribble into. Consider expanding, or at minimum adding /var as a parent guard with explicit allowlist for safe subpaths.

🟢 Minor

4. Inconsistent path_validator access in file_io.py

read_file/write_python_file use self.path_validator.is_path_allowed(...) directly (src/gaia/agents/code/tools/file_io.py:54, 224, 269), while the new write_file/edit_file use getattr(self, "path_validator", None) and gracefully no-op when missing (lines 563, 655). Pick one — silent degradation hides bugs (and the no-validator path would violate the "no silent fallback" rule in CLAUDE.md).

5. WebClient._consume_body_capped mutates private requests attrs (src/gaia/web/client.py:285-308)

Setting response._content / response._content_consumed is the right idea for capping decompression bombs, but it depends on requests internals that have changed across versions. Worth a unit test that reads response.text after the cap and asserts truncation, plus a comment that this needs revisiting when bumping requests.

6. ScratchpadService.clear_all drops tables but doesn't reclaim disk

After DROP TABLE, SQLite holds the pages until VACUUM. With MAX_TOTAL_SIZE_BYTES = 100MB enforced via the get_size_bytes() heuristic (src/gaia/scratchpad/service.py:349-369), a heavy create/drop cycle can leave the DB at 95 MB on disk while reporting 0 bytes — eventually breaking writes against the disk-size cap rather than the row cap. Either run VACUUM after clear_all, or document that the limit is row-based by design.

✅ Things that are clearly right

• _normalize_macos_symlinks (security.py:141) — solves a real bypass that's easy to miss
• is_path_allowed appending os.sep to defeat the /home/user/project vs /home/user/project-secrets prefix attack (security.py:303-313)
• Manual redirect following with per-hop SSRF re-validation in _request (client.py:204-272)
• _validate_columns with hard-deny on ;/--//* plus type-root allowlist — robust SQL DDL validation without a full parser (scratchpad/service.py:421-512)
• _strip_sql_string_literals for keyword scanning (avoids false positives on 'UPDATE PENDING')
• Fail-loud audit logging for write/edit operations
• _sanitize_filename covering Windows reserved device names + leading-dot + length cap

Suggested next steps

1. Fix SSRF DNS rebinding (Update installer and workflows/actions for CI/CD #1) before merge — this is the only finding I'd consider blocking.
2. Add the sensitive-filename check to download_file (Use public Lemonade hybrid installer #2).
3. Reconcile the path_validator access pattern in file_io.py (Documentation update from v0.7.2 tag. #4).
4. Items Update Driver Check #3, Update Documentation #5, Prevent Users from Installing Hybrid mode on Unsupported Systems #6 can be follow-ups if the user wants to keep this PR focused.

Tagging @kovtcharov-amd for the SSRF item — DNS rebinding is a known-tricky bypass and worth a maintainer's eye on the fix.