Enhance ChatAgent with file navigation, web browsing, scratchpad tools, and write security guardrails

[kovtcharov]

Summary

This PR adds comprehensive file system navigation, web browsing tools, structured data analysis, and write security guardrails to the ChatAgent.

Write Security Guardrails (src/gaia/security.py)

• Blocked system directories: Windows (C:\Windows, Program Files) and Unix (/etc, /bin, /usr/lib) system paths are blocked for writes
• Sensitive file protection: .env, credentials.json, SSH keys (id_rsa, id_ed25519), certificates (.pem, .key, .crt), and other secrets are never writable
• Write size limits: 10 MB maximum per write operation to prevent runaway file creation
• Overwrite confirmation prompts: User is prompted before overwriting existing files
• Timestamped backups: Automatic .bak copies created before file modification
• Audit logging: All write operations logged to ~/.gaia/cache/file_audit.log with timestamp, operation type, path, size, and status
• Symlink resolution: Paths resolved via os.path.realpath() to prevent TOCTOU bypass
• Fixed ChatAgent write_file: Previously had zero security checks — now enforces full PathValidator + write guardrails
• Fixed CodeAgent write_file/edit_file: Generic file tools were missing PathValidator — now enforced

File System Navigation Tools (src/gaia/agents/tools/filesystem_tools.py)

• browse_directory: List folder contents with file sizes, dates, and type indicators
• tree: Visual directory tree with configurable depth, exclusion patterns, and platform-aware defaults
• find_files: Search by name, content, size, date, and file type with multi-scope search (current dir → common locations → full drives)
• file_info: Detailed metadata — size, type, MIME, modification date, line counts, PDF page counts
• read_file: Smart file reading with type detection — text, CSV (tabular), JSON (formatted), PDF (text extraction)
• bookmark: Save, list, and remove bookmarks for quick access to important locations

File System Index Service (src/gaia/filesystem/)

• FileSystemIndexService: Persistent SQLite-backed file index with FTS5 full-text search
• auto_categorize: Automatic file categorization by extension (code, document, spreadsheet, image, video, audio, data, archive, config)
• Supports incremental scanning and update-on-change for efficient re-indexing

Browser Tools (src/gaia/agents/tools/browser_tools.py)

• fetch_page: Fetch web pages with content extraction modes — readable text, raw HTML, links, or tables as JSON
• search_web: DuckDuckGo web search (no API key required) with configurable result count
• download_file: Download files from the web to local disk with size limits and path validation

Web Client (src/gaia/web/client.py)

• Rate limiting: Per-domain request throttling (configurable delay between requests)
• SSRF prevention: Blocked schemes (file://, ftp://), blocked ports (SSH, SMTP, DB ports), private IP detection
• Content extraction: BeautifulSoup-based text extraction with boilerplate removal (nav, footer, scripts stripped)
• Table extraction: HTML tables parsed to structured JSON
• Size limits: Configurable max download size (default 100 MB)
• User-Agent rotation: Realistic browser user-agent strings

Scratchpad Tools (src/gaia/agents/tools/scratchpad_tools.py)

• create_table: Create SQLite tables for structured data accumulation
• insert_data: Insert rows from extracted document data
• query_data: Run SQL queries (SELECT only) with formatted results — supports SUM, AVG, GROUP BY for analysis
• list_tables: Show all scratchpad tables with row counts and schemas
• drop_table: Clean up tables when analysis is complete

Scratchpad Service (src/gaia/scratchpad/service.py)

• SQLite-backed working memory for multi-document data analysis
• Table name prefixing (scratch_) to isolate scratchpad data
• Read-only query enforcement (SELECT only) to prevent data mutation via query tool
• Schema introspection and row count tracking

ChatAgent Integration (src/gaia/agents/chat/agent.py)

• Integrated FileSystemToolsMixin, ScratchpadToolsMixin, and BrowserToolsMixin
• Config toggles: enable_filesystem, enable_scratchpad, enable_browser (all default to True)
• Updated system prompt with new tool workflows: file search + auto-index, data analysis pipeline, web research, download + analyze
• Replaced legacy search_file/search_directory tools with enhanced find_files/browse_directory
• Graceful degradation: each service initializes independently with fallback on import errors

CI Updates (.github/workflows/test_unit.yml)

• Added beautifulsoup4 and requests to test dependencies for browser tool tests

New Modules

Module	Files	Description
src/gaia/filesystem/	index.py, categorizer.py	Persistent file index with FTS5 search and auto-categorization
src/gaia/web/	client.py	HTTP client with rate limiting, SSRF prevention, content extraction
src/gaia/scratchpad/	service.py	SQLite working memory for structured data analysis
src/gaia/agents/tools/filesystem_tools.py	—	File system navigation mixin (6 tools)
src/gaia/agents/tools/browser_tools.py	—	Web browsing mixin (3 tools)
src/gaia/agents/tools/scratchpad_tools.py	—	Data analysis mixin (5 tools)

Test Coverage

Test File	Focus
test_file_write_guardrails.py	Blocked directories, sensitive files, size limits, backups, audit logging, overwrite prompts
test_security_edge_cases.py	Symlink resolution, path traversal, platform-specific edge cases
test_filesystem_tools_mixin.py	browse_directory, tree, find_files, file_info, read_file, bookmarks
test_filesystem_index.py	FTS5 search, incremental scanning, categorization
test_categorizer.py	Extension-based file categorization
test_browser_tools.py	URL validation, SSRF prevention, content extraction, rate limiting
test_web_client_edge_cases.py	Timeout handling, redirect limits, encoding detection
test_scratchpad_service.py	Table CRUD, SQL injection prevention, schema introspection
test_scratchpad_tools_mixin.py	Tool registration, query formatting, error handling
test_service_edge_cases.py	Concurrent access, large datasets, cleanup
test_chat_agent_integration.py	End-to-end ChatAgent with all new mixins

Test plan

• All unit tests pass (11 new test files, ~8000 lines of test code)
• All 3 modified source files parse and import cleanly
• Integration test: write to safe file succeeds, write to .env blocked, edit creates backup
• Platform test: case-insensitive path comparison on Windows verified
• Manual: run gaia chat and test file browsing, web search, scratchpad tools
• Manual: verify audit log written to ~/.gaia/cache/file_audit.log after write operations

🤖 Generated with Claude Code

[kovtcharov]

@claude review this PR.

[github-actions]

Claude finished @kovtcharov's task —— View job

---

PR Review Complete

• Read PR diff and changed files
• Review security module (src/gaia/security.py)
• Review filesystem tools (src/gaia/agents/tools/filesystem_tools.py)
• Review browser tools and web client
• Review scratchpad service
• Review ChatAgent integration
• Review CodeAgent file_io.py security fixes
• Check CodeQL security alerts
• Provide comprehensive review feedback

---

Overall Assessment: ✅ Approve with Minor Suggestions

This is a well-designed, security-conscious PR that adds significant functionality to the ChatAgent. The write security guardrails, SSRF prevention, and audit logging are particularly well-implemented. Below are my detailed findings.

---

🟢 Strengths

Security Module (src/gaia/security.py)

• Excellent blocked directory coverage for both Windows and Unix systems
• Sensitive file protection (.env, keys, certificates) is comprehensive
• Symlink resolution via os.path.realpath() prevents TOCTOU attacks
• Fail-closed design: blocks writes when path validation fails
• Audit logging provides valuable forensics capability

Web Client (src/gaia/web/client.py)

• Robust SSRF prevention: validates scheme, port, and resolved IP addresses
• Manual redirect following with validation at each hop prevents redirect-based SSRF
• Per-domain rate limiting prevents accidental DoS
• Content-Length validation before download prevents memory exhaustion
• Path traversal protection in download() via save_path.resolve() check

Scratchpad Service (src/gaia/scratchpad/service.py)

• Good SQL injection prevention via _sanitize_name()
• Read-only query enforcement (SELECT only) is correctly implemented
• Resource limits (100 tables, 1M rows) prevent runaway resource usage
• WAL mode enables concurrent access

CodeAgent File I/O (src/gaia/agents/code/tools/file_io.py)

• Fixed missing PathValidator checks on write_file and edit_file tools
• Automatic backup creation before overwrites
• Audit logging integrated

---

🟡 Minor Issues (Non-blocking)

1. CodeQL Alert: URL Substring Sanitization in Tests
tests/unit/test_browser_tools.py:155-162

The CodeQL alerts about URL substring sanitization are false positives in this context. These are unit tests validating rate limiter domain tracking, not security-sensitive URL validation. The test code:

self.client._rate_limit_wait("example.com")
assert "example.com" in self.client._domain_last_request

is simply checking that domains are correctly stored in the rate limit tracking dictionary. No action needed.

2. CodeQL Alert: DOM Text as HTML
src/gaia/apps/jira/webui/public/js/modules/chat-ui.js:51

The sanitizeHTML() function at line 49-64 already addresses this by removing dangerous elements (script, iframe, etc.) and stripping event handlers. However, consider adding data: and vbscript: to the URL scheme check at line 58:

// Current check:
if (attr.name === 'href' && attr.value.trimStart().toLowerCase().startsWith('javascript:'))

// Suggested enhancement:
const dangerousSchemes = ['javascript:', 'data:', 'vbscript:'];
if (attr.name === 'href' && dangerousSchemes.some(s => attr.value.trimStart().toLowerCase().startsWith(s)))

3. CodeQL Alert: Polynomial Regex
src/gaia/agents/emr/dashboard/server.py:76-84

The regex patterns in sanitize_error_response() could theoretically cause ReDoS with crafted input. However, these are applied to error messages (internal content), not user input. The risk is low since error messages come from Python exceptions. If you want to harden this, consider using simpler patterns or adding a length limit before regex processing.

4. Potential Enhancement: Browser Tools Path Validation
src/gaia/agents/tools/browser_tools.py:259-266

The download_file tool checks is_path_allowed but doesn't call validate_write() to check blocked directories. Consider:

# After path validation:
is_blocked, reason = mixin._path_validator.is_write_blocked(resolved_dir)
if is_blocked:
    return f"Error: {reason}"

---

🔴 Security Alerts from CodeQL (Already Addressed or Pre-existing)

The following alerts appear to be pre-existing issues or already addressed in this PR:

Alert	File	Status
Server-side URL redirect	docs/server.js:295	✅ Fixed in this PR - Added // prefix check
Clear-text password logging	jira/agent.py:654	⚠️ Pre-existing - Only logs URL path, not credentials
Uncontrolled path expression	emr/dashboard/server.py:1659	✅ Fixed in this PR - Added path validation
CDN integrity check	eval/webapp/public/index.html:9-10	🟡 Pre-existing - Consider adding SRI hashes

---

📋 Test Coverage

The test coverage is excellent:

• 11 new test files with ~8000 lines of tests
• Comprehensive security edge case testing (symlinks, TOCTOU, path traversal)
• SSRF prevention tests
• SQL injection prevention tests

---

Summary

This PR significantly improves GAIA's security posture by:

1. Adding comprehensive write guardrails
2. Implementing robust SSRF prevention for web browsing
3. Fixing missing PathValidator checks in CodeAgent tools
4. Adding audit logging for all file mutations

The minor suggestions above are non-blocking. The code is ready to merge.

---

[kovtcharov]

Bulletproofing / CI status after final push (e03682e)

✅ All 10 critical CI checks pass

MCP Tests (Linux + Windows), RAG Unit Tests, Run Code Quality Checks, Run Unit Tests, Security Tests (Linux), Test Chat Agent, Test Code Agent, Test GAIA CLI on Linux + Windows.

Local: 581 new-PR-code tests pass / 0 fail / 36 skip, black + isort clean.

🟡 2 CodeQL alerts remain — documented false positives

Both on src/gaia/agents/emr/dashboard/server.py. Every structural code change I attempted (regex-group rebuild, os.path.normpath round-trip, compile-time-constant returns, int() coercion, named sanitizer constants, dict-comp removal) either had no effect or made them worse. They are analyzer limitations, not code defects.

Alert #263 — py/path-injection at update_watch_dir

Flags Path(validated_watch_dir).expanduser().resolve(). The input reaches this call only after a 5-layer validation chain:

1. Length bound (1–4096 chars)
2. Character-class allowlist via _VALID_WATCH_DIR_RE
3. Regex-group re-extraction (m.group(0))
4. .. traversal rejection
5. Post-resolve symlink / home-prefix-boundary / sensitive-dir-denylist checks

CodeQL's taint-flow analyzer cannot follow composite validation. Requested action: dismiss as false positive with rationale "layered validation proven by unit tests in tests/unit/test_security_edge_cases.py and tests/unit/test_file_write_guardrails.py".

Alert #266 — py/stack-trace-exposure at clear_database success return

Flags the return {...} dict. Every value in the dict is either:

• A compile-time string constant ("Database cleared successfully")
• A compile-time bool constant (True)
• An int()-coerced integer (extracted via int(result.get("deleted", {}).get(KEY, 0) or 0))

No untrusted string / exception text can flow to the response. The error branch separately routes through _sanitize_response_text (ReDoS-hardened, strips tracebacks / file refs / exception names).

Requested action: dismiss as false positive with rationale "return body is fixed-shape constants + int()-coerced integers; no taint path to client".

Optional future follow-up

• Migrate CodeQL from GitHub's default setup → advanced setup
• Add .github/codeql/codeql-config.yml registering _sanitize_response_text and _VALID_WATCH_DIR_RE as custom sanitizers
• This would close both alerts automatically without any code changes
• Out of scope for this PR; repo-level decision

🟠 Non-critical CI failures (pre-existing, not this PR)

• Test Electron Framework: fails on main since the forge→electron-builder migration. tests/electron/test_electron_chat_app.js references pkg.config.forge which no longer exists. Needs its own fix PR.
• pr-review: GitHub Action internal tooling bug ("directory mismatch for directory /home/runner/work/_actions/anthropics/claude-code-action/…"). Not reproducible locally.

28 commits over this session

Full list: git log --oneline feature/chat-agent-file-navigation ^origin/main

All addressing the original two reviewer comments + the CodeQL scan diff + bulletproofing edge cases (SQL injection, SSRF, OOM, decompression bombs, Windows reserved filenames, corruption recovery, resource leaks). See individual commit messages for rationale.

---

🤖 Generated with Claude Code

[github-actions]

Summary

This PR is a substantial expansion of ChatAgent capabilities — three new tool surfaces (filesystem navigation, web browsing, SQLite scratchpad) plus a serious security hardening pass on the file-write path, EMR dashboard server, and the Jira webui XSS surface. The security work is genuinely the strongest part: layered SSRF defense in gaia.web.client.WebClient, defense-in-depth SQL-injection prevention in gaia.scratchpad.service.ScratchpadService, write guardrails (allowlist + blocklist + size cap + audit log) in gaia.security, and DOMParser-based XSS sanitization in the Jira webui.

The diff is large (+20,370 / −124, 39 files) and spans security, agents, performance, and tests. Test coverage is substantial (~8K LOC across 11 new test files), and the previously-flagged review items (e.g. edit_file bypassing the size guard, integrity checks cross-contaminating across services) appear to have been addressed.

A few rough edges remain — most notably (a) the new tool mixins are not registered in KNOWN_TOOLS so BuilderAgent can't compose them by name, and (b) the new spec docs aren't wired into docs/docs.json so they won't render on amd-gaia.ai. Neither blocks merge but both should land before the v0.17.6 cut.

Issues

🟡 Important

1. New tool mixins not registered in KNOWN_TOOLS

src/gaia/agents/registry.py:28-37 exposes a curated KNOWN_TOOLS map that BuilderAgent uses to scaffold tool-mixin imports for generated agents. This PR adds three new mixins (BrowserToolsMixin, FileSystemToolsMixin, ScratchpadToolsMixin) but does not register them. Per CLAUDE.md ("When adding a new tool mixin, register it in KNOWN_TOOLS so other agents can compose it by name"), this is a documented convention.

KNOWN_TOOLS: Dict[str, tuple] = {
    "rag": ("gaia.agents.chat.tools.rag_tools", "RAGToolsMixin"),
    "code_index": ("gaia.agents.code_index.tools.mixin", "CodeIndexToolsMixin"),
    "file_search": ("gaia.agents.tools.file_tools", "FileSearchToolsMixin"),
    "file_io": ("gaia.agents.code.tools.file_io", "FileIOToolsMixin"),
    "shell": ("gaia.agents.chat.tools.shell_tools", "ShellToolsMixin"),
    "screenshot": ("gaia.agents.tools.screenshot_tools", "ScreenshotToolsMixin"),
    "sd": ("gaia.sd.mixin", "SDToolsMixin"),
    "vlm": ("gaia.vlm.mixin", "VLMToolsMixin"),
    "filesystem": ("gaia.agents.tools.filesystem_tools", "FileSystemToolsMixin"),
    "scratchpad": ("gaia.agents.tools.scratchpad_tools", "ScratchpadToolsMixin"),
    "browser": ("gaia.agents.tools.browser_tools", "BrowserToolsMixin"),
}

2. New spec docs not wired into docs/docs.json

docs/spec/browser-tools.mdx and docs/spec/file-system-agent.mdx are added but docs/docs.json is not updated, so neither will render on amd-gaia.ai. CLAUDE.md flags this explicitly: "Update docs/docs.json — Add new pages to the appropriate navigation section." Add both under the Specifications group.

3. Both .md and .mdx versions of new specs committed

docs/spec/browser-tools.md + docs/spec/browser-tools.mdx, and docs/spec/file-system-agent.md + docs/spec/file-system-agent.mdx — Mintlify only consumes .mdx, so the .md copies are dead weight that will drift. Drop the .md versions before merging.

🟢 Nits / Polish

4. PR size — splitting helps future bisects.
+20,370 LOC across 6 logical pillars (filesystem service, scratchpad service, web client, browser tools, write guardrails, XSS hardening) is a lot to revert as a unit if any single pillar regresses. Not a blocker — but going forward, separate PRs for "new feature surface" vs "security hardening of existing code paths" would make git bisect and incident response cheaper.

5. DuckDuckGo uddg= extraction is fragile.
gaia.web.client.WebClient.search_duckduckgo relies on the uddg= URL-parameter convention to recover real result URLs from DDG's redirect wrapper. If DDG changes their HTML (they have, historically), search silently returns wrapper URLs. Worth a # TODO: revisit if DDG changes redirect format comment plus an integration test that asserts result.url is not a duckduckgo.com host. Lowest-cost defense: log a warning when uddg= is missing so regressions surface in logs rather than silently degrading result quality.

6. BLOCKED_DIRECTORIES is computed once at module import.
Per-process Path.home() is captured at module load. Acceptable given home rarely changes mid-process, but worth a one-line comment so future maintainers don't expect dynamic env handling.

7. gaia.scratchpad.service.ScratchpadService.get_size_bytes swallows all Exception.
scratchpad/service.py:11269-11270 returns 0 on any error. Per CLAUDE.md ("No Silent Fallbacks"), this should at minimum log at warning. Returning 0 here also means the MAX_TOTAL_SIZE_BYTES cap silently no-ops if list_tables() fails — i.e. the security guard fails open. Recommend log.warning("scratchpad size estimate failed: %s", exc) and either re-raise or return a conservative high estimate so the cap fails closed.

8. System prompt ↔ tool name drift.
The ChatAgent system prompt block in agents/chat/agent.py references find_files_content in at least one place, but the actual tool is find_files with search_type="content". Worth a grep pass to catch any other stale references before the model starts hallucinating tool names.

Strengths

• SQL-injection defense in depth (scratchpad/service.py). Hard-deny of ;, --, /*, */ before parsing; balanced-paren check; identifier regex; type allowlist; per-row column-name validation in insert_rows. Comments explicitly call out why each layer is needed (executescript multi-statement risk, dict-key interpolation in DatabaseMixin.insert). This is exemplary.
• SSRF mitigation in WebClient.validate_url. Private-IP / loopback / link-local rejection via ipaddress, scheme allowlist, blocked-port set, manual redirect chain (5 hops max) with per-hop revalidation, and response.close() on validation failure to prevent FD leaks.
• TOCTOU mitigation via os.path.realpath in the write path — caught after the symlink is resolved, not before.
• TTY-aware _is_interactive() prevents validate_write from blocking the Agent UI / API server on input() — a real footgun avoided.
• Audit logging. All write operations land in ~/.gaia/cache/file_audit.log via the dedicated gaia.security.audit logger. Good separation from the main logger.
• ReDoS hardening (emr/dashboard/server.py:_sanitize_response_text) — bounded regex quantifiers + 100KB input cap.
• XSS hardening in the Jira webui uses DOMParser + element/attribute scrubbing + URL-scheme allowlist (the javascript:/data:/vbscript: denial in URL_ATTRS). Error/system messages now use textContent only — proper layered defense.
• Auto-healing DBs. Both FileSystemIndexService and ScratchpadService run PRAGMA integrity_check on open and rebuild on corruption — much better than crashing every turn with sqlite3.DatabaseError: file is not a database.
• Service isolation. Splitting ~/.gaia/file_index.db and ~/.gaia/scratchpad.db into separate files is the right call — each integrity_check only sees its own schema.
• OOM guards in ScratchpadToolsMixin (10MB JSON cap, 10K rows per call) — sensible upper bounds at the tool boundary.
• edit_file now enforces MAX_WRITE_SIZE_BYTES on new_content.encode("utf-8") — closes the previously-flagged silent-bypass.
• Bare except: pass cleanup in gaia/security.py — replaced with a logged warning per CLAUDE.md's "No Silent Fallbacks" rule.
• Test surface (~8K LOC, 11 files). Edge cases cover security paths (test_security_edge_cases.py, test_file_write_guardrails.py), service corner cases (test_service_edge_cases.py, test_web_client_edge_cases.py), integration (test_chat_agent_integration.py), and the new mixin code paths.

Verdict

Approve, after Important items addressed. The security and data-handling work is high-quality and well-documented. The KNOWN_TOOLS registration and docs/docs.json wiring are small, mechanical follow-ups that should land before the v0.17.6 release cut. The .md/.mdx duplication should be cleaned up in the same pass. The Nits can land in a follow-up.

No private security escalation needed — this PR is a security hardening pass and the patterns are sound.