feat: Python 3.14 compatibility, REPL mode, NVIDIA provider, memory systems

Features:
- Python 3.14 compatibility (forward-compatible pydantic, puremagic)
- Interactive REPL mode with tab completion
- NVIDIA provider with qwen3-coder-480b default
- Three memory backends: hashtable, memvid QR, DuckDB
- Split install: full (2GB) vs lite (200MB)

Implementation:
- src/ii_agent/cli/: REPL entry point
- src/ii_agent/db/: Async SQLite, DuckDB
- src/ii_agent/storage/: Memory backends
- requirements-lite.txt: Minimal install

35 files changed, 3904 insertions(+), 1644 deletions(-)

Author: jnorthrup

CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+## collector branch (2025-11-19)
+
+### Features
+- **Python 3.14 compatibility**: Forward-compatible pydantic, puremagic instead of imghdr
+- **REPL mode**: Interactive CLI with tab completion for model/provider selection
+- **NVIDIA provider**: Qwen3-coder-480b default, Bayesian model ranking
+- **Memory systems**: Hashtable, memvid QR, DuckDB vector search
+- **Installation options**: Full (2GB) or lite (200MB via requirements-lite.txt)
+
+### Implementation
+- `src/ii_agent/cli/`: REPL mode entry point and implementation
+- `src/ii_agent/db/`: Async SQLite, DuckDB integration
+- `src/ii_agent/storage/`: Three memory backend options
+- `src/ii_agent/server/api/nvidia_models.py`: NVIDIA model fetching
+- `scripts/fetch_nvidia_models.py`: Bayesian ranking script
+
+### Breaking Changes
+- Unpinned pydantic (was ==2.11.7, now >=2.11.7)
+- Removed deprecated test files
+
+### Documentation
+- `docs/INSTALL.md`: Installation guide
+- `requirements-lite.txt`: Minimal REPL-only install

docs/INSTALL.md

@@ -0,0 +1,153 @@
+# Installation Guide
+
+## Installation Modes
+
+### Full Installation (Default)
+
+```bash
+pip install -e .
+```
+
+**Includes everything:**
+- Core LLM providers: Anthropic, OpenAI, LiteLLM, NVIDIA
+- REPL mode + Server mode
+- File processing tools
+- MCP support
+- Search/Web tools
+- **GCP:** Vertex AI, Cloud Storage, Google APIs
+- **Server mode:** FastAPI, WebSocket, Redis, Postgres, SQLite
+- **Sandboxing:** E2B, Playwright
+- **Auth:** OAuth, JWT, Stripe
+- **Extras:** APScheduler, transformers, pytest
+
+**Size:** ~2GB | **Install time:** 3-5 minutes (GCP dependency resolution)
+
+### Lite Installation (REPL only, no GCP)
+
+```bash
+pip install -r requirements-lite.txt
+pip install -e . --no-deps
+```
+
+**Includes:**
+- Core LLM providers: Anthropic, OpenAI, LiteLLM
+- REPL mode only
+- File processing tools
+- MCP support
+- Search/Web tools
+- **No GCP, no server, no databases**
+
+**Size:** ~200MB | **Install time:** 30 seconds
+
+## Quick Start
+
+### REPL Mode (Lite)
+```bash
+# Install lite
+pip install -r requirements-lite.txt
+pip install -e . --no-deps
+
+# Set API key
+export NVIDIA_API_KEY=nvapi-...
+
+# Run REPL
+ii-agent --repl
+```
+
+### REPL Mode (Full)
+```bash
+# Install full
+pip install -e .
+
+# Set API key
+export NVIDIA_API_KEY=nvapi-...
+
+# Run REPL
+ii-agent --repl
+```
+
+### Server Mode
+```bash
+# Install full
+pip install -e .
+
+# Set up database
+# Configure .env
+
+# Run server
+ii-agent
+# OR
+ii-agent --port 8080
+```
+
+## Python 3.14 Compatibility
+
+If using Python 3.14, set this environment variable before installing:
+
+```bash
+export PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
+pip install -e .
+```
+
+## Dependencies
+
+### Lite Install (requirements-lite.txt)
+```
+Core: anthropic, litellm, openai, pydantic
+MCP/Tools: fastmcp, libtmux
+REPL: prompt-toolkit, rich
+File: pandas, pymupdf, python-pptx, weasyprint, mammoth
+Search: ddgs, duckduckgo-search, tavily-python
+Utils: python-dotenv, tenacity, ast-grep-cli
+```
+**Total:** ~45 packages, ~200MB
+
+### Full Install (pip install -e .) - Adds:
+```
+Server: alembic, aiosqlite, asyncpg, psycopg2-binary, redis
+        fastapi, fastapi-sso, python-socketio, starlette, uvicorn
+
+GCP: anthropic[vertex], gcloud-aio-storage
+     google-api-python-client, google-auth-oauthlib
+     google-cloud-aiplatform, google-genai
+
+Sandboxing: e2b-code-interpreter, playwright
+
+Auth: bcrypt, cryptography, email-validator, PyJWT, stripe
+
+Extras: apscheduler, ii-researcher, pydub, pytest
+        python-crontab, speechrecognition, transformers
+```
+**Total:** ~95 packages, ~2GB
+
+## Known Issues
+
+**GCP packages may cause slow dependency resolution:**
+- `google-api-python-client` + `google-cloud-aiplatform` + `anthropic[vertex]` have complex dependency trees
+- Pip resolver may take 3-5 minutes
+- Total install size: ~2GB with all dependencies
+
+## Troubleshooting
+
+### Dependency resolver hangs
+```bash
+# Be patient - GCP packages take 3-5 minutes to resolve
+# Or use --no-deps if you know dependencies are satisfied
+pip install -e .
+```
+
+### Python 3.14 build errors
+```bash
+export PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
+pip install -e .
+```
+
+## Development
+
+```bash
+# Full install (default)
+pip install -e .
+
+# GAIA benchmark tasks (adds datasets, huggingface-hub)
+pip install -e .[gaia]
+```

docs/docs/required-environment-variables/core-infra.md

@@ -5,9 +5,9 @@ slug: /required-environment-variables/core-infra
 sidebar_position: 18
 ---
 
-These variables cover databases, caches, and host port mappings used by the Docker stack. Most of them can stay at their defaults unless you have port conflicts or custom credentials.
+These variables cover databases, caches, and host port mappings. For local development without Docker, SQLite is used by default with no configuration required. For Docker deployments, configure PostgreSQL as described below.
 
-## Postgres (`POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`, `POSTGRES_PORT`)
+## Postgres (Docker Only) (`POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`, `POSTGRES_PORT`)
 
 1. Pick credentials you are comfortable using for local development. Example:
    ```bash
@@ -21,8 +21,15 @@ These variables cover databases, caches, and host port mappings used by the Dock
 
 ## `DATABASE_URL`
 
-- Async connection string used by the backend (e.g., `postgresql+asyncpg://app:changeme@postgres:5432/ii`).
-- Ensure the host matches the service name inside Docker (`postgres` by default) instead of `localhost`.
+**Default (Local Development):** SQLite at `~/.ii_agent/ii_agent.sqlite` - no setup required.
+
+**Docker/Production (PostgreSQL):**
+- Async connection string: `postgresql+asyncpg://app:changeme@postgres:5432/ii`
+- Ensure the host matches the service name inside Docker (`postgres` by default) instead of `localhost`
+
+**Other Options:**
+- DuckDB: `duckdb:///~/.ii_agent/ii_agent.duckdb`
+- In-memory SQLite (testing): `sqlite+aiosqlite:///:memory:`
 
 ## Sandbox database (`SANDBOX_DB_NAME`, `SANDBOX_DATABASE_URL`)
 

pyproject.toml

@@ -6,68 +6,92 @@ readme = "README.md"
 authors = [{ name = "Intelligent Internet", email = "[email protected]" }]
 requires-python = ">=3.10"
 dependencies = [
+  # Core agent
+  "anthropic>=0.72.0",
+  "litellm>=1.63.14",
+  "openai>=1.68.2",
+  "pydantic>=2.11.7",
+
+  # MCP/Tools
+  "fastmcp==2.10.6",
+  "libtmux==0.46.2",
+
+  # REPL
+  "prompt-toolkit>=3.0.51",
+  "rich==14.1.0",
+
+  # File processing
   "dataclasses-json>=0.6.7",
-  "duckduckgo-search>=8.0.1",
-  "fastapi>=0.115.12",
-  "google-cloud-aiplatform>=1.90.0",
-  "google-genai>=1.14.0",
-  "ii-researcher>=0.1.5",
   "jsonschema>=4.23.0",
+  "langchain-text-splitters>=1.0.0",
   "mammoth>=1.9.0",
   "markdownify>=1.1.0",
   "pandas>=2.2.3",
   "pathvalidate>=3.2.3",
   "pdfminer-six>=20250506",
-  "openai>=1.68.2",
-  "pexpect>=4.9.0",
   "pillow>=11.2.1",
-  "pip>=25.1.1",
-  "playwright==1.55.0",
-  "prompt-toolkit>=3.0.51",
   "puremagic>=1.29",
-  "pydub>=0.25.1",
   "pymupdf>=1.25.5",
-  "pytest>=8.3.5",
-  "python-dotenv>=1.1.0",
+  "pypdf>=6.1.1",
+  "python-magic>=0.4.27",
   "python-pptx>=1.0.2",
-  "rich==14.1.0",
-  "speechrecognition>=3.14.2",
+  "weasyprint>=66.0",
+
+  # Search/Web
+  "ddgs>=9.9.1",
+  "duckduckgo-search>=8.0.1",
   "tavily-python>=0.7.2",
+  "youtube-transcript-api>=1.0.3",
+
+  # Utils
+  "pexpect>=4.9.0",
+  "pip>=25.1.1",
+  "python-dotenv>=1.1.0",
   "tenacity>=9.1.2",
   "termcolor>=3.0.1",
-  "uvicorn[standard]>=0.29.0,<0.30.0",
-  "youtube-transcript-api>=1.0.3",
+
+  # AST/Code tools
+  "ast-grep-cli>=0.39.5",
+
+  # Server mode
   "alembic>=1.16.1",
-  "fastmcp==2.10.6",
-  "libtmux==0.46.2",
-  "bcrypt>=4.0.0",
-  "PyJWT>=2.8.0",
-  "email-validator>=2.0.0",
-  "cryptography>=42.0.0",
-  "fastapi-sso>=0.16.0",
   "aiosqlite>=0.21.0",
-  "pydantic==2.11.7",
-  "e2b-code-interpreter==1.2.0b5",
-  "python-socketio>=5.13.0",
-  "gcloud-aio-storage==9.5.0",
-  "transformers>=4.44.0",
-  "weasyprint>=66.0",
-  "python-magic>=0.4.27",
   "asyncpg>=0.30.0",
-  "stripe>=11.1.0",
-  "python-crontab>=2.7.1",
   "psycopg2-binary>=2.9.10",
-  "ast-grep-cli>=0.39.5",
-  "starlette[full]>=0.46.2",
   "redis>=5.0.0",
-  "pypdf>=6.1.1",
-  "apscheduler>=3.11.0",
-  "litellm>=1.63.14",
+  "fastapi>=0.115.12",
+  "fastapi-sso>=0.16.0",
+  "python-socketio>=5.13.0",
+  "starlette[full]>=0.46.2",
+  "uvicorn[standard]>=0.29.0,<0.30.0",
+
+  # Auth/Security
+  "bcrypt>=4.0.0",
+  "cryptography>=42.0.0",
+  "email-validator>=2.0.0",
+  "PyJWT>=2.8.0",
+  "stripe>=11.1.0",
+
+  # GCP
   "anthropic[vertex]>=0.72.0",
-  "langchain-text-splitters>=1.0.0",
-  "google-auth-oauthlib>=1.2.3",
+  "gcloud-aio-storage==9.5.0",
   "google-api-python-client>=2.150.0",
-  "ddgs>=9.9.1",
+  "google-auth-oauthlib>=1.2.3",
+  "google-cloud-aiplatform>=1.90.0",
+  "google-genai>=1.14.0",
+
+  # Sandboxing
+  "e2b-code-interpreter==1.2.0b5",
+  "playwright==1.55.0",
+
+  # Extra tools
+  "apscheduler>=3.11.0",
+  "ii-researcher>=0.1.5",
+  "pydub>=0.25.1",
+  "pytest>=8.3.5",
+  "python-crontab>=2.7.1",
+  "speechrecognition>=3.14.2",
+  "transformers>=4.44.0",
 ]
 
 [project.optional-dependencies]

requirements-lite.txt

@@ -0,0 +1,49 @@
+# ii-agent Lite - REPL only, no GCP/server/sandboxing
+# Install: pip install -r requirements-lite.txt
+
+# Core agent
+anthropic>=0.72.0
+litellm>=1.63.14
+openai>=1.68.2
+pydantic>=2.11.7
+
+# MCP/Tools
+fastmcp==2.10.6
+libtmux==0.46.2
+
+# REPL
+prompt-toolkit>=3.0.51
+rich==14.1.0
+
+# File processing
+dataclasses-json>=0.6.7
+jsonschema>=4.23.0
+langchain-text-splitters>=1.0.0
+mammoth>=1.9.0
+markdownify>=1.1.0
+pandas>=2.2.3
+pathvalidate>=3.2.3
+pdfminer-six>=20250506
+pillow>=11.2.1
+puremagic>=1.29
+pymupdf>=1.25.5
+pypdf>=6.1.1
+python-magic>=0.4.27
+python-pptx>=1.0.2
+weasyprint>=66.0
+
+# Search/Web
+ddgs>=9.9.1
+duckduckgo-search>=8.0.1
+tavily-python>=0.7.2
+youtube-transcript-api>=1.0.3
+
+# Utils
+pexpect>=4.9.0
+pip>=25.1.1
+python-dotenv>=1.1.0
+tenacity>=9.1.2
+termcolor>=3.0.1
+
+# AST/Code tools
+ast-grep-cli>=0.39.5