Merge pull request #218 from chrishayuk/streaming_refactor

Streaming refactor

Author: chrishayuk

README.md

@@ -7,9 +7,29 @@ A powerful, feature-rich command-line interface for interacting with Model Conte
 
 **Default Configuration**: MCP CLI defaults to using Ollama with the `gpt-oss` reasoning model for local, privacy-focused operation without requiring API keys.
 
-## 🆕 Recent Updates (v0.12.0)
-
-### Performance & Polish (Tier 3)
+## 🆕 Recent Updates (v0.14.0)
+
+### Production Hardening (Tier 5)
+- **Secret Redaction**: All log output (console and file) is automatically redacted for Bearer tokens, API keys, OAuth tokens, and Authorization headers
+- **Structured File Logging**: Optional `--log-file` flag enables rotating JSON log files (10MB, 3 backups) at DEBUG level with secret redaction
+- **Per-Server Timeouts**: Server configs now support `tool_timeout` and `init_timeout` overrides, resolved per-server → global → default
+- **Thread-Safe OAuth**: Concurrent OAuth flows are serialized with `asyncio.Lock` and copy-on-write header mutation
+
+### Code Quality (Tier 4)
+- **Core/UI Separation**: Core modules (`chat/conversation.py`, `chat/tool_processor.py`, `chat/chat_context.py`) no longer import `chuk_term.ui.output` — all logging goes through `logging` module
+- **Message Class Clarity**: Local `Message` renamed to `HistoryMessage` (backward-compat alias preserved) to distinguish from `chuk_llm.core.models.Message`
+- **Removed Global Singletons**: `_GLOBAL_TOOL_MANAGER` and associated getter/setter functions deleted
+- **Integration Test Framework**: Real MCP server tests with `@pytest.mark.integration` marker (SQLite server)
+- **Coverage Reporting**: Branch coverage enabled with `fail_under = 60` threshold in pyproject.toml
+
+### Previous: MCP Apps (SEP-1865)
+- **Interactive HTML UIs**: MCP servers can now serve interactive HTML applications (charts, tables, maps, markdown viewers) that render in your browser
+- **Sandboxed iframes**: Apps run in secure sandboxed iframes with CSP protection
+- **WebSocket bridge**: Real-time bidirectional communication between browser apps and MCP servers
+- **Automatic launch**: Tools with `_meta.ui` annotations automatically open in the browser when called
+- **Session reliability**: Message queuing, reconnection with exponential backoff, deferred tool result delivery
+
+### Previous: Performance & Polish (Tier 3)
 - **O(1) Tool Lookups**: Indexed tool lookup replacing O(n) linear scans in both ToolManager and ChatContext
 - **Cached LLM Tool Metadata**: Per-provider caching of tool definitions with automatic invalidation
 - **Startup Progress**: Real-time progress messages during initialization instead of a single spinner
@@ -18,21 +38,6 @@ A powerful, feature-rich command-line interface for interacting with Model Conte
 - **Conversation Export**: Export conversations as Markdown or JSON with metadata (`/export`)
 - **Trusted Domains**: Tools from trusted server domains (e.g. chukai.io) skip confirmation prompts
 
-### Architecture & Performance
-- **Updated to chuk-llm v0.16+**: Dynamic model discovery with capability-based selection, llama.cpp integration (1.53x faster), 52x faster imports
-- **Updated to chuk-tool-processor v0.13+**: Now using CTP's production-grade middleware (retry, circuit breaker, rate limiting)
-- **Slimmed ToolManager**: Reduced from 2000+ lines to ~800 lines by delegating to StreamManager while keeping OAuth, filtering, and LLM adaptation
-
-### Reliability Improvements
-- **Transport Failure Detection**: Automatic tracking of consecutive transport failures with warnings and recovery suggestions
-- **Enhanced Tool Processing**: Improved MCP SDK ToolResult handling with proper content extraction from nested structures
-- **Connection Monitoring**: Built-in health checks with automatic detection of unhealthy connections
-
-### Bug Fixes
-- **Fixed cmd mode**: `--provider` and `--model` flags now work correctly in command mode (PR #188)
-- **OAuth Improvements**: Enhanced OAuth token handling and storage
-- **Pydantic Migration**: Clean migration to Pydantic for better validation and type safety
-
 ## 🔄 Architecture Overview
 
 The MCP CLI is built on a modular architecture with clean separation of concerns:
@@ -88,6 +93,14 @@ MCP CLI supports all providers and models from CHUK-LLM, including cutting-edge
 - **Middleware**: Retry with exponential backoff, circuit breakers, and rate limiting via CTP
 - **Streaming Tool Calls**: Support for tools that return streaming data
 
+### MCP Apps (Interactive UIs)
+- **Browser-based UIs**: MCP servers can serve interactive HTML applications that render in your browser
+- **Automatic Detection**: Tools with `_meta.ui` annotations automatically launch browser apps on tool call
+- **Sandboxed Execution**: Apps run in secure sandboxed iframes with Content Security Policy protection
+- **WebSocket Bridge**: Real-time JSON-RPC bridge between browser apps and MCP tool servers
+- **Session Persistence**: Message queuing during disconnects, automatic reconnection, deferred tool result delivery
+- **structuredContent Support**: Full MCP spec compliance including structured content extraction and forwarding
+
 ### Advanced Configuration Management
 - **Environment Integration**: API keys and settings via environment variables
 - **File-based Config**: YAML and JSON configuration files
@@ -112,6 +125,7 @@ Comprehensive documentation is available in the `docs/` directory:
 - **[Token Management](docs/TOKEN_MANAGEMENT.md)** - Comprehensive token management for providers and servers including OAuth, bearer tokens, and API keys
 
 ### Specialized Documentation
+- **[MCP Apps](docs/MCP_APPS.md)** - Interactive browser UIs served by MCP servers (SEP-1865)
 - **[OAuth Authentication](docs/OAUTH.md)** - OAuth flows, storage backends, and MCP server integration
 - **[Streaming Integration](docs/STREAMING.md)** - Real-time response streaming architecture
 - **[Package Management](docs/PACKAGE_MANAGEMENT.md)** - Dependency organization and feature groups
@@ -167,6 +181,9 @@ git clone https://github.com/chrishayuk/mcp-cli
 cd mcp-cli
 pip install -e "."
 mcp-cli --help
+
+# Optional: Enable MCP Apps (interactive browser UIs)
+pip install -e ".[apps]"
 ```
 
 ### Using Different Models
@@ -233,6 +250,7 @@ Global options available for all modes and commands:
 - `--token-backend`: Override token storage backend (`auto`, `keychain`, `windows`, `secretservice`, `encrypted`, `vault`)
 - `--verbose`: Enable detailed logging
 - `--quiet`: Suppress non-essential output
+- `--log-file`: Write debug logs to a rotating file (secrets auto-redacted)
 
 ### Environment Variables
 
@@ -343,6 +361,47 @@ mcp-cli token backends            # Show available storage backends
 mcp-cli --token-backend encrypted token list  # Use specific backend
 ```
 
+## 🌐 MCP Apps (Interactive Browser UIs)
+
+MCP Apps allow tool servers to provide interactive HTML UIs that render in your browser. When a tool has a `_meta.ui` annotation pointing to a UI resource, mcp-cli automatically launches a local web server and opens the app in your browser.
+
+### Prerequisites
+
+```bash
+# Install the apps extra (adds websockets dependency)
+pip install "mcp-cli[apps]"
+```
+
+### How It Works
+
+1. Connect to an MCP server that provides app-enabled tools
+2. Call a tool that has `_meta.ui` metadata (e.g., `show_chart`, `show_table`)
+3. mcp-cli automatically fetches the UI resource, starts a local server, and opens your browser
+4. The app receives tool results in real-time via WebSocket
+
+### Example
+
+```bash
+# Connect to a server with app-enabled tools
+mcp-cli --server view_demo
+
+# In chat, ask for something visual:
+> Show me the sales data as a chart
+# Browser opens automatically with an interactive chart
+
+# The /tools command shows which tools have app UIs (APP column)
+> /tools
+```
+
+### Architecture
+
+- **Host page** serves a sandboxed iframe with the app HTML
+- **WebSocket bridge** proxies JSON-RPC between the browser and MCP servers
+- **Security**: Iframe sandbox, CSP protection, XSS prevention, URL scheme validation
+- **Reliability**: Message queuing during disconnects, exponential backoff reconnection, deferred tool result delivery
+
+See [MCP Apps Documentation](docs/MCP_APPS.md) for the full guide.
+
 ## 🤖 Using Chat Mode
 
 Chat mode provides the most advanced interface with streaming responses and intelligent tool usage.
@@ -1138,6 +1197,9 @@ Enable verbose logging for troubleshooting:
 ```bash
 mcp-cli --verbose chat --server sqlite
 mcp-cli --log-level DEBUG interactive --server sqlite
+
+# Write debug logs to a rotating file (secrets are automatically redacted)
+mcp-cli --log-file ~/.mcp-cli/logs/debug.log --server sqlite
 ```
 
 ## 🔒 Security Considerations
@@ -1152,6 +1214,10 @@ mcp-cli --log-level DEBUG interactive --server sqlite
 - **API Keys**: Only needed for cloud providers (OpenAI, Anthropic, etc.), stored securely using token management system
 - **OAuth 2.0 Support**: Secure authentication for MCP servers using PKCE and resource indicators (RFC 7636, RFC 8707)
 
+### Log Security
+- **Secret Redaction**: All log output (console and file) is automatically redacted for Bearer tokens, API keys (sk-*), OAuth access tokens, and Authorization headers
+- **Rotating File Logs**: Optional `--log-file` with JSON format, 10MB rotation, and 3 backup files
+
 ### Execution Security
 - **Tool Validation**: All tool calls are validated before execution
 - **Timeout Protection**: Configurable timeouts prevent hanging operations (v0.13+)
@@ -1160,6 +1226,13 @@ mcp-cli --log-level DEBUG interactive --server sqlite
 - **File Access**: Filesystem access can be disabled with `--disable-filesystem`
 - **Transport Monitoring**: Automatic detection of connection failures with warnings (v0.11+)
 
+### MCP Apps Security
+- **Iframe Sandbox**: Apps run in sandboxed iframes with restricted permissions
+- **Content Security Policy**: Server-supplied CSP domains are validated and sanitized
+- **XSS Prevention**: Tool names and user-supplied content are HTML-escaped before template injection
+- **URL Scheme Validation**: `ui/open-link` only allows `http://` and `https://` schemes
+- **Tool Name Validation**: Bridge rejects tool names not matching the MCP spec character set
+
 ## 🚀 Performance Features
 
 ### LLM Provider Performance (v0.16+)
@@ -1196,6 +1269,7 @@ Install with specific features:
 ```bash
 pip install "mcp-cli[cli]"        # Basic CLI features
 pip install "mcp-cli[cli,dev]"    # CLI with development tools
+pip install "mcp-cli[apps]"       # MCP Apps (interactive browser UIs)
 ```
 
 ## 🤝 Contributing

architecture.md

@@ -112,12 +112,93 @@ Every user-facing feature must have a working example in the `examples/` directo
 
 ---
 
-## Known Violations (Tier 4 Backlog)
+## MCP Apps (SEP-1865)
 
-Architecture review performed after Tier 2. These are tracked for remediation in Tier 4 (Code Quality).
+MCP Apps are interactive HTML UIs served by MCP servers and rendered in the user's browser via sandboxed iframes. When a tool has a `_meta.ui` annotation, mcp-cli launches a local web server that bridges the browser and the MCP backend.
+
+### Architecture
+
+```
+Browser                    Python Backend                MCP Server
+┌─────────────────┐       ┌──────────────────┐       ┌──────────────┐
+│  Host Page (JS)  │──WS──│  AppBridge        │──MCP──│  Tool Server │
+│  ┌─────────────┐ │      │  (bridge.py)      │       │              │
+│  │ App iframe  │ │      └──────────────────┘       └──────────────┘
+│  │ (sandboxed) │ │              │
+│  └─────────────┘ │      ┌──────────────────┐
+│   postMessage ↕  │      │  AppHostServer   │
+└─────────────────┘       │  (host.py)        │
+                          └──────────────────┘
+```
+
+- **`host.py`** — `AppHostServer` manages lifecycle: port allocation, HTTP serving (host page + app HTML), WebSocket server, browser launch
+- **`host_page.py`** — JavaScript host page template; bridges iframe postMessage ↔ WebSocket, handles `ui/initialize`, display modes, reconnection
+- **`bridge.py`** — `AppBridge` handles JSON-RPC protocol: proxies `tools/call` and `resources/read` to MCP servers, manages message queue for disconnected WS, formats tool results per MCP spec
+- **`models.py`** — Pydantic models: `AppInfo`, `AppState` (PENDING → INITIALIZING → READY → CLOSED), `HostContext`
+
+### Security Model
+
+- **Iframe sandbox:** `allow-scripts allow-forms allow-same-origin allow-popups allow-popups-to-escape-sandbox`
+- **XSS prevention:** Tool names are `html.escape()`d before template injection
+- **CSP domain sanitization:** Server-supplied domains validated against `^[a-zA-Z0-9\-.:/*]+$`
+- **Tool name validation:** Bridge rejects tool names not matching `^[a-zA-Z0-9_\-./]+$`
+- **URL scheme validation:** `ui/open-link` only allows `http://` and `https://` schemes
+- **Safe JSON serialization:** `_safe_json_dumps()` with `_to_serializable()` fallback; circular reference protection
+
+### Session Reliability
+
+- **Message queue:** `_pending_notifications: deque[str]` (maxlen=50) queues notifications when WS is disconnected
+- **Drain on reconnect:** `drain_pending()` flushes queued messages when WS reconnects
+- **State reset:** `set_ws()` resets state to INITIALIZING, closes old WS
+- **Reconnect notification:** Host page sends `ui/notifications/reconnected` to app iframe on WS reconnect
+- **Exponential backoff:** WS reconnection uses 1s→30s exponential backoff with reset on success
+- **Initialization timeout:** Configurable JS timeout (default 30s) shows "initialization timed out" if app never initializes
+- **Deferred tool result delivery:** Initial tool results are stored on the bridge and pushed only after the app sends `ui/notifications/initialized`, preventing race conditions where postMessage is dropped before the app sets up its listener
+- **Duplicate prevention:** `launch_app()` closes previous instance before launching new one
+- **Push to existing:** `tool_processor.py` pushes new tool results to running apps instead of re-launching
+
+### Spec Compliance
+
+- `ui/initialize` response includes protocol version, host capabilities (with sandbox details), host info, host context
+- `ui/resource-teardown` sent to iframe on `beforeunload`
+- `ui/notifications/host-context-changed` sent after display mode changes
+- `structuredContent` recovered from JSON text blocks when transport loses it (CTP normalization)
+
+---
+
+## Two Message Classes
+
+The codebase has two classes that represent messages, serving different purposes:
+
+- **`chuk_llm.core.models.Message`** (re-exported via `chat/response_models.py`) — canonical LLM message with typed `ToolCall` objects. Used by `tool_processor.py` and `conversation.py`.
+- **`mcp_cli.chat.models.HistoryMessage`** (aliased as `Message` for backward compat) — SessionManager-compatible message with `tool_calls: list[dict]`. Used by `chat_context.py`.
+
+The roundtrip: chuk_llm Message → `to_dict()` → SessionEvent → `from_dict()` → HistoryMessage → `to_dict()` → API.
+
+## Secret Redaction
+
+`SecretRedactingFilter` in `config/logging.py` is always active on all log handlers (console and file). It redacts:
+
+- Bearer tokens (`Authorization: Bearer eyJ...`)
+- API keys (`sk-proj-...`, `sk-...`)
+- Generic `api_key=...` / `api-key: ...` values
+- OAuth access tokens in JSON (`"access_token": "..."`)
+- Authorization headers (`Authorization: Basic ...`)
+
+The filter is a module-level singleton (`secret_filter`) that can be added to custom handlers.
+
+---
+
+## Known Violations (Remaining)
+
+Architecture review performed after Tier 2. Tier 4 (Code Quality) resolved the most impactful issues. Remaining items are tracked here.
 
 ### Core/UI Separation (#5)
 
+**Resolved in Tier 4.3:** `chat/conversation.py`, `chat/tool_processor.py`, and `chat/chat_context.py` no longer import `chuk_term.ui.output`. All core logging goes through the `logging` module.
+
+**Remaining:**
+
 | File | Issue | Severity |
 |------|-------|----------|
 | `chat/ui_manager.py` | Imports `prompt_toolkit`, `display/`, `commands/` | HIGH — move to `interactive/` |
@@ -130,15 +211,17 @@ Architecture review performed after Tier 2. These are tracked for remediation in
 | File | Issue | Severity |
 |------|-------|----------|
 | `chat/chat_context.py` | `openai_tools: list[dict]` instead of typed model | MEDIUM |
-| `chat/models.py` | `Message.tool_calls: list[dict]` instead of `list[ToolCallData]` | MEDIUM |
+| `chat/models.py` | `HistoryMessage.tool_calls: list[dict]` instead of `list[ToolCallData]` | MEDIUM — by design for SessionManager compat |
 | `chat/conversation.py` | `_validate_tool_messages()` works on raw dicts | MEDIUM — by design at serialization boundary |
 
 ### Explicit Dependencies (#7)
 
+**Resolved in Tier 4.1:** `_GLOBAL_TOOL_MANAGER` singleton removed. ToolManager is constructor-injected everywhere.
+
+**Remaining (deferred — low impact):**
+
 | File | Issue | Severity |
 |------|-------|----------|
-| `chat/tool_processor.py` | Uses `get_tool_state()`, `get_search_engine()` globals | MEDIUM |
-| `chat/conversation.py` | Uses `get_tool_state()` global | MEDIUM |
-| `chat/tool_processor.py` | Uses `get_preference_manager()` global | LOW |
-
-These are deferred to **Tier 4.1 (Replace Global Singletons)** and **Tier 4.2 (Consolidate Message Classes)**.
+| `chat/tool_processor.py` | Uses `get_tool_state()`, `get_search_engine()` globals | MEDIUM — external library singletons |
+| `chat/conversation.py` | Uses `get_tool_state()` global | MEDIUM — external library singleton |
+| `chat/tool_processor.py` | Uses `get_preference_manager()` global | LOW — 15 call sites, marginal payoff |