feat: integrate Readability.js and refine content extraction strategy

Author: miziodel

README.md

@@ -15,7 +15,6 @@ AI Sidekick is a minimalist, "Arc-style" Chrome Extension that brings multi-LLM
 - **Secure Vault**: Client-side encryption (PBKDF2 + AES-GCM) for API keys.
   - **Session Persistence**: Keys stay unlocked for the browser session.
   - **Auto-Lock**: Automatically locks after 15 minutes of inactivity.
-- **Web Mode**: Fallback to `gemini.google.com` (free) if no API key is available.
 - **Contextual Actions**:
   - Right-click to Explain, Summarize, or Analyze pages.
   - **Summarize Button**: One-click summary of the current conversation.
@@ -85,14 +84,21 @@ node tests/run_tests.js
 
 You can customize the prompts used for context menu actions and page analysis in the **Options** page (`Right-click extension icon -> Options`).
 
+#### Automatic Content Extraction
+
+AI Sidekick uses **Mozilla Readability.js** to automatically extract the main content of the active page. This includes:
+- **Link Preservation**: Links are preserved in the text as `Text [URL]`.
+- **Whitespace Cleaning**: Noise and excessive whitespace are removed for a cleaner prompt.
+- **Smart Logic**: Boilerplate (menus, ads) is automatically filtered out.
+
 #### Available Variables
 
 Use these placeholders to insert dynamic content into your prompts:
 
 | Variable        | Description                                                        | Context             |
 | :-------------- | :----------------------------------------------------------------- | :------------------ |
 | `{{selection}}` | The text currently selected by the user.                           | Select Menu Actions |
-| `{{content}}`   | The full text content of the active page (truncated if necessary). | Analyze Page        |
+| `{{content}}`   | The full text content of the active page (including links).        | Analyze Page        |
 | `{{url}}`       | The URL of the active page.                                        | All Actions         |
 | `{{title}}`     | The title of the active page.                                      | All Actions         |
 

THIRD-PARTY-NOTICES.txt

@@ -26,3 +26,22 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
 ================================================================================
+
+MOZILLA READABILITY (Readability.js)
+Source: https://github.com/mozilla/readability
+License: Apache License 2.0
+Copyright (c) 2010 Arc90 Inc
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+================================================================================

docs/changelog.md

@@ -5,22 +5,29 @@ All notable changes to this project will be documented in this file.
 ## [1.0.2] - 2026-01-18
 
 ### Added
+- **Smart Content Extraction**: Integrated Mozilla's Readability.js library for production-grade article extraction (same algorithm as Firefox Reader View).
+- **Path Audit Integration**: `tests/audit_paths.js` now runs during `npm test` to verify asset integrity.
 - **Versioning Protocol**: Established strict version sync rules in `.agent/rules/02-versioning.md`.
 
 ### Changed
 - **Documentation**: Removed "draft" warnings from README; project promoted to Beta status.
 - **Audit**: Validated repository consistency and cleanup.
+- **Refined Content Extraction**: 
+    - Always-on strategy (removed Auto/URL-only toggle for simplicity).
+    - Preserves links in `Text [URL]` format using live DOM analysis.
+    - Improved whitespace cleaning and boilerplate removal.
+- Refactored `analyzeCurrentPage` in `sidepanel.js` to use Mozilla Readability.js.
+- **Static Analysis**: Added `tests/audit_paths.js` to prevent broken links during refactors.
+- **Side Panel Path**: Corrected `openExtension` logic to point to `src/sidepanel.html` after migration.
+- **Debug Logging**: Added stylized console logs in the Side Panel to monitor full chat exchanges (prompts, history context, responses).
+- **Multi-Model Support**: Integrated Gemini 1.5 Flash/Pro and DeepSeek V3/R1.
+- **Client-Side Encryption**: Implemented `CryptoUtils` class for PBKDF2 + AES-GCM local key storage.
 
 ## [1.0.1] - 2026-01-18
 
 ### Added
 - **Antigravity Structure**: Established `.agent/rules` and strict project identity.
 - **Source Reorganization**: Moved all extension code to `src/` for cleaner root.
-- **Static Analysis**: Added `tests/audit_paths.js` to prevent broken links during refactors.
-- **Side Panel Path**: Corrected `openExtension` logic to point to `src/sidepanel.html` after migration.
-- **Debug Logging**: Added stylized console logs in the Side Panel to monitor full chat exchanges (prompts, history context, responses).
-- **Multi-Model Support**: Integrated Gemini 1.5 Flash/Pro and DeepSeek V3/R1.
-- **Client-Side Encryption**: Implemented `CryptoUtils` class for PBKDF2 + AES-GCM local key storage.
 - **Arc Browser Support**:
     - **[Implemented]** **Single Instance Window**: Added logic to `background.js` using `chrome.storage.session` to track and focus the existing extension window, preventing duplicates.
     - **[Added]** **UI Context Bar**: Visual indicator of the active page URL being analyzed in the Side Panel/Popup.

docs/context.md

@@ -17,6 +17,7 @@
 - Implemented `chrome.storage.session` for resilient window tracking in Arc.
 - Added UI Context Bar and "Open Sidekick Here" menu action.
 - Integrated ESLint into the build pipeline.
+- **Integrated Mozilla Readability.js for production-grade content extraction**.
 
 ## Active Questions
 - (Empty)

docs/drafts/ui-id-collision-fix.md

@@ -0,0 +1,41 @@
+# Draft: Fix Chat UI ID Collision
+
+## Problem
+In `src/sidepanel.js`, the `addMessage` function generates message IDs using `Date.now()`:
+```javascript
+function addMessage(role, text, isRaw = false) {
+  const div = document.createElement('div');
+  div.className = `message ${role}`;
+  div.id = 'msg-' + Date.now(); // <-- RISK OF COLLISION
+  // ...
+}
+```
+When a User message is added and an AI message (placeholder) follows immediately, they may receive the same ID. `updateMessage` then finds the first element with that ID (the User bubble) and overwrites its content with the AI response.
+
+## Proposed Strategy
+Replace `Date.now()` with a more robust unique identifier.
+
+### Option A: Timestamp + Metadata
+```javascript
+div.id = `msg-${Date.now()}-${role}-${Math.floor(Math.random() * 1000)}`;
+```
+
+### Option B: Global Counter (Cleaner)
+Add a counter to the global state:
+```javascript
+const state = {
+  // ...
+  msgCounter: 0
+};
+
+function addMessage(role, text, isRaw = false) {
+  state.msgCounter++;
+  const div.id = `msg-${Date.now()}-${state.msgCounter}`;
+  // ...
+}
+```
+
+## Implementation Note
+- Apply changes to `addMessage` in `src/sidepanel.js`.
+- No changes needed to `updateMessage` or `scrollToBottom`.
+- Ensure `loadSettings` (rendering history) also uses unique IDs if it calls `addMessage`.

docs/drafts/web-mode-implementation.md

@@ -0,0 +1,235 @@
+# Web Mode Implementation - Technical Design
+
+> **Status**: 🚧 Draft - Not Yet Implemented  
+> **Priority**: Medium  
+> **Related**: [Roadmap](../roadmap.md) - "Web Mode Fallback (BROKEN)"
+
+## Problem Statement
+
+The current "Gemini Web (Free)" option ([`sidepanel.js:746`](file:///Users/NOBKP/ai-sidekick/src/sidepanel.js#L746)) simply opens `gemini.google.com` in a new tab, providing no integration with the extension. Users without API keys have no fallback for in-panel chat.
+
+## Goals
+
+1. **Zero-Friction Fallback**: Users without API keys can still use AI assistants directly from the side panel.
+2. **Multi-Provider Support**: Extend to Gemini, DeepSeek, ChatGPT, and Claude.
+3. **Seamless UX**: Minimize context-switching and manual copy-paste.
+
+---
+
+## Technical Approaches
+
+### Option A: Iframe Embedding ⚠️ Limited Viability
+
+**Concept**: Embed web chat interfaces directly in the side panel using `<iframe>`.
+
+**Pros**:
+- User stays in side panel.
+- No tab switching.
+
+**Cons**:
+- **Security Restrictions**: Most AI providers block `<iframe>` embedding via `X-Frame-Options: DENY` or CSP headers.
+  - ✅ **Gemini**: May work (needs testing).
+  - ❌ **ChatGPT**: Blocked.
+  - ❌ **Claude**: Blocked.
+  - ⚠️ **DeepSeek**: Unknown (needs testing).
+
+**Verdict**: Only viable for Gemini. Not a universal solution.
+
+---
+
+### Option B: Managed Tab with Context Injection
+
+**Concept**: Open web interface in a new tab, but inject user's prompt automatically.
+
+**Implementation**:
+1. User selects "Gemini Web (Free)" and types a message.
+2. Extension opens `https://gemini.google.com/app` in a new tab.
+3. Use `chrome.scripting.executeScript` to:
+   - Wait for page load.
+   - Find chat input field.
+   - Insert user's prompt.
+   - Optionally trigger "Send" button.
+
+**Pros**:
+- Works with all providers (no iframe restrictions).
+- User sees response in native interface.
+
+**Cons**:
+- **Login Required**: User must be logged into each service.
+- **DOM Fragility**: Injection relies on stable DOM selectors (e.g., `textarea[aria-label="Chat input"]`). Providers can break this at any time.
+- **Privacy Concerns**: Users may not want extension manipulating web pages.
+
+**Verdict**: Technically feasible but fragile and requires maintenance.
+
+---
+
+### Option C: Hybrid Approach (Recommended)
+
+**Concept**: Show an "onboarding dialog" when user selects Web Mode without API key.
+
+**UX Flow**:
+```
+User selects "Gemini Web (Free)" → No API key detected
+
+┌─────────────────────────────────────────────┐
+│ ⚠️ No API Key Found                         │
+│                                             │
+│ To use AI Sidekick, you need:              │
+│                                             │
+│ [Option 1] Enter your API keys (Recommended)│
+│   → Private, encrypted, full control       │
+│   → Get keys: [Gemini Key] [DeepSeek Key]  │
+│                                             │
+│ [Option 2] Use free web version            │
+│   → Opens gemini.google.com in new tab     │
+│   → Requires Google account login          │
+│                                             │
+│ [ Set Up API Keys ]  [ Continue to Web ]   │
+└─────────────────────────────────────────────┘
+```
+
+If user clicks **"Continue to Web"**:
+- Open `gemini.google.com/app` in new tab.
+- Copy user's prompt to clipboard automatically.
+- Show toast: "Prompt copied! Paste it in Gemini to continue."
+
+**Pros**:
+- ✅ **Clear UX**: Sets expectations (no magic integration).
+- ✅ **No Fragility**: No reliance on DOM selectors.
+- ✅ **Privacy-Friendly**: Extension doesn't manipulate web pages.
+- ✅ **Extensible**: Easy to add ChatGPT, Claude, DeepSeek with same pattern.
+
+**Cons**:
+- User must manually paste (1 extra step).
+
+**Verdict**: Best balance of simplicity, reliability, and user experience.
+
+---
+
+## Recommended Implementation (Phase 1)
+
+### 1. Update Model Selection Logic
+
+**File**: `src/sidepanel.js`
+
+**Current Code** (Line 746):
+```javascript
+if (model === 'gemini-web') {
+  window.open('https://gemini.google.com/app', '_blank');
+  return;
+}
+```
+
+**New Code**:
+```javascript
+if (model === 'gemini-web') {
+  showWebModeDialog('gemini', userMessage);
+  return;
+}
+```
+
+### 2. Create Web Mode Dialog
+
+**File**: `src/lib/web-mode-dialog.js` (new)
+
+```javascript
+function showWebModeDialog(provider, userPrompt) {
+  const providerConfig = {
+    gemini: {
+      name: 'Gemini',
+      url: 'https://gemini.google.com/app',
+      keyUrl: 'https://aistudio.google.com/app/apikey'
+    },
+    deepseek: {
+      name: 'DeepSeek',
+      url: 'https://chat.deepseek.com',
+      keyUrl: 'https://platform.deepseek.com/api_keys'
+    },
+    chatgpt: {
+      name: 'ChatGPT',
+      url: 'https://chatgpt.com',
+      keyUrl: null // No API key option
+    },
+    claude: {
+      name: 'Claude',
+      url: 'https://claude.ai',
+      keyUrl: 'https://console.anthropic.com/'
+    }
+  };
+
+  const config = providerConfig[provider];
+  
+  // Show dialog with options:
+  // 1. "Set Up API Keys" → chrome.runtime.openOptionsPage()
+  // 2. "Continue to Web" → copyToClipboard(userPrompt) + open(config.url)
+}
+```
+
+### 3. Add Clipboard Permission
+
+**File**: `manifest.json`
+
+```json
+"permissions": [
+  "clipboardWrite",  // Already exists ✅
+  // ...
+]
+```
+
+---
+
+## Future Enhancements (Phase 2+)
+
+### Multi-Provider Dropdown
+
+Instead of just "🌐 Gemini Web (Free)", show:
+```html
+<optgroup label="🌐 Web Mode (Free, No Keys)">
+  <option value="gemini-web">Gemini (Google Account)</option>
+  <option value="deepseek-web">DeepSeek (Free Account)</option>
+  <option value="chatgpt-web">ChatGPT (OpenAI Account)</option>
+  <option value="claude-web">Claude (Anthropic Account)</option>
+</optgroup>
+```
+
+### Browser Action Integration
+
+Add a "Quick Switch" button in the side panel:
+```
+[API Mode: Gemini Flash ▼]
+  → Gemini 2.5 Pro (API)
+  → DeepSeek R1 (API)
+  ───────────────
+  → Gemini Web (Free) 🌐
+  → ChatGPT Web (Free) 🌐
+```
+
+---
+
+## Testing Checklist
+
+- [ ] Verify dialog appears when Web Mode selected without API key.
+- [ ] Confirm clipboard copy works (test on Mac/Windows/Linux).
+- [ ] Ensure correct URL opens for each provider.
+- [ ] Test "Set Up API Keys" button redirects to Options.
+- [ ] Validate toast notification shows after clipboard copy.
+
+---
+
+## Open Questions
+
+1. **Should we support iframe for Gemini if it works?**  
+   → Decision: Test first. If unreliable, use unified "open tab + copy" approach.
+
+2. **Should we inject prompt automatically using content scripts?**  
+   → Decision: No. Too fragile. Manual paste is acceptable trade-off.
+
+3. **Should we track which web interface the user prefers?**  
+   → Decision: Phase 2. Store in `chrome.storage.local`.
+
+---
+
+## References
+
+- Current Implementation: [`src/sidepanel.js:746`](file:///Users/NOBKP/ai-sidekick/src/sidepanel.js#L746)
+- Roadmap Task: [`docs/roadmap.md`](file:///Users/NOBKP/ai-sidekick/docs/roadmap.md)