New docs incoming

Author: riofriz

.coderabbit/CODERABBIT.md

@@ -0,0 +1,58 @@
+# CodeRabbit Code Review Context for Degoog
+
+Degoog is a Bun and Hono TypeScript search aggregator created by fccview. It combines server-side search orchestration, a browser UI, extension registries, themes, admin settings, a plugin store, optional Valkey caching, Docker deployment, and tests. 
+
+As an AI code reviewer, use the following guidelines to evaluate PRs, generate feedback, and flag regressions.
+
+## Key Review Directives
+
+- **Enforce Standards:** Verify that PRs align with `STANDARDS.md` (located in the root folder). Flag any violations.
+- **Protect App Behavior:** Flag any changes that alter existing app behavior unless the PR explicitly notes that the behavior change is intended and approved.
+- **Protect Public Contracts:** Ensure public API routes, response shapes, settings keys, environment variables, plugin APIs, extension IDs, theme behavior, and UI expectations remain strictly intact.
+- **Scope Checking:** Call out massive rewrites or scope creep. Suggest breaking large, unstructured PRs into smaller, reviewable changes.
+- **Test Coverage:** Verify that focused tests have been added or updated for the behavior touched in the PR. Flag missing test coverage for new logic.
+- **Reject Style Churn:** Flag and discourage style-only code churn across unrelated files.
+- **CSS/SCSS Enforcement:** Check that UI styling is done in modularised `.scss` files. Flag any modifications to standard `.css` files unless the changes are strictly within a `./store` folder.
+- **Class Naming:** Ensure new UI elements follow the existing styling patterns and the `degoog-*` class name convention.
+
+## Architecture Context (For validating file placement)
+
+When reviewing, ensure changes are logically placed according to this architecture:
+- **Boot/Entry:** Server entrypoint and app boot live around `src/server/index.ts`.
+- **Routing:** Hono routes live under `src/server/routes/`.
+- **Search Logic:** Core search lives around `src/server/search.ts`. Route-specific handlers are under `src/server/routes/search/` and streaming search is in `src/server/routes/search-stream.ts`.
+- **Extensions:** Registries live under `src/server/extensions/` (sharing behavior via `src/server/extensions/registry-factory.ts`). Store install, update, uninstall, and repo handling live under `src/server/extensions/store/`.
+- **Utilities:** Server settings, plugin settings, cache, rate limiting, proxy handling, auth helpers, and path helpers live under `src/server/utils/`.
+- **Frontend:** Client UI code lives under `src/client/`. Public templates and theme files live under `src/public/`.
+- **Testing:** Tests live under `tests/`.
+
+## Important Project Values
+
+- **Strict UI Consistency:** The Front End tech lead is highly particular about UI consistency. Reject PRs that introduce borders, blur, or transparency. Enforce existing styling paradigms strictly.
+- **No Developer Comments:** Flag and request the removal of any inline code comments added in the PR. Only the human lead developer is permitted to manually add comments.
+- **Code Quality:** Prioritize readability and maintainability over clever, overly terse code in your review suggestions.
+- **Backward Compatibility:** Existing users must not be forced to change configuration, URLs, plugins, themes, or workflows due to a PR. Flag any breaking migration impacts immediately.
+- **Security vs. Compatibility:** Security fixes are welcome, but scrutinize them heavily for compatibility and migration impact. 
+- **Extension Trust Model:** Treat installed plugins, themes, engines, and transports as a trusted extension system unless the PR explicitly introduces a stricter user-requested trust model.
+
+## What to Look For During Refactors / Cleanups
+
+Praise or suggest the following improvements during PR reviews:
+- **Deduplication:** Look for opportunities to reduce duplicated logic between streaming and non-streaming search, provided the response formats do not change.
+- **Cache Integrity:** Verify that cache keys remain complete and behavior-specific.
+- **Deterministic IDs:** Ensure extension IDs and settings IDs remain deterministic and backward compatible.
+- **Reliable Registries:** Check that registry loading remains deterministic, paying special attention to duplicate triggers, duplicate names, and skip behavior.
+- **Data Safety:** Verify that file writes remain atomic for persistent JSON settings or store metadata.
+- **Route Consistency:** Ensure route JSON parsing, auth checks, and rate limiting remain consistent across endpoints.
+- **Path Safety:** Scrutinize path handling for plugin, theme, proxy, and store assets to prevent directory traversal or unsafe access.
+- **Modularization:** Encourage developers moving large modules toward smaller, responsibility-focused modules.
+
+## PR Rejection Criteria (What to Flag Immediately)
+
+Leave blocking review comments if a PR attempts to do any of the following:
+- Rewrite the app or replace core architectural decisions.
+- Replace Bun or Hono.
+- Rename public routes or settings without a clear, approved compatibility strategy.
+- Break plugin, theme, engine, transport, or store compatibility.
+- Redesign the UI as a byproduct of a cleanup/refactor.
+- Change production defaults without explicit approval documented in the PR.

.coderabbit/STANDARDS.md

@@ -0,0 +1,123 @@
+# CodeRabbit PR Review Standards for Degoog
+
+Degoog is a Bun + Hono TypeScript search aggregator. As an AI code reviewer, use these standards to evaluate pull requests, ensuring the project remains maintainable without forcing rewrites or breaking public behavior. 
+
+Apply these standards strictly to new code. For existing code, suggest improvements only when the module is already being touched for a feature, bug fix, or security update.
+
+## 1. Core Principles Review
+
+### Directives
+- **Protect Contracts:** Flag any unannounced changes to existing APIs, UI contracts, config names, environment variables, plugin/theme/engine interfaces, store layouts, and route behavior. Require a documented migration path or `@deprecated` shim.
+- **Scope Control:** Reject massive, style-only rewrites. Praise and encourage small, behavior-preserving changes.
+- **Test Enforcement:** Block refactors of routes, search orchestration, persistence, extension loading, security gates, or user settings if they lack tests covering observable behavior.
+- **Readability:** Push back on overly clever abstractions. Code must be readable so future contributors can trace route/search/registry behavior easily.
+- **Trust Boundaries:** Treat installed extensions/themes as trusted, but rigorously verify that PRs treat their inputs, paths, URLs, rendered HTML, and persisted metadata as untrusted.
+
+## 2. TypeScript Style & Module Boundaries
+
+### What to Look For
+- **Strict DTOs:** Verify strict types for data crossing boundaries (client/server, route/orchestration, registry/extensions). Ensure shared shapes use `src/shared` to avoid client/server drift.
+- **Return Types:** Flag missing explicit return types on exported functions, route helpers, registry helpers, persistence functions, and security-sensitive utilities.
+- **Typing Rules:** Enforce `unknown` at external boundaries (followed by validation/narrowing) over `any`.
+- **Naming Conventions:**
+  - Types/Interfaces: `PascalCase`.
+  - Constants: `UPPER_SNAKE_CASE`.
+  - Internal Helpers: Leading underscore `_` only if file-private and matching local convention.
+  - Verbs: `parse*`, `is*`/`assert*`, `to*`/`from*`, `load*`/`write*`.
+- **Function Size:** Suggest splitting functions that exceed ~60 lines or try to handle parsing, validation, persistence, rendering, and logging all at once.
+- **Correct Placement:** Ensure changes respect module boundaries (e.g., HTTP concerns in `routes/`, shared logic in `utils/search.ts`, UI orchestration in `client/modules/`).
+
+## 3. Hono Route Standards
+
+### Route PR Checklist
+- **Guard Placement:** Verify that rate limiting and auth guards (`guardApiKey`, settings guards) are placed at the *top* of the route handler, before expensive operations.
+- **JSON Parsing:** Flag repeated `try/catch` blocks for body parsing; suggest extracting or using existing JSON parser helpers.
+- **Error Envelopes:** Ensure JSON routes return consistent `{ error: string }` envelopes and appropriate status codes. Do not allow plain text errors unless the route is a binary/text proxy with an established contract.
+- **Separation of Concerns:** Route handlers must focus on HTTP. Suggest moving store mutations, search orchestration, and persistence into helper functions.
+- **Default Preservation:** Scrutinize parser refactors to ensure default values (search type, page, lang, streaming toggles) remain intact.
+
+## 4. Extension Registry Standards
+
+### Registry PR Checklist
+- **Determinism:** Verify that directory reads/entries are explicitly sorted to guarantee stable load order across restarts.
+- **ID Stability:** Enforce canonical ID structures `<folder>-<kind>` (`-engine`, `-slot`, `-command`, `-tab`, etc.). Reject renames of built-in IDs or settings IDs without a valid migration.
+- **Duplicate Handling:** Ensure duplicate extension IDs are handled gracefully (logged with context) and do not silently merge unrelated settings.
+- **Lifecycle Semantics:** `match() === null` should not log as an error. `onLoad` failures should log context without leaking secrets and safely skip the extension.
+- **Immutability:** Ensure callers do not mutate registry-owned arrays (`items()`).
+
+## 5. Store & Installation Standards
+
+### Installation PR Checklist
+- **Transparency:** Reject PRs that silently run package manager commands or hide dependency installations.
+- **Path Containment:** Scrutinize repository operations. Verify URL scheme validation, git error sanitization, timeouts, and containment checks (reject `..`, absolute child paths, symlink escapes). Never trust repo-provided filenames for writes.
+- **Atomic Writes:** Ensure persistence updates for store metadata are atomic (e.g., temp-file creation followed by rename).
+- **Concurrency:** Look for locks on store writes, settings writes, and install/uninstall operations to prevent race conditions.
+- **ID Preservation:** Ensure installed item IDs and `installedAs` names are preserved across updates unless explicitly changed by the user.
+
+## 6. Search Orchestration Standards
+
+### Search PR Checklist
+- **Streaming Parity:** Enforce identical orchestration paths between `/stream` and non-streaming search. Query parsing, engine selection, interceptors, scoring, and cache writes must not be duplicated or drifted.
+- **Cache Integrity:** Verify that cache keys include *all* inputs (query, overrides, engine config, page, time, lang, image filters).
+- **Interceptor Overrides:** Ensure `searchType`, `lang`, and `timeFilter` overrides from interceptors are correctly applied *before* cache key construction and engine selection.
+- **Timeouts/Signals:** Verify that engine fetches receive `AbortSignal` and that streaming stops when the client disconnects.
+- **Engine Type Model (CRITICAL):**
+  - Reject restrictive unions for `EngineSearchType` (it must remain `string`).
+  - Ensure type arrays (`["web", "karakeep"]`) are supported.
+  - Verify `resolveTypes` in `engines/registry.ts` is the single source of truth for type resolution.
+  - Verify `selectActiveEngines` uses unified paths (`getActiveWebEngines` vs `getEnginesForCustomType`). Ensure `includeCustom` is not reintroduced.
+
+## 7. Client UI Standards
+
+### Frontend PR Checklist
+- **Layer Separation:** Suggest splitting UI functions that mix parsing, fetching, state updates, and DOM rendering.
+- **Selector Stability:** Reject changes to stable DOM IDs, classes (`degoog-*`), and `data-*` attributes. These are public APIs for themes, plugins, and browser extensions.
+- **Event Handlers:** Praise/suggest event delegation (`data-action`) over rebinding handlers on every render.
+- **DOM Safety:** Strongly flag the use of `innerHTML` unless handling explicitly trusted templates or sanitized HTML. Recommend `textContent` for general data.
+- **Accessibility:** Ensure interactive elements are semantic (`<button>`, `<a>`), have `aria-label`s if icon-only, preserve keyboard navigation, and handle loading states visibly.
+
+## 8. Security Standards
+
+### Security PR Checklist
+- **SSRF Prevention:** Ensure proxied/fetched URLs strictly allow `http:` and `https:`, re-check protocols after redirects, and use signed proxy URLs for exposed assets.
+- **Path Verification:** Assert that all extension/store paths are resolved and checked for containment before reads/writes.
+- **Secret Hygiene:** Flag PRs that log settings/admin/search API tokens or nonces. Ensure secret settings are masked in UI/metadata responses.
+- **Header Trust:** Do not allow trust of `X-Forwarded-*` headers unless explicit proxy trust settings are enabled.
+- **Error Safety:** Ensure error responses do not leak local paths, tokens, repo internals, or stack traces.
+
+## 9. Persistence & Cache Standards
+
+### Persistence PR Checklist
+- **JSON Schema:** Ensure JSON persistence logic tolerates missing fields, preserves unknown fields, and recovers safely.
+- **Atomicity:** Flag direct overwrites of critical JSON files. Require atomic write patterns (write to temp file -> fsync -> rename).
+- **Caching:** Ensure new cache APIs use async `useCache`. Verify cache invalidation clears both local memory and Valkey state. Ensure TTLs rely on safe defaults/env vars.
+
+## 10. Logging & Observability
+
+### Logging PR Checklist
+- **Console Usage:** Reject raw `console.*` in server code (except for startup scripts). Enforce the central `logger` utility.
+- **Namespaces:** Ensure logs use feature namespaces (e.g., `search`, `store:repo`, `settings`).
+- **Telemetry Value:** Ensure logs contain meaningful metrics (query lengths, result counts, timings) and *never* log sensitive payloads, passwords, or tokens.
+- **Structured Formats:** Encourage `key=value` paired strings for easier scanning.
+
+## 11. Testing Standards
+
+### Test PR Checklist
+- **Coverage:** Reject bug fix PRs that lack regression tests (if testable). Demand tests for route shapes, auth guards, cache keys, and store safety.
+- **Isolation:** Verify tests isolate runtime data using env vars/data paths.
+- **Mocks:** Ensure network/git mocks are used sparingly and assert the critical commands/options.
+- **Determinism:** Flag flaky tests. Inputs must be sorted, time controlled, and external search dependencies mocked or removed.
+
+## 12. Duplication Control
+
+### Refactoring PR Checklist
+- **Rule of Two:** Do not praise generic abstractions created for a single call site. Require at least two real use cases before extracting shared helpers.
+- **Focus:** Prefer small, narrowly-named helpers over dumping unrelated functions into large utility files.
+
+## 13. Final Approval Gate (Rule of Thumb)
+
+Before approving a PR, verify:
+1. Does it preserve user-facing behavior? (Unless explicitly marked as a breaking change).
+2. Are compatibility risks for extensions/plugins/themes considered?
+3. Are secrets, paths, and HTML boundaries safely handled?
+4. Is the PR small enough to review confidently? (If not, suggest splitting it up).

.github/workflows/pages.yml

@@ -0,0 +1,59 @@
+name: Deploy docs to GitHub Pages
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+
+      - name: Build the docs
+        run: npm run build
+
+      - name: Assemble the published site
+        run: |
+          mkdir -p _site
+          rsync -a \
+            --exclude='.git' \
+            --exclude='.github' \
+            --exclude='.gitignore' \
+            --exclude='_site' \
+            --exclude='src' \
+            --exclude='scripts' \
+            --exclude='assets' \
+            --exclude='node_modules' \
+            --exclude='package.json' \
+            --exclude='package-lock.json' \
+            --exclude='README.md' \
+            ./ _site/
+
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: _site
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@v4

.gitignore

@@ -0,0 +1,13 @@
+node_modules/
+/*.html
+/style.css
+/docs.js
+/docs-ui.js
+/search-index.json
+/search-index.js
+/images/
+/fontawesome/
+/placeholders/
+/_site/
+/dist/
+.DS_Store

README.md

@@ -0,0 +1,24 @@
+# Official degoog docs
+
+Official docs for the search aggregator [degoog](https://github.io/degoog-org/degoog)
+
+Learn about how everything works here: https://degoog-org.github.io/docs/
+
+## Run
+
+```sh
+npm run build
+npm run serve
+```
+
+The server defaults to `http://127.0.0.1:4173`. Set `PORT=4174 npm run serve` to use another port.
+
+## Editing pages
+
+Add or edit HTML fragments under `src/pages/user/` or `src/pages/developer/`. Keep page chrome, search, sidebar, scripts, and theme controls out of those fragments; the build script adds the shared shell.
+
+When adding a page, add one entry to `src/data/pages.mjs`. The generated cross-document search index uses the combined list, while each generated page renders only the sidebar section for its active docs mode.
+
+Never edit generated root pages by hand. Update `src/pages/`, `src/data/pages.mjs`, or `assets/`, then run `npm run build`.
+
+The build also removes known stale legacy root pages from the previous static docs layout so old generated HTML does not survive a root-served replacement.

assets/docs-ui.js

@@ -0,0 +1,48 @@
+document.addEventListener("DOMContentLoaded", function () {
+  var page = window.location.pathname.split("/").pop() || "index.html";
+  if (!page || page === "") page = "index.html";
+  document.querySelectorAll(".degoog-docs-nav-item").forEach(function (link) {
+    if (link.getAttribute("href") === page) {
+      link.classList.add("degoog-docs-nav-active");
+    }
+  });
+
+  var burger = document.getElementById("degoog-docs-burger");
+  var sidebar = document.querySelector(".degoog-docs-sidebar");
+  var backdrop = document.getElementById("degoog-docs-backdrop");
+
+  if (burger && sidebar && backdrop) {
+    burger.addEventListener("click", function () {
+      var isOpen = sidebar.classList.toggle("degoog-docs-sidebar-open");
+      backdrop.style.display = isOpen ? "block" : "none";
+    });
+
+    backdrop.addEventListener("click", function () {
+      sidebar.classList.remove("degoog-docs-sidebar-open");
+      backdrop.style.display = "none";
+    });
+  }
+
+  var themeBtn = document.getElementById("doc-theme-toggle");
+  if (themeBtn) {
+    themeBtn.addEventListener("click", function () {
+      var current = document.documentElement.getAttribute("data-theme");
+      var next = current === "dark" ? "light" : "dark";
+      document.documentElement.setAttribute("data-theme", next);
+      try {
+        localStorage.setItem("ade:theme", next);
+      } catch (e) {}
+    });
+  }
+
+  var modeSwitch = document.querySelector(".doc-mode-switch");
+  if (modeSwitch) {
+    modeSwitch.addEventListener("click", function (event) {
+      var link = event.target.closest("[data-doc-mode]");
+      if (!link) return;
+      try {
+        localStorage.setItem("degoog:docs-mode", link.getAttribute("data-doc-mode"));
+      } catch (e) {}
+    });
+  }
+});