From 86ece0a02feddf3d4582da4fc9791e08731adc2b Mon Sep 17 00:00:00 2001 From: toughhou Date: Sat, 2 May 2026 01:20:33 -0700 Subject: [PATCH 1/6] =?UTF-8?q?feat(skill):=20=E6=8F=90=E5=8F=96=E5=90=8E?= =?UTF-8?q?=E7=AB=AF=E9=80=BB=E8=BE=91=E4=B8=BA=20Hermes=20Skill=EF=BC=88G?= =?UTF-8?q?UI=E2=86=92CLI=20=E9=80=82=E9=85=8D=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 新增 SKILL.md（技能定义，10个命令：init/ingest/search/graph/insights/deep-research 等） - 新增 HERMES.md（Hermes 入口骨架） - 新增 install.sh（支持 --platform hermes/claude） - 新增 skill/src/fs-node.ts（Tauri IPC → Node.js fs 完整替换层） - 新增 skill/src/stores-node.ts（React stores → 模块状态替换） - 新增 skill/src/cli.ts（CLI 入口，graph/insights/search/status 命令当前可用） - 新增 skill/package.json + tsconfig.skill.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- HERMES.md | 40 +++++ SKILL.md | 293 ++++++++++++++++++++++++++++++++++++ install.sh | 63 ++++++++ skill/package.json | 20 +++ skill/src/cli.ts | 305 ++++++++++++++++++++++++++++++++++++++ skill/src/fs-node.ts | 156 +++++++++++++++++++ skill/src/stores-node.ts | 160 ++++++++++++++++++++ skill/tsconfig.skill.json | 28 ++++ 8 files changed, 1065 insertions(+) create mode 100644 HERMES.md create mode 100644 SKILL.md create mode 100644 install.sh create mode 100644 skill/package.json create mode 100644 skill/src/cli.ts create mode 100644 skill/src/fs-node.ts create mode 100644 skill/src/stores-node.ts create mode 100644 skill/tsconfig.skill.json diff --git a/HERMES.md b/HERMES.md new file mode 100644 index 00000000..714d30f8 --- /dev/null +++ b/HERMES.md @@ -0,0 +1,40 @@ +# llm-wiki-nashsu — Hermes Skill 入口 + +> **适配状态**：⚠️ 部分适配（需完成 GUI→CLI 工程改造后方可完整使用） +> **当前可用**：graph、insights、search 命令（无需 LLM） +> **待完成**：ingest、deep-research 命令（需替换 Tauri IPC → Node.js fs） + +## 触发条件 + +加载本技能当用户明确提到： +- "图谱分析"、"知识图谱"、"图谱洞察" +- "深度研究" +- "知识缺口"、"惊人连接" + +## 与 llm-wiki-skill 的关系 + +本技能**补充** llm-wiki-skill，提供更高级的图谱分析能力： +- llm-wiki-skill：负责日常 ingest、Hermes 调度、中文内容源 +- llm-wiki-nashsu：负责高级图谱分析、深度研究 + +## 主要工作流 + +详见 `SKILL.md`。 + +## 安装路径 + +```bash +# Hermes 安装 +bash install.sh --platform hermes + +# 直接使用 +node skill/cli.js graph +node skill/cli.js insights +node skill/cli.js search +``` + +## 注意事项 + +- Node.js >= 20 运行时必须可用 +- 中文素材源（微信/知乎/小红书）请使用 llm-wiki-skill +- ingest 功能目前需要完成 Tauri IPC 替换工程（约 10-13 人日） diff --git a/SKILL.md b/SKILL.md new file mode 100644 index 00000000..cc27b77f --- /dev/null +++ b/SKILL.md @@ -0,0 +1,293 @@ +--- +name: llm-wiki-nashsu +version: 0.4.6-skill +author: nashsu (GUI→Skill 适配: bid-sys team) +license: MIT +description: | + 基于 nashsu/llm_wiki 后端逻辑提取的知识库技能（无 GUI）。 + 核心算法包括：4 信号图谱相关度模型、Louvain 社区检测、图谱洞察（惊人连接+知识缺口）、 + RRF 混合搜索（BM25+向量）、深度研究（网络搜索+自动消化）、异步审核队列。 + 触发条件：用户明确提到知识库、wiki、图谱分析、深度研究，或要求对已初始化的知识库执行 + 消化、搜索、健康检查等操作。 +metadata: + hermes: + tags: + - knowledge-base + - wiki + - graph-analysis + - deep-research + - semantic-search + origin: nashsu/llm_wiki (GUI stripped, backend extracted) + runtime: node >= 20 + adapted_from: https://github.com/nashsu/llm_wiki +--- + +# llm-wiki-nashsu — 高级知识库后端技能 + +> 从 nashsu/llm_wiki 提取的后端逻辑，去除 Tauri GUI 后适配为 Hermes Skill。 +> 与 llm-wiki-skill 相比，本技能具有**显著更强的图谱分析能力**，但需要 Node.js 运行时。 + +## 核心差异化能力 + +| 能力 | 本技能 | llm-wiki-skill | +|------|-------|----------------| +| **图谱相关度** | 4 信号模型（直接链接×3 + 来源重叠×4 + Adamic-Adar×1.5 + 类型亲和×1）| 仅 wikilink，无权重 | +| **社区检测** | Louvain 算法 + 凝聚度评分 | 主题页→社区（启发式）| +| **图谱洞察** | 惊人连接 + 知识缺口 + 桥节点检测 | 无 | +| **搜索** | RRF 混合（BM25 + 向量） | Grep 关键词 | +| **深度研究** | 网络搜索→LLM 综合→自动消化 | 无 | +| **审核队列** | 异步异步 sweep-reviews 系统 | 无 | +| **图像处理** | 视觉 API 图像标注管线 | 无 | + +--- + +## Script Directory + +Scripts located in `skill/` subdirectory relative to this SKILL.md. + +**Path Resolution**: +1. `SKILL_DIR` = this SKILL.md's directory +2. Script path = `${SKILL_DIR}/skill/` + +--- + +## 依赖要求 + +``` +node >= 20 +npm >= 9 +``` + +**可选依赖（启用向量搜索）**： +- 配置 `EMBEDDING_API_BASE` 和 `EMBEDDING_MODEL` 环境变量（OpenAI 兼容端点） + +--- + +## 工作流命令 + +### 1. init — 初始化知识库 + +```bash +node ${SKILL_DIR}/skill/cli.js init [topic] [lang] +``` + +**参数**： +- `wiki_root`：wiki 工作目录（绝对路径） +- `topic`：知识库主题（可选，默认 "My Knowledge Base"） +- `lang`：语言（可选，`zh`|`en`，默认 `en`） + +**产物**： +``` +/ +├── wiki/ +│ ├── entities/ +│ ├── concepts/ +│ ├── sources/ +│ ├── queries/ +│ └── index.md +├── raw/ +└── .wiki-config.json +``` + +--- + +### 2. ingest — 消化素材 + +```bash +node ${SKILL_DIR}/skill/cli.js ingest [--llm-api-key=KEY] +``` + +**参数**： +- `wiki_root`：wiki 工作目录 +- `file_path`：待消化的文件路径（支持 .md / .txt；PDF/DOCX 需先转为文本） +- `--llm-api-key`：LLM API Key（也可通过 `OPENAI_API_KEY` 环境变量传入） + +**处理流程**（源自 `ingest.ts`）： +1. **Step 1**：LLM 分析素材 → 生成结构化 JSON（实体、概念、关系） +2. **Step 2**：基于 JSON 生成 wiki 页面： + - `wiki/sources/{slug}.md` — 素材摘要页（含 `sources: []` frontmatter） + - `wiki/entities/{name}.md` — 实体页（仅限新实体） + - `wiki/concepts/{name}.md` — 概念页（仅限新概念） +3. **自动消化**：生成的页面自动进入 wiki 图谱（下次 graph 命令时生效） +4. **审核标记**：LLM 自动标记需人工判断的条目（`review: true` frontmatter） + +**产物示例**： +```json +{ + "status": "success", + "pages": [ + "wiki/sources/2026-04-30-企业资质证书.md", + "wiki/entities/市政公用工程施工总承包壹级.md" + ], + "reviews_pending": 1 +} +``` + +--- + +### 3. batch-ingest — 批量消化 + +```bash +node ${SKILL_DIR}/skill/cli.js batch-ingest +``` + +按目录递归处理所有 `.md`/`.txt` 文件，保留目录结构作为分类上下文。 +失败不阻塞后续文件（标记失败项，继续）。 + +--- + +### 4. search — 智能搜索 + +```bash +node ${SKILL_DIR}/skill/cli.js search [--limit=20] +``` + +**算法**（源自 `search.ts`，18KB）： +1. **BM25 词法搜索**：中文 CJK bigram 分词 + 停用词过滤 + 精确词组匹配加权 +2. **向量语义搜索**（可选）：LanceDB ANN 检索（需配置嵌入端点） +3. **RRF 融合**：倒数秩融合（K=60），避免量纲不一致 + +**输出**：JSON 格式检索结果（path, title, snippet, score, images） + +--- + +### 5. graph — 构建知识图谱 + +```bash +node ${SKILL_DIR}/skill/cli.js graph [--output=graph-data.json] +``` + +**算法**（源自 `wiki-graph.ts` + `graph-relevance.ts`）： +1. **读取所有 wiki 页面**，提取标题、类型、wikilink +2. **4 信号相关度计算**（每条边）： + - 直接链接（weight 3.0） + - 来源重叠（weight 4.0，基于 `sources: []` frontmatter） + - Adamic-Adar 共同邻居（weight 1.5） + - 类型亲和度（weight 1.0） +3. **Louvain 社区检测**（graphology-communities-louvain）： + - 自动聚类，计算每个社区凝聚度（实际边/可能边） + - 低凝聚度社区（<0.15）标记为警告 +4. **输出**：`graph-data.json`（nodes + edges + communities） + +**输出格式**： +```json +{ + "nodes": [{ "id": "xxx", "label": "...", "type": "entity", "linkCount": 5, "community": 0 }], + "edges": [{ "source": "xxx", "target": "yyy", "weight": 7.2 }], + "communities": [{ "id": 0, "nodeCount": 12, "cohesion": 0.24, "topNodes": ["..."] }] +} +``` + +--- + +### 6. insights — 图谱洞察 + +```bash +node ${SKILL_DIR}/skill/cli.js insights +``` + +**算法**（源自 `graph-insights.ts`，193 行）： +1. **惊人连接**（Surprising Connections）： + - 跨社区边 +3，跨类型边 +2，边缘↔枢纽耦合 +2，弱连接 +1 + - 阈值 ≥3 才输出 +2. **知识缺口**（Knowledge Gaps）： + - 孤立节点（degree ≤1） + - 稀疏社区（cohesion <0.15 且 ≥3 节点） + - 桥节点（连接 ≥3 个社区） + +**输出**：Markdown 格式洞察报告 + +--- + +### 7. deep-research — 深度研究 + +```bash +node ${SKILL_DIR}/skill/cli.js deep-research [--queries="q1|q2|q3"] +``` + +**流程**（源自 `deep-research.ts`，244 行）： +1. **网络搜索**：多查询并行搜索（Tavily API），URL 去重合并 +2. **LLM 综合**：搜索结果 → wiki 页面（带 `[[wikilink]]` 交叉引用） +3. **保存**：`wiki/queries/research-{slug}-{date}.md` +4. **自动消化**：研究结果自动 ingest，提取实体/概念 + +**环境变量**：`TAVILY_API_KEY`（或 `SERPER_API_KEY`） + +--- + +### 8. lint — 健康检查 + +```bash +node ${SKILL_DIR}/skill/cli.js lint +``` + +**检查项**（源自 `lint.ts`）： +- 孤立页面（无入链且无出链） +- 断链（`[[wikilink]]` 指向不存在的页面） +- 过短页面（< 100 字） +- 语言不一致（frontmatter `lang` 与内容不符） +- 重复内容（相似度过高的页面） + +--- + +### 9. sweep-reviews — 处理审核队列 + +```bash +node ${SKILL_DIR}/skill/cli.js sweep-reviews +``` + +**功能**（源自 `sweep-reviews.ts`，14KB）： +- 扫描所有 `review: true` 的 wiki 页面 +- 基于规则匹配 + LLM 语义判断自动解决 +- 预定义动作：Create Page / Skip（防止 LLM 幻觉任意动作） + +--- + +### 10. status — 知识库状态 + +```bash +node ${SKILL_DIR}/skill/cli.js status +``` + +**输出**：JSON 格式统计（页面数、实体数、概念数、源数、待审核数） + +--- + +## 配置环境变量 + +| 变量 | 用途 | 示例 | +|------|------|------| +| `OPENAI_API_KEY` | LLM API Key（OpenAI/Anthropic 兼容）| `sk-...` | +| `OPENAI_API_BASE` | 自定义 LLM 端点（Ollama/代理）| `http://localhost:11434/v1` | +| `LLM_MODEL` | 模型名称 | `gpt-4o` / `claude-3-5-sonnet` | +| `EMBEDDING_API_BASE` | 嵌入端点（可选，启用向量搜索）| `http://localhost:11434/v1` | +| `EMBEDDING_MODEL` | 嵌入模型（可选）| `text-embedding-3-small` | +| `TAVILY_API_KEY` | 深度研究搜索 API（可选）| `tvly-...` | + +--- + +## 与 llm-wiki-skill 的关键互补 + +本技能建议**配合** llm-wiki-skill 使用而非替代： + +| 场景 | 推荐方案 | +|------|---------| +| 日常 ingest（速度优先）| llm-wiki-skill（Shell，零开销）| +| 图谱质量分析 | 本技能（graph + insights 命令）| +| 深度研究专项 | 本技能（deep-research 命令）| +| 中文内容源（微信/知乎/小红书）| llm-wiki-skill | +| Hermes Runtime 集成 | llm-wiki-skill（已有 HERMES.md）| +| 本技能 Hermes 集成 | 参见 HERMES.md（需手动适配）| + +--- + +## 安装 + +```bash +# 安装 CLI 依赖 +cd ${SKILL_DIR}/skill +npm install + +# 验证安装 +node cli.js --version +``` diff --git a/install.sh b/install.sh new file mode 100644 index 00000000..71d1d6f3 --- /dev/null +++ b/install.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# llm-wiki-nashsu install script +# Installs the nashsu backend skill (CLI-based, no GUI) + +set -euo pipefail + +SKILL_DIR="$(cd "$(dirname "$0")" && pwd)" +PLATFORM="${1:-}" + +install_skill_files() { + local target_dir="$1" + mkdir -p "$target_dir" + cp -r "$SKILL_DIR"/* "$target_dir/" + echo "✅ Skill files installed to $target_dir" +} + +install_npm_deps() { + local skill_dir="${1:-$SKILL_DIR}" + if [ -d "$skill_dir/skill" ]; then + echo "📦 Installing Node.js dependencies..." + cd "$skill_dir/skill" + npm install --quiet + echo "✅ Dependencies installed" + fi +} + +echo "🔧 llm-wiki-nashsu Skill Installer" +echo " Source: $SKILL_DIR" +echo "" + +case "$PLATFORM" in + --platform=hermes|--platform\ hermes) + HERMES_SKILLS="${HOME}/.hermes/skills" + TARGET="${HERMES_SKILLS}/llm-wiki-nashsu" + echo "🎯 Platform: Hermes" + install_skill_files "$TARGET" + install_npm_deps "$TARGET" + echo "" + echo "✅ Installed to: $TARGET" + echo " Usage: hermes run llm-wiki-nashsu graph " + ;; + + --platform=claude|--platform\ claude) + echo "🎯 Platform: Claude Code" + echo " Add to CLAUDE.md:" + echo " @${SKILL_DIR}/SKILL.md" + install_npm_deps + ;; + + "") + echo "📦 Local installation (no platform)" + install_npm_deps + echo "" + echo "✅ Ready. Usage:" + echo " node ${SKILL_DIR}/skill/src/cli.ts graph " + ;; + + *) + echo "⚠️ Unknown platform: $PLATFORM" + echo " Supported: --platform hermes, --platform claude" + exit 1 + ;; +esac diff --git a/skill/package.json b/skill/package.json new file mode 100644 index 00000000..411610b1 --- /dev/null +++ b/skill/package.json @@ -0,0 +1,20 @@ +{ + "name": "llm-wiki-nashsu-skill", + "version": "0.4.6-skill", + "description": "nashsu/llm_wiki backend extracted as Node.js CLI skill (no GUI)", + "main": "dist/cli.js", + "scripts": { + "build": "tsc -p tsconfig.skill.json", + "dev": "ts-node --project tsconfig.skill.json src/cli.ts", + "cli": "node dist/cli.js" + }, + "dependencies": { + "graphology": "^0.25.4", + "graphology-communities-louvain": "^2.0.0" + }, + "devDependencies": { + "@types/node": "^20.0.0", + "ts-node": "^10.9.0", + "typescript": "^5.0.0" + } +} diff --git a/skill/src/cli.ts b/skill/src/cli.ts new file mode 100644 index 00000000..b21d5084 --- /dev/null +++ b/skill/src/cli.ts @@ -0,0 +1,305 @@ +#!/usr/bin/env node +/** + * llm-wiki-nashsu CLI + * + * Entry point for the nashsu/llm_wiki backend skill (no GUI). + * Replaces Tauri IPC with Node.js fs, React stores with module state. + * + * Usage: + * node cli.js [args...] + * + * Commands: + * init [topic] [lang] + * graph [--output=graph-data.json] + * insights + * search [--limit=20] + * status + * lint (requires LLM) + * ingest (requires LLM) + * deep-research (requires LLM + search API) + * sweep-reviews (requires LLM) + */ + +import * as fs from "fs" +import * as path from "path" +// Import core library modules (Tauri deps patched via tsconfig paths) +import { buildWikiGraph } from "../../src/lib/wiki-graph" +import { findSurprisingConnections, detectKnowledgeGaps } from "../../src/lib/graph-insights" +import { searchWiki } from "../../src/lib/search" +import { configureWikiStore } from "./stores-node" + +// --------------------------------------------------------------------------- +// Main dispatch +// --------------------------------------------------------------------------- + +async function main() { + const [, , command, wikiRoot, ...rest] = process.argv + + if (!command || command === "--help" || command === "-h") { + printHelp() + process.exit(0) + } + + if (command === "--version") { + console.log("0.4.6-skill") + process.exit(0) + } + + if (!wikiRoot) { + console.error("Error: wiki_root is required") + process.exit(1) + } + + const resolvedRoot = path.resolve(wikiRoot) + + // Configure store state (replaces React store initialization) + configureWikiStore({ + projectPath: resolvedRoot, + llmConfig: { + provider: "openai", + apiKey: process.env.OPENAI_API_KEY ?? "", + model: process.env.LLM_MODEL ?? "gpt-4o", + baseUrl: process.env.OPENAI_API_BASE, + }, + embeddingConfig: { + enabled: !!(process.env.EMBEDDING_MODEL), + model: process.env.EMBEDDING_MODEL ?? "", + apiBase: process.env.EMBEDDING_API_BASE, + }, + }) + + switch (command) { + case "init": + await cmdInit(resolvedRoot, rest[0] ?? "My Knowledge Base", rest[1] ?? "en") + break + + case "graph": + await cmdGraph(resolvedRoot, rest) + break + + case "insights": + await cmdInsights(resolvedRoot) + break + + case "search": + if (!rest[0]) { console.error("Error: query is required"); process.exit(1) } + await cmdSearch(resolvedRoot, rest[0], rest) + break + + case "status": + await cmdStatus(resolvedRoot) + break + + default: + console.error(`Unknown command: ${command}`) + console.error("Run with --help for usage") + process.exit(1) + } +} + +// --------------------------------------------------------------------------- +// Commands +// --------------------------------------------------------------------------- + +async function cmdInit(wikiRoot: string, topic: string, lang: string) { + const dirs = ["wiki/entities", "wiki/concepts", "wiki/sources", "wiki/queries", "raw"] + for (const d of dirs) { + fs.mkdirSync(path.join(wikiRoot, d), { recursive: true }) + } + + const indexContent = `--- +type: overview +title: "${topic}" +lang: ${lang} +created: ${new Date().toISOString().slice(0, 10)} +--- + +# ${topic} + +> 这是一个 llm-wiki-nashsu 知识库。 + +## 实体 + +## 概念 + +## 素材来源 +` + const indexPath = path.join(wikiRoot, "wiki", "index.md") + if (!fs.existsSync(indexPath)) { + fs.writeFileSync(indexPath, indexContent, "utf-8") + } + + const configPath = path.join(wikiRoot, ".wiki-config.json") + if (!fs.existsSync(configPath)) { + fs.writeFileSync(configPath, JSON.stringify({ topic, lang, version: "0.4.6-skill" }, null, 2)) + } + + console.log(JSON.stringify({ status: "success", wiki_root: wikiRoot, topic, lang })) +} + +async function cmdGraph(wikiRoot: string, args: string[]) { + const outputArg = args.find((a) => a.startsWith("--output=")) + const outputPath = outputArg + ? outputArg.replace("--output=", "") + : path.join(wikiRoot, "graph-data.json") + + console.error(`[graph] Building wiki graph for: ${wikiRoot}`) + const { nodes, edges, communities } = await buildWikiGraph(wikiRoot) + console.error(`[graph] Found ${nodes.length} nodes, ${edges.length} edges, ${communities.length} communities`) + + const graphData = { + nodes, + edges, + communities, + generated: new Date().toISOString(), + } + + fs.mkdirSync(path.dirname(outputPath), { recursive: true }) + fs.writeFileSync(outputPath, JSON.stringify(graphData, null, 2)) + console.log(JSON.stringify({ status: "success", output: outputPath, ...graphData })) +} + +async function cmdInsights(wikiRoot: string) { + console.error(`[insights] Analyzing graph for: ${wikiRoot}`) + const { nodes, edges, communities } = await buildWikiGraph(wikiRoot) + + const surprising = findSurprisingConnections(nodes, edges, communities, 10) + const gaps = detectKnowledgeGaps(nodes, edges, communities, 10) + + // Format as markdown report + const lines: string[] = [ + "# 图谱洞察报告", + "", + `> 生成时间：${new Date().toISOString().slice(0, 16)}`, + `> 节点总数：${nodes.length}，边总数：${edges.length}，社区总数：${communities.length}`, + "", + "---", + "", + "## 惊人连接（Surprising Connections）", + "", + ] + + if (surprising.length === 0) { + lines.push("_暂无惊人连接。随着知识库增长，跨社区连接会在这里显示。_") + } else { + for (const conn of surprising) { + lines.push(`### ${conn.source.label} ↔ ${conn.target.label}`) + lines.push(`- **惊喜评分**：${conn.score}`) + lines.push(`- **原因**：${conn.reasons.join("；")}`) + lines.push("") + } + } + + lines.push("---", "", "## 知识缺口（Knowledge Gaps）", "") + + if (gaps.length === 0) { + lines.push("_暂无知识缺口。知识库连接良好！_") + } else { + for (const gap of gaps) { + const typeLabel = { + "isolated-node": "🔴 孤立节点", + "sparse-community": "🟡 稀疏社区", + "bridge-node": "🔵 桥节点", + }[gap.type] ?? gap.type + lines.push(`### ${typeLabel}：${gap.title}`) + lines.push(`- **描述**：${gap.description}`) + lines.push(`- **建议**：${gap.suggestion}`) + lines.push("") + } + } + + lines.push("---", "", "## 社区凝聚度", "") + for (const comm of communities) { + const warning = comm.cohesion < 0.15 ? " ⚠️ 低凝聚度" : "" + lines.push( + `- **社区 ${comm.id}**（${comm.nodeCount} 个页面）：` + + `凝聚度 ${comm.cohesion.toFixed(2)}${warning}` + + ` — ${comm.topNodes.slice(0, 3).join("、")}`, + ) + } + + const report = lines.join("\n") + console.log(report) +} + +async function cmdSearch(wikiRoot: string, query: string, args: string[]) { + const limitArg = args.find((a) => a.startsWith("--limit=")) + const _limit = limitArg ? parseInt(limitArg.replace("--limit=", ""), 10) : 20 + + console.error(`[search] Searching "${query}" in: ${wikiRoot}`) + const results = await searchWiki(wikiRoot, query) + + console.log(JSON.stringify({ query, results, total: results.length })) +} + +async function cmdStatus(wikiRoot: string) { + const wikiDir = path.join(wikiRoot, "wiki") + if (!fs.existsSync(wikiDir)) { + console.log(JSON.stringify({ status: "not_initialized", wiki_root: wikiRoot })) + return + } + + function countMd(dir: string): number { + if (!fs.existsSync(dir)) return 0 + return fs + .readdirSync(dir, { withFileTypes: true }) + .filter((e) => e.isFile() && e.name.endsWith(".md")) + .length + } + + const stats = { + status: "ready", + wiki_root: wikiRoot, + entities: countMd(path.join(wikiDir, "entities")), + concepts: countMd(path.join(wikiDir, "concepts")), + sources: countMd(path.join(wikiDir, "sources")), + queries: countMd(path.join(wikiDir, "queries")), + total: 0, + } + stats.total = stats.entities + stats.concepts + stats.sources + stats.queries + + console.log(JSON.stringify(stats)) +} + +// --------------------------------------------------------------------------- +// Help +// --------------------------------------------------------------------------- + +function printHelp() { + console.log(` +llm-wiki-nashsu v0.4.6-skill +nashsu/llm_wiki backend extracted as Node.js CLI skill (no GUI) + +USAGE: + node cli.js [options] + +COMMANDS: + init [topic] [lang] Initialize knowledge base + graph [--output=] Build graph data (JSON) + insights Graph insights (markdown) + search [--limit=N] Hybrid BM25+vector search + status Knowledge base statistics + + (Requires LLM config via env vars:) + ingest Ingest document + sweep-reviews Process review queue + deep-research Deep research via web search + +ENVIRONMENT VARIABLES: + OPENAI_API_KEY LLM API key + OPENAI_API_BASE Custom LLM endpoint (Ollama/proxy) + LLM_MODEL Model name (default: gpt-4o) + EMBEDDING_API_BASE Embedding endpoint (enables vector search) + EMBEDDING_MODEL Embedding model + TAVILY_API_KEY Web search API key (for deep-research) +`) +} + +// --------------------------------------------------------------------------- +// Run +// --------------------------------------------------------------------------- + +main().catch((err) => { + console.error("Error:", err.message) + process.exit(1) +}) diff --git a/skill/src/fs-node.ts b/skill/src/fs-node.ts new file mode 100644 index 00000000..0512ed89 --- /dev/null +++ b/skill/src/fs-node.ts @@ -0,0 +1,156 @@ +/** + * Node.js drop-in replacement for Tauri's @/commands/fs IPC layer. + * Maps all Tauri invoke() calls to standard Node.js fs operations. + * + * Original (Tauri): invoke("read_file", { path }) + * Replacement: fs.readFileSync(path, 'utf-8') + */ +import * as fs from "fs" +import * as path from "path" + +export interface FileNode { + name: string + path: string + is_dir: boolean + children?: FileNode[] +} + +export interface WikiProject { + id: string + name: string + path: string +} + +export interface FileBase64 { + base64: string + mimeType: string +} + +// --------------------------------------------------------------------------- +// Core file operations (replaces Tauri IPC) +// --------------------------------------------------------------------------- + +export async function readFile(filePath: string): Promise { + return fs.readFileSync(filePath, "utf-8") +} + +export async function writeFile(filePath: string, contents: string): Promise { + fs.mkdirSync(path.dirname(filePath), { recursive: true }) + fs.writeFileSync(filePath, contents, "utf-8") +} + +export async function listDirectory(dirPath: string): Promise { + if (!fs.existsSync(dirPath)) { + throw new Error(`Directory not found: ${dirPath}`) + } + return _listDirRecursive(dirPath) +} + +function _listDirRecursive(dirPath: string): FileNode[] { + const entries = fs.readdirSync(dirPath, { withFileTypes: true }) + return entries.map((entry) => { + const fullPath = path.join(dirPath, entry.name) + if (entry.isDirectory()) { + return { + name: entry.name, + path: fullPath, + is_dir: true, + children: _listDirRecursive(fullPath), + } + } + return { + name: entry.name, + path: fullPath, + is_dir: false, + } + }) +} + +export async function copyFile(source: string, destination: string): Promise { + fs.mkdirSync(path.dirname(destination), { recursive: true }) + fs.copyFileSync(source, destination) +} + +export async function deleteFile(filePath: string): Promise { + if (fs.existsSync(filePath)) { + fs.unlinkSync(filePath) + } +} + +export async function createDirectory(dirPath: string): Promise { + fs.mkdirSync(dirPath, { recursive: true }) +} + +export async function fileExists(filePath: string): Promise { + return fs.existsSync(filePath) +} + +export async function readFileAsBase64(filePath: string): Promise { + const buffer = fs.readFileSync(filePath) + const base64 = buffer.toString("base64") + const ext = path.extname(filePath).toLowerCase() + const mimeMap: Record = { + ".jpg": "image/jpeg", + ".jpeg": "image/jpeg", + ".png": "image/png", + ".gif": "image/gif", + ".webp": "image/webp", + ".pdf": "application/pdf", + } + return { base64, mimeType: mimeMap[ext] ?? "application/octet-stream" } +} + +export async function preprocessFile(filePath: string): Promise { + // For non-GUI mode: just read the file as text + // Real Tauri version uses Rust pdf-extract / docx-rs for binary formats + return fs.readFileSync(filePath, "utf-8") +} + +export async function findRelatedWikiPages( + projectPath: string, + sourceName: string, +): Promise { + const wikiDir = path.join(projectPath, "wiki") + if (!fs.existsSync(wikiDir)) return [] + + const results: string[] = [] + const searchTerm = path.basename(sourceName, path.extname(sourceName)).toLowerCase() + + function searchDir(dir: string) { + const entries = fs.readdirSync(dir, { withFileTypes: true }) + for (const entry of entries) { + const fullPath = path.join(dir, entry.name) + if (entry.isDirectory()) { + searchDir(fullPath) + } else if (entry.name.endsWith(".md")) { + try { + const content = fs.readFileSync(fullPath, "utf-8") + if (content.toLowerCase().includes(searchTerm)) { + results.push(fullPath) + } + } catch { + // skip unreadable files + } + } + } + } + + searchDir(wikiDir) + return results +} + +export async function createProject(name: string, projectPath: string): Promise { + fs.mkdirSync(projectPath, { recursive: true }) + return { id: path.basename(projectPath), name, path: projectPath } +} + +export async function openProject(projectPath: string): Promise { + if (!fs.existsSync(projectPath)) { + throw new Error(`Project not found: ${projectPath}`) + } + return { + id: path.basename(projectPath), + name: path.basename(projectPath), + path: projectPath, + } +} diff --git a/skill/src/stores-node.ts b/skill/src/stores-node.ts new file mode 100644 index 00000000..b18d2a62 --- /dev/null +++ b/skill/src/stores-node.ts @@ -0,0 +1,160 @@ +/** + * Node.js state management replacement for React stores. + * Replaces zustand-based stores with simple module-level state. + * + * Affected stores: + * @/stores/wiki-store → wikiStore + * @/stores/research-store → researchStore + * @/stores/chat-store → chatStore + * @/stores/activity-store → activityStore + * @/stores/review-store → reviewStore + */ + +// --------------------------------------------------------------------------- +// Types +// --------------------------------------------------------------------------- + +export interface LlmConfig { + provider: "openai" | "anthropic" | "google" | "ollama" | "custom" + apiKey: string + model: string + baseUrl?: string +} + +export interface EmbeddingConfig { + enabled: boolean + model: string + apiBase?: string +} + +export interface SearchApiConfig { + provider: "tavily" | "serper" | "none" + apiKey?: string +} + +export interface ReviewItem { + id: string + filePath: string + content: string + reason: string +} + +// --------------------------------------------------------------------------- +// Wiki Store (replaces useWikiStore) +// --------------------------------------------------------------------------- + +const _wikiState = { + projectPath: "", + dataVersion: 0, + embeddingConfig: { + enabled: false, + model: "", + apiBase: undefined as string | undefined, + } as EmbeddingConfig, + llmConfig: { + provider: "openai" as const, + apiKey: process.env.OPENAI_API_KEY ?? "", + model: process.env.LLM_MODEL ?? "gpt-4o", + baseUrl: process.env.OPENAI_API_BASE, + } as LlmConfig, + fileTree: [] as unknown[], +} + +export const useWikiStore = { + getState: () => ({ + ..._wikiState, + setFileTree: (tree: unknown[]) => { _wikiState.fileTree = tree }, + bumpDataVersion: () => { _wikiState.dataVersion++ }, + }), +} + +export function configureWikiStore(opts: { + projectPath: string + llmConfig?: Partial + embeddingConfig?: Partial +}) { + _wikiState.projectPath = opts.projectPath + if (opts.llmConfig) Object.assign(_wikiState.llmConfig, opts.llmConfig) + if (opts.embeddingConfig) Object.assign(_wikiState.embeddingConfig, opts.embeddingConfig) +} + +// --------------------------------------------------------------------------- +// Research Store (replaces useResearchStore) +// --------------------------------------------------------------------------- + +interface ResearchTask { + id: string + topic: string + status: "queued" | "searching" | "synthesizing" | "saving" | "done" | "error" + searchQueries?: string[] + webResults?: unknown[] + synthesis?: string + savedPath?: string + error?: string +} + +const _researchState = { + tasks: [] as ResearchTask[], + maxConcurrent: 3, + panelOpen: false, +} + +export const useResearchStore = { + getState: () => ({ + ..._researchState, + addTask: (topic: string) => { + const id = `task-${Date.now()}-${Math.random().toString(36).slice(2)}` + _researchState.tasks.push({ id, topic, status: "queued" }) + return id + }, + updateTask: (id: string, updates: Partial) => { + const task = _researchState.tasks.find((t) => t.id === id) + if (task) Object.assign(task, updates) + }, + getNextQueued: () => _researchState.tasks.find((t) => t.status === "queued") ?? null, + getRunningCount: () => _researchState.tasks.filter( + (t) => t.status === "searching" || t.status === "synthesizing" || t.status === "saving" + ).length, + setPanelOpen: (open: boolean) => { _researchState.panelOpen = open }, + }), +} + +// --------------------------------------------------------------------------- +// Activity Store (replaces useActivityStore) +// --------------------------------------------------------------------------- + +export const useActivityStore = { + getState: () => ({ + addActivity: (msg: string) => { + console.log(`[Activity] ${msg}`) + }, + }), +} + +// --------------------------------------------------------------------------- +// Chat Store (replaces useChatStore) +// --------------------------------------------------------------------------- + +export const useChatStore = { + getState: () => ({ + addMessage: () => {}, + }), +} + +// --------------------------------------------------------------------------- +// Review Store (replaces useReviewStore) +// --------------------------------------------------------------------------- + +const _reviewState = { + items: [] as ReviewItem[], +} + +export const useReviewStore = { + getState: () => ({ + items: _reviewState.items, + addItem: (item: ReviewItem) => { _reviewState.items.push(item) }, + removeItem: (id: string) => { + _reviewState.items = _reviewState.items.filter((i) => i.id !== id) + }, + }), +} diff --git a/skill/tsconfig.skill.json b/skill/tsconfig.skill.json new file mode 100644 index 00000000..1b37550e --- /dev/null +++ b/skill/tsconfig.skill.json @@ -0,0 +1,28 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "CommonJS", + "moduleResolution": "node", + "outDir": "./dist", + "rootDir": "./src", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "paths": { + "@/commands/fs": ["./src/fs-node.ts"], + "@/stores/wiki-store": ["./src/stores-node.ts"], + "@/stores/research-store": ["./src/stores-node.ts"], + "@/stores/chat-store": ["./src/stores-node.ts"], + "@/stores/activity-store": ["./src/stores-node.ts"], + "@/stores/review-store": ["./src/stores-node.ts"], + "@/lib/*": ["../../src/lib/*"], + "@/types/*": ["../../src/types/*"] + } + }, + "include": ["src/**/*", "../../src/lib/**/*", "../../src/types/**/*"], + "exclude": [ + "../../src/components/**/*", + "../../src/App.tsx", + "../../src/main.tsx" + ] +} From 36ae48736488ae9c6691c37058ae408eb059425d Mon Sep 17 00:00:00 2001 From: toughhou Date: Sat, 2 May 2026 01:49:38 -0700 Subject: [PATCH 2/6] feat(mcp+skill): add MCP server + self-contained Node.js skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MCP Server (mcp-server/): - 5 MCP tools: wiki_status, wiki_search, wiki_graph, wiki_insights, wiki_lint - Standalone Node.js package — no Tauri/GUI required - Claude Desktop + VS Code Copilot chat compatible - tsconfig path aliases replaced with explicit patched lib copies - Degrades gracefully: BM25 search works without vector/LLM config Skill CLI (skill/): - Rebuilt as self-contained package (no path aliases, no Tauri deps) - src/lib/: patched copies of graph-relevance, wiki-graph, graph-insights, search, path-utils - src/shims/: fs-node.ts (Tauri IPC → Node.js fs), stores-node.ts, embedding-stub.ts - src/types/wiki.ts: extracted FileNode/WikiProject types - Commands: graph, insights, search, status, init, lint - All commands tested and working Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- mcp-server/README.md | 78 ++ mcp-server/package-lock.json | 1269 ++++++++++++++++++++++++ mcp-server/package.json | 26 + mcp-server/src/index.ts | 251 +++++ mcp-server/src/lib/graph-insights.ts | 150 +++ mcp-server/src/lib/graph-relevance.ts | 229 +++++ mcp-server/src/lib/path-utils.ts | 38 + mcp-server/src/lib/search.ts | 228 +++++ mcp-server/src/lib/wiki-graph.ts | 211 ++++ mcp-server/src/shims/embedding-stub.ts | 32 + mcp-server/src/shims/fs-node.ts | 101 ++ mcp-server/src/shims/stores-node.ts | 135 +++ mcp-server/src/types/wiki.ts | 18 + mcp-server/tsconfig.json | 18 + skill/package-lock.json | 1269 ++++++++++++++++++++++++ skill/package.json | 19 +- skill/src/cli.ts | 390 +++----- skill/src/lib/graph-insights.ts | 150 +++ skill/src/lib/graph-relevance.ts | 229 +++++ skill/src/lib/path-utils.ts | 38 + skill/src/lib/search.ts | 228 +++++ skill/src/lib/wiki-graph.ts | 211 ++++ skill/src/mcp-server.ts | 251 +++++ skill/src/shims/embedding-stub.ts | 32 + skill/src/shims/fs-node.ts | 101 ++ skill/src/shims/stores-node.ts | 135 +++ skill/src/types/wiki.ts | 18 + skill/tsconfig.json | 19 + 28 files changed, 5599 insertions(+), 275 deletions(-) create mode 100644 mcp-server/README.md create mode 100644 mcp-server/package-lock.json create mode 100644 mcp-server/package.json create mode 100644 mcp-server/src/index.ts create mode 100644 mcp-server/src/lib/graph-insights.ts create mode 100644 mcp-server/src/lib/graph-relevance.ts create mode 100644 mcp-server/src/lib/path-utils.ts create mode 100644 mcp-server/src/lib/search.ts create mode 100644 mcp-server/src/lib/wiki-graph.ts create mode 100644 mcp-server/src/shims/embedding-stub.ts create mode 100644 mcp-server/src/shims/fs-node.ts create mode 100644 mcp-server/src/shims/stores-node.ts create mode 100644 mcp-server/src/types/wiki.ts create mode 100644 mcp-server/tsconfig.json create mode 100644 skill/package-lock.json create mode 100644 skill/src/lib/graph-insights.ts create mode 100644 skill/src/lib/graph-relevance.ts create mode 100644 skill/src/lib/path-utils.ts create mode 100644 skill/src/lib/search.ts create mode 100644 skill/src/lib/wiki-graph.ts create mode 100644 skill/src/mcp-server.ts create mode 100644 skill/src/shims/embedding-stub.ts create mode 100644 skill/src/shims/fs-node.ts create mode 100644 skill/src/shims/stores-node.ts create mode 100644 skill/src/types/wiki.ts create mode 100644 skill/tsconfig.json diff --git a/mcp-server/README.md b/mcp-server/README.md new file mode 100644 index 00000000..bcdeb72f --- /dev/null +++ b/mcp-server/README.md @@ -0,0 +1,78 @@ +# llm_wiki MCP Server + +An [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server that exposes **llm_wiki** backend operations as AI-callable tools. + +Use it with Claude Desktop, VS Code Copilot Chat, Cursor, or any MCP-compatible host to give your AI assistant direct access to your wiki knowledge base. + +## Tools + +| Tool | Description | +|------|-------------| +| `wiki_status` | Page count and type breakdown | +| `wiki_search` | BM25 keyword search (+ optional vector via `EMBEDDING_ENABLED=true`) | +| `wiki_graph` | Build Louvain knowledge graph — nodes, edges, community clusters | +| `wiki_insights` | Find surprising cross-community connections + knowledge gaps | +| `wiki_lint` | Structural lint: orphaned pages, isolated nodes, broken links | + +## Quick Start + +```bash +cd mcp-server +npm install +npm run build +``` + +### Claude Desktop + +Add to `~/Library/Application Support/Claude/claude_desktop_config.json`: + +```json +{ + "mcpServers": { + "llm-wiki": { + "command": "node", + "args": ["/path/to/llm_wiki/mcp-server/dist/index.js"], + "env": { + "WIKI_PATH": "/path/to/your/wiki-project" + } + } + } +} +``` + +### VS Code Copilot (`.vscode/mcp.json`) + +```json +{ + "servers": { + "llm-wiki": { + "type": "stdio", + "command": "node", + "args": ["${workspaceFolder}/mcp-server/dist/index.js"], + "env": { "WIKI_PATH": "${workspaceFolder}" } + } + } +} +``` + +## Environment Variables + +| Variable | Description | Default | +|----------|-------------|---------| +| `WIKI_PATH` | Default project path (used when `project_path` not specified) | `process.cwd()` | +| `EMBEDDING_ENABLED` | Enable vector search via LanceDB | `false` | +| `EMBEDDING_MODEL` | Embedding model name (e.g. `text-embedding-3-small`) | — | +| `OPENAI_API_KEY` | API key for LLM + embedding calls | — | +| `SKILL_VERBOSE` | Set to `1` for verbose activity logging | — | + +## Architecture + +The MCP server runs entirely in Node.js without the Tauri desktop app. It replaces the Tauri IPC layer (`@/commands/fs`) with standard Node.js `fs` operations, making it suitable for headless server and CI/CD environments. + +**Capabilities without Tauri**: +- ✅ `wiki_search` — BM25 keyword search +- ✅ `wiki_graph` — Louvain community detection +- ✅ `wiki_insights` — Surprising connections + knowledge gaps +- ✅ `wiki_lint` — Structural lint +- ⚠️ `wiki_search` with vector — requires `EMBEDDING_ENABLED=true` + configured API +- ❌ `ingest` — PDF/DOCX extraction not supported (use pre-converted Markdown) diff --git a/mcp-server/package-lock.json b/mcp-server/package-lock.json new file mode 100644 index 00000000..68e090ca --- /dev/null +++ b/mcp-server/package-lock.json @@ -0,0 +1,1269 @@ +{ + "name": "llm-wiki-mcp-server", + "version": "0.4.6", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "llm-wiki-mcp-server", + "version": "0.4.6", + "license": "MIT", + "dependencies": { + "@modelcontextprotocol/sdk": "^1.0.0", + "graphology": "^0.25.4", + "graphology-communities-louvain": "^2.0.0" + }, + "bin": { + "llm-wiki-mcp": "dist/index.js" + }, + "devDependencies": { + "@types/node": "^20.0.0", + "typescript": "^5.0.0" + } + }, + "node_modules/@hono/node-server": { + "version": "1.19.14", + "resolved": "https://registry.npmjs.org/@hono/node-server/-/node-server-1.19.14.tgz", + "integrity": "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw==", + "license": "MIT", + "engines": { + "node": ">=18.14.1" + }, + "peerDependencies": { + "hono": "^4" + } + }, + "node_modules/@modelcontextprotocol/sdk": { + "version": "1.29.0", + "resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.29.0.tgz", + "integrity": "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ==", + "license": "MIT", + "dependencies": { + "@hono/node-server": "^1.19.9", + "ajv": "^8.17.1", + "ajv-formats": "^3.0.1", + "content-type": "^1.0.5", + "cors": "^2.8.5", + "cross-spawn": "^7.0.5", + "eventsource": "^3.0.2", + "eventsource-parser": "^3.0.0", + "express": "^5.2.1", + "express-rate-limit": "^8.2.1", + "hono": "^4.11.4", + "jose": "^6.1.3", + "json-schema-typed": "^8.0.2", + "pkce-challenge": "^5.0.0", + "raw-body": "^3.0.0", + "zod": "^3.25 || ^4.0", + "zod-to-json-schema": "^3.25.1" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "@cfworker/json-schema": "^4.1.1", + "zod": "^3.25 || ^4.0" + }, + "peerDependenciesMeta": { + "@cfworker/json-schema": { + "optional": true + }, + "zod": { + "optional": false + } + } + }, + "node_modules/@types/node": { + "version": "20.19.39", + "resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.39.tgz", + "integrity": "sha512-orrrD74MBUyK8jOAD/r0+lfa1I2MO6I+vAkmAWzMYbCcgrN4lCrmK52gRFQq/JRxfYPfonkr4b0jcY7Olqdqbw==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": "~6.21.0" + } + }, + "node_modules/accepts": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz", + "integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==", + "license": "MIT", + "dependencies": { + "mime-types": "^3.0.0", + "negotiator": "^1.0.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/ajv": { + "version": "8.20.0", + "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", + "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", + "license": "MIT", + "dependencies": { + "fast-deep-equal": "^3.1.3", + "fast-uri": "^3.0.1", + "json-schema-traverse": "^1.0.0", + "require-from-string": "^2.0.2" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/epoberezkin" + } + }, + "node_modules/ajv-formats": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/ajv-formats/-/ajv-formats-3.0.1.tgz", + "integrity": "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==", + "license": "MIT", + "dependencies": { + "ajv": "^8.0.0" + }, + "peerDependencies": { + "ajv": "^8.0.0" + }, + "peerDependenciesMeta": { + "ajv": { + "optional": true + } + } + }, + "node_modules/body-parser": { + "version": "2.2.2", + "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.2.tgz", + "integrity": "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA==", + "license": "MIT", + "dependencies": { + "bytes": "^3.1.2", + "content-type": "^1.0.5", + "debug": "^4.4.3", + "http-errors": "^2.0.0", + "iconv-lite": "^0.7.0", + "on-finished": "^2.4.1", + "qs": "^6.14.1", + "raw-body": "^3.0.1", + "type-is": "^2.0.1" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/bytes": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", + "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/call-bind-apply-helpers": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", + "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/call-bound": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz", + "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "get-intrinsic": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/content-disposition": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.1.0.tgz", + "integrity": "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/content-type": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz", + "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", + "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie-signature": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz", + "integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==", + "license": "MIT", + "engines": { + "node": ">=6.6.0" + } + }, + "node_modules/cors": { + "version": "2.8.6", + "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.6.tgz", + "integrity": "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==", + "license": "MIT", + "dependencies": { + "object-assign": "^4", + "vary": "^1" + }, + "engines": { + "node": ">= 0.10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/cross-spawn": { + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", + "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", + "license": "MIT", + "dependencies": { + "path-key": "^3.1.0", + "shebang-command": "^2.0.0", + "which": "^2.0.1" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/depd": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", + "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/dunder-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", + "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.1", + "es-errors": "^1.3.0", + "gopd": "^1.2.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/ee-first": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", + "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==", + "license": "MIT" + }, + "node_modules/encodeurl": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", + "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-object-atoms": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz", + "integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/escape-html": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", + "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", + "license": "MIT" + }, + "node_modules/etag": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", + "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/events": { + "version": "3.3.0", + "resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz", + "integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==", + "license": "MIT", + "engines": { + "node": ">=0.8.x" + } + }, + "node_modules/eventsource": { + "version": "3.0.7", + "resolved": "https://registry.npmjs.org/eventsource/-/eventsource-3.0.7.tgz", + "integrity": "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA==", + "license": "MIT", + "dependencies": { + "eventsource-parser": "^3.0.1" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/eventsource-parser": { + "version": "3.0.8", + "resolved": "https://registry.npmjs.org/eventsource-parser/-/eventsource-parser-3.0.8.tgz", + "integrity": "sha512-70QWGkr4snxr0OXLRWsFLeRBIRPuQOvt4s8QYjmUlmlkyTZkRqS7EDVRZtzU3TiyDbXSzaOeF0XUKy8PchzukQ==", + "license": "MIT", + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/express": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", + "integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==", + "license": "MIT", + "dependencies": { + "accepts": "^2.0.0", + "body-parser": "^2.2.1", + "content-disposition": "^1.0.0", + "content-type": "^1.0.5", + "cookie": "^0.7.1", + "cookie-signature": "^1.2.1", + "debug": "^4.4.0", + "depd": "^2.0.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "finalhandler": "^2.1.0", + "fresh": "^2.0.0", + "http-errors": "^2.0.0", + "merge-descriptors": "^2.0.0", + "mime-types": "^3.0.0", + "on-finished": "^2.4.1", + "once": "^1.4.0", + "parseurl": "^1.3.3", + "proxy-addr": "^2.0.7", + "qs": "^6.14.0", + "range-parser": "^1.2.1", + "router": "^2.2.0", + "send": "^1.1.0", + "serve-static": "^2.2.0", + "statuses": "^2.0.1", + "type-is": "^2.0.1", + "vary": "^1.1.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/express-rate-limit": { + "version": "8.4.1", + "resolved": "https://registry.npmjs.org/express-rate-limit/-/express-rate-limit-8.4.1.tgz", + "integrity": "sha512-NGVYwQSAyEQgzxX1iCM978PP9AdO/hW93gMcF6ZwQCm+rFvLsBH6w4xcXWTcliS8La5EPRN3p9wzItqBwJrfNw==", + "license": "MIT", + "dependencies": { + "ip-address": "10.1.0" + }, + "engines": { + "node": ">= 16" + }, + "funding": { + "url": "https://github.com/sponsors/express-rate-limit" + }, + "peerDependencies": { + "express": ">= 4.11" + } + }, + "node_modules/fast-deep-equal": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz", + "integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==", + "license": "MIT" + }, + "node_modules/fast-uri": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.0.tgz", + "integrity": "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "license": "BSD-3-Clause" + }, + "node_modules/finalhandler": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz", + "integrity": "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "on-finished": "^2.4.1", + "parseurl": "^1.3.3", + "statuses": "^2.0.1" + }, + "engines": { + "node": ">= 18.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/forwarded": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", + "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/fresh": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz", + "integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/function-bind": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", + "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-intrinsic": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", + "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "es-define-property": "^1.0.1", + "es-errors": "^1.3.0", + "es-object-atoms": "^1.1.1", + "function-bind": "^1.1.2", + "get-proto": "^1.0.1", + "gopd": "^1.2.0", + "has-symbols": "^1.1.0", + "hasown": "^2.0.2", + "math-intrinsics": "^1.1.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", + "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", + "license": "MIT", + "dependencies": { + "dunder-proto": "^1.0.1", + "es-object-atoms": "^1.0.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/graphology": { + "version": "0.25.4", + "resolved": "https://registry.npmjs.org/graphology/-/graphology-0.25.4.tgz", + "integrity": "sha512-33g0Ol9nkWdD6ulw687viS8YJQBxqG5LWII6FI6nul0pq6iM2t5EKquOTFDbyTblRB3O9I+7KX4xI8u5ffekAQ==", + "license": "MIT", + "dependencies": { + "events": "^3.3.0", + "obliterator": "^2.0.2" + }, + "peerDependencies": { + "graphology-types": ">=0.24.0" + } + }, + "node_modules/graphology-communities-louvain": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/graphology-communities-louvain/-/graphology-communities-louvain-2.0.2.tgz", + "integrity": "sha512-zt+2hHVPYxjEquyecxWXoUoIuN/UvYzsvI7boDdMNz0rRvpESQ7+e+Ejv6wK7AThycbZXuQ6DkG8NPMCq6XwoA==", + "license": "MIT", + "dependencies": { + "graphology-indices": "^0.17.0", + "graphology-utils": "^2.4.4", + "mnemonist": "^0.39.0", + "pandemonium": "^2.4.1" + }, + "peerDependencies": { + "graphology-types": ">=0.19.0" + } + }, + "node_modules/graphology-indices": { + "version": "0.17.0", + "resolved": "https://registry.npmjs.org/graphology-indices/-/graphology-indices-0.17.0.tgz", + "integrity": "sha512-A7RXuKQvdqSWOpn7ZVQo4S33O0vCfPBnUSf7FwE0zNCasqwZVUaCXePuWo5HBpWw68KJcwObZDHpFk6HKH6MYQ==", + "license": "MIT", + "dependencies": { + "graphology-utils": "^2.4.2", + "mnemonist": "^0.39.0" + }, + "peerDependencies": { + "graphology-types": ">=0.20.0" + } + }, + "node_modules/graphology-types": { + "version": "0.24.8", + "resolved": "https://registry.npmjs.org/graphology-types/-/graphology-types-0.24.8.tgz", + "integrity": "sha512-hDRKYXa8TsoZHjgEaysSRyPdT6uB78Ci8WnjgbStlQysz7xR52PInxNsmnB7IBOM1BhikxkNyCVEFgmPKnpx3Q==", + "license": "MIT", + "peer": true + }, + "node_modules/graphology-utils": { + "version": "2.5.2", + "resolved": "https://registry.npmjs.org/graphology-utils/-/graphology-utils-2.5.2.tgz", + "integrity": "sha512-ckHg8MXrXJkOARk56ZaSCM1g1Wihe2d6iTmz1enGOz4W/l831MBCKSayeFQfowgF8wd+PQ4rlch/56Vs/VZLDQ==", + "license": "MIT", + "peerDependencies": { + "graphology-types": ">=0.23.0" + } + }, + "node_modules/has-symbols": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", + "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/hasown": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.3.tgz", + "integrity": "sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg==", + "license": "MIT", + "dependencies": { + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/hono": { + "version": "4.12.16", + "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.16.tgz", + "integrity": "sha512-jN0ZewiNAWSe5khM3EyCmBb250+b40wWbwNILNfEvq84VREWwOIkuUsFONk/3i3nqkz7Oe1PcpM2mwQEK2L9Kg==", + "license": "MIT", + "engines": { + "node": ">=16.9.0" + } + }, + "node_modules/http-errors": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", + "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==", + "license": "MIT", + "dependencies": { + "depd": "~2.0.0", + "inherits": "~2.0.4", + "setprototypeof": "~1.2.0", + "statuses": "~2.0.2", + "toidentifier": "~1.0.1" + }, + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/iconv-lite": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.2.tgz", + "integrity": "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw==", + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", + "license": "ISC" + }, + "node_modules/ip-address": { + "version": "10.1.0", + "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.1.0.tgz", + "integrity": "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q==", + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, + "node_modules/ipaddr.js": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", + "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==", + "license": "MIT", + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/is-promise": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz", + "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==", + "license": "MIT" + }, + "node_modules/isexe": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", + "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", + "license": "ISC" + }, + "node_modules/jose": { + "version": "6.2.3", + "resolved": "https://registry.npmjs.org/jose/-/jose-6.2.3.tgz", + "integrity": "sha512-YYVDInQKFJfR/xa3ojUTl8c2KoTwiL1R5Wg9YCydwH0x0B9grbzlg5HC7mMjCtUJjbQ/YnGEZIhI5tCgfTb4Hw==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/panva" + } + }, + "node_modules/json-schema-traverse": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", + "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", + "license": "MIT" + }, + "node_modules/json-schema-typed": { + "version": "8.0.2", + "resolved": "https://registry.npmjs.org/json-schema-typed/-/json-schema-typed-8.0.2.tgz", + "integrity": "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA==", + "license": "BSD-2-Clause" + }, + "node_modules/math-intrinsics": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", + "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/media-typer": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.0.tgz", + "integrity": "sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/merge-descriptors": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz", + "integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/mime-db": { + "version": "1.54.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", + "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime-types": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", + "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", + "license": "MIT", + "dependencies": { + "mime-db": "^1.54.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/mnemonist": { + "version": "0.39.8", + "resolved": "https://registry.npmjs.org/mnemonist/-/mnemonist-0.39.8.tgz", + "integrity": "sha512-vyWo2K3fjrUw8YeeZ1zF0fy6Mu59RHokURlld8ymdUPjMlD9EC9ov1/YPqTgqRvUN9nTr3Gqfz29LYAmu0PHPQ==", + "license": "MIT", + "dependencies": { + "obliterator": "^2.0.1" + } + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/negotiator": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.0.0.tgz", + "integrity": "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/object-assign": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", + "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/object-inspect": { + "version": "1.13.4", + "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", + "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/obliterator": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/obliterator/-/obliterator-2.0.5.tgz", + "integrity": "sha512-42CPE9AhahZRsMNslczq0ctAEtqk8Eka26QofnqC346BZdHDySk3LWka23LI7ULIw11NmltpiLagIq8gBozxTw==", + "license": "MIT" + }, + "node_modules/on-finished": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", + "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", + "license": "MIT", + "dependencies": { + "ee-first": "1.1.1" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", + "license": "ISC", + "dependencies": { + "wrappy": "1" + } + }, + "node_modules/pandemonium": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/pandemonium/-/pandemonium-2.4.1.tgz", + "integrity": "sha512-wRqjisUyiUfXowgm7MFH2rwJzKIr20rca5FsHXCMNm1W5YPP1hCtrZfgmQ62kP7OZ7Xt+cR858aB28lu5NX55g==", + "license": "MIT", + "dependencies": { + "mnemonist": "^0.39.2" + } + }, + "node_modules/parseurl": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", + "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/path-key": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", + "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/path-to-regexp": { + "version": "8.4.2", + "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.4.2.tgz", + "integrity": "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA==", + "license": "MIT", + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/pkce-challenge": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/pkce-challenge/-/pkce-challenge-5.0.1.tgz", + "integrity": "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ==", + "license": "MIT", + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/proxy-addr": { + "version": "2.0.7", + "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", + "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==", + "license": "MIT", + "dependencies": { + "forwarded": "0.2.0", + "ipaddr.js": "1.9.1" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/qs": { + "version": "6.15.1", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.1.tgz", + "integrity": "sha512-6YHEFRL9mfgcAvql/XhwTvf5jKcOiiupt2FiJxHkiX1z4j7WL8J/jRHYLluORvc1XxB5rV20KoeK00gVJamspg==", + "license": "BSD-3-Clause", + "dependencies": { + "side-channel": "^1.1.0" + }, + "engines": { + "node": ">=0.6" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/range-parser": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", + "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/raw-body": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.2.tgz", + "integrity": "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==", + "license": "MIT", + "dependencies": { + "bytes": "~3.1.2", + "http-errors": "~2.0.1", + "iconv-lite": "~0.7.0", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/require-from-string": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", + "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/router": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz", + "integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.0", + "depd": "^2.0.0", + "is-promise": "^4.0.0", + "parseurl": "^1.3.3", + "path-to-regexp": "^8.0.0" + }, + "engines": { + "node": ">= 18" + } + }, + "node_modules/safer-buffer": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", + "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", + "license": "MIT" + }, + "node_modules/send": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz", + "integrity": "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.3", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "fresh": "^2.0.0", + "http-errors": "^2.0.1", + "mime-types": "^3.0.2", + "ms": "^2.1.3", + "on-finished": "^2.4.1", + "range-parser": "^1.2.1", + "statuses": "^2.0.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/serve-static": { + "version": "2.2.1", + "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz", + "integrity": "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==", + "license": "MIT", + "dependencies": { + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "parseurl": "^1.3.3", + "send": "^1.2.0" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/setprototypeof": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", + "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==", + "license": "ISC" + }, + "node_modules/shebang-command": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", + "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", + "license": "MIT", + "dependencies": { + "shebang-regex": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/shebang-regex": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", + "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/side-channel": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz", + "integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.3", + "side-channel-list": "^1.0.0", + "side-channel-map": "^1.0.1", + "side-channel-weakmap": "^1.0.2" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-list": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz", + "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-map": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz", + "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-weakmap": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz", + "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3", + "side-channel-map": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/statuses": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", + "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/toidentifier": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", + "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==", + "license": "MIT", + "engines": { + "node": ">=0.6" + } + }, + "node_modules/type-is": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.0.1.tgz", + "integrity": "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw==", + "license": "MIT", + "dependencies": { + "content-type": "^1.0.5", + "media-typer": "^1.1.0", + "mime-types": "^3.0.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/typescript": { + "version": "5.9.3", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", + "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/undici-types": { + "version": "6.21.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", + "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/unpipe": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", + "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/vary": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", + "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/which": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", + "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", + "license": "ISC", + "dependencies": { + "isexe": "^2.0.0" + }, + "bin": { + "node-which": "bin/node-which" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/wrappy": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", + "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", + "license": "ISC" + }, + "node_modules/zod": { + "version": "4.4.2", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.4.2.tgz", + "integrity": "sha512-IynmDyxsEsb9RKzO3J9+4SxXnl2FTFSzNBaKKaMV6tsSk0rw9gYw9gs+JFCq/qk2LCZ78KDwyj+Z289TijSkUw==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } + }, + "node_modules/zod-to-json-schema": { + "version": "3.25.2", + "resolved": "https://registry.npmjs.org/zod-to-json-schema/-/zod-to-json-schema-3.25.2.tgz", + "integrity": "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA==", + "license": "ISC", + "peerDependencies": { + "zod": "^3.25.28 || ^4" + } + } + } +} diff --git a/mcp-server/package.json b/mcp-server/package.json new file mode 100644 index 00000000..523cd426 --- /dev/null +++ b/mcp-server/package.json @@ -0,0 +1,26 @@ +{ + "name": "llm-wiki-mcp-server", + "version": "0.4.6", + "description": "MCP server for llm_wiki — exposes wiki graph, search, insights, and lint as Model Context Protocol tools", + "main": "dist/index.js", + "bin": { + "llm-wiki-mcp": "dist/index.js" + }, + "scripts": { + "build": "tsc -p tsconfig.json", + "typecheck": "tsc -p tsconfig.json --noEmit", + "start": "node dist/index.js", + "dev": "ts-node src/index.ts" + }, + "keywords": ["mcp", "wiki", "knowledge-graph", "llm", "model-context-protocol"], + "license": "MIT", + "dependencies": { + "graphology": "^0.25.4", + "graphology-communities-louvain": "^2.0.0", + "@modelcontextprotocol/sdk": "^1.0.0" + }, + "devDependencies": { + "@types/node": "^20.0.0", + "typescript": "^5.0.0" + } +} diff --git a/mcp-server/src/index.ts b/mcp-server/src/index.ts new file mode 100644 index 00000000..b07ee8df --- /dev/null +++ b/mcp-server/src/index.ts @@ -0,0 +1,251 @@ +#!/usr/bin/env node +/** + * llm-wiki MCP Server + * + * Exposes nashsu/llm_wiki backend operations as Model Context Protocol tools. + * Works with Claude Desktop, VS Code Copilot, and any MCP-compatible host. + * + * Tools: + * wiki_status — Page count and type breakdown for a project + * wiki_search — BM25 keyword search (+ optional vector via EMBEDDING_ENABLED) + * wiki_graph — Build knowledge graph (nodes, edges, Louvain communities) + * wiki_insights — Surprising connections and knowledge gaps analysis + * wiki_lint — Structural lint: orphans, no-outlinks, broken links + * + * Usage: + * node dist/index.js + * WIKI_PATH=/path/to/project node dist/index.js (default project path) + */ +import { Server } from "@modelcontextprotocol/sdk/server/index.js" +import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js" +import { + CallToolRequestSchema, + ListToolsRequestSchema, + ErrorCode, + McpError, +} from "@modelcontextprotocol/sdk/types.js" + +import * as path from "path" +import { buildWikiGraph } from "./lib/wiki-graph" +import { findSurprisingConnections, detectKnowledgeGaps } from "./lib/graph-insights" +import { searchWiki } from "./lib/search" + +const DEFAULT_WIKI_PATH = process.env.WIKI_PATH ?? process.cwd() +const PKG_VERSION = "0.4.6-mcp" + +const server = new Server( + { name: "llm-wiki", version: PKG_VERSION }, + { capabilities: { tools: {} } }, +) + +// ── Tool definitions ────────────────────────────────────────────────────────── +server.setRequestHandler(ListToolsRequestSchema, async () => ({ + tools: [ + { + name: "wiki_status", + description: "Get page count and type breakdown for a wiki project. Returns statistics about the knowledge base.", + inputSchema: { + type: "object", + properties: { + project_path: { + type: "string", + description: "Absolute path to the wiki project directory (contains wiki/ subdirectory)", + }, + }, + required: [], + }, + }, + { + name: "wiki_search", + description: "Search wiki pages using BM25 keyword matching with optional vector search (RRF fusion). Returns ranked results with snippets.", + inputSchema: { + type: "object", + properties: { + query: { type: "string", description: "Search query (supports Chinese and English)" }, + project_path: { type: "string", description: "Path to wiki project (defaults to WIKI_PATH env var)" }, + limit: { type: "number", description: "Max results to return (default: 10)" }, + }, + required: ["query"], + }, + }, + { + name: "wiki_graph", + description: "Build knowledge graph from wiki pages: wikilinks, type-based edges, Louvain community detection. Returns nodes, edges, and community clusters.", + inputSchema: { + type: "object", + properties: { + project_path: { type: "string", description: "Path to wiki project" }, + format: { + type: "string", + enum: ["json", "summary"], + description: "Output format: 'json' for full graph data, 'summary' for human-readable overview (default: summary)", + }, + }, + required: [], + }, + }, + { + name: "wiki_insights", + description: "Analyze wiki graph structure to find surprising cross-community connections and knowledge gaps (isolated pages, sparse clusters, bridge nodes).", + inputSchema: { + type: "object", + properties: { + project_path: { type: "string", description: "Path to wiki project" }, + max_connections: { type: "number", description: "Max surprising connections to return (default: 5)" }, + max_gaps: { type: "number", description: "Max knowledge gaps to return (default: 8)" }, + }, + required: [], + }, + }, + { + name: "wiki_lint", + description: "Structural lint of wiki pages: find orphaned pages (no links), no-outlinks, and connectivity issues.", + inputSchema: { + type: "object", + properties: { + project_path: { type: "string", description: "Path to wiki project" }, + }, + required: [], + }, + }, + ], +})) + +// ── Tool handlers ───────────────────────────────────────────────────────────── +server.setRequestHandler(CallToolRequestSchema, async (request) => { + const { name, arguments: args = {} } = request.params + const projectPath = path.resolve((args.project_path as string | undefined) ?? DEFAULT_WIKI_PATH) + + try { + switch (name) { + case "wiki_status": { + const { nodes, communities } = await buildWikiGraph(projectPath) + const typeCounts: Record = {} + for (const n of nodes) typeCounts[n.type] = (typeCounts[n.type] ?? 0) + 1 + const summary = [ + `Wiki: ${projectPath}`, + `Total pages: ${nodes.length}`, + `Communities: ${communities.length}`, + ...Object.entries(typeCounts) + .sort((a, b) => b[1] - a[1]) + .map(([t, c]) => ` ${t}: ${c}`), + ].join("\n") + return { content: [{ type: "text", text: summary }] } + } + + case "wiki_search": { + if (!args.query) throw new McpError(ErrorCode.InvalidParams, "query is required") + const results = await searchWiki(projectPath, args.query as string) + const limit = typeof args.limit === "number" ? args.limit : 10 + const top = results.slice(0, limit) + if (top.length === 0) { + return { content: [{ type: "text", text: `No results for: "${args.query}"` }] } + } + const lines = [`# Search: "${args.query}"\n`] + for (const r of top) { + const relPath = path.relative(projectPath, r.path) + lines.push(`## ${r.title}`) + lines.push(`**Path**: ${relPath} | **Score**: ${r.score.toFixed(4)}`) + lines.push(r.snippet) + lines.push("") + } + return { content: [{ type: "text", text: lines.join("\n") }] } + } + + case "wiki_graph": { + const graphData = await buildWikiGraph(projectPath) + const format = (args.format as string | undefined) ?? "summary" + if (format === "json") { + return { content: [{ type: "text", text: JSON.stringify(graphData, null, 2) }] } + } + // Summary format + const { nodes, edges, communities } = graphData + const typeCounts: Record = {} + for (const n of nodes) typeCounts[n.type] = (typeCounts[n.type] ?? 0) + 1 + const lines = [ + `# Knowledge Graph Summary`, + ``, + `**Nodes**: ${nodes.length} | **Edges**: ${edges.length} | **Communities**: ${communities.length}`, + ``, + `## Node Types`, + ...Object.entries(typeCounts).sort((a, b) => b[1] - a[1]).map(([t, c]) => `- ${t}: ${c}`), + ``, + `## Top Communities`, + ...communities.slice(0, 5).map((c, i) => + `### Community ${i + 1} (${c.nodeCount} pages, cohesion: ${c.cohesion.toFixed(2)})\nKey pages: ${c.topNodes.join(", ")}` + ), + ``, + `## Top Hubs (by link count)`, + ...nodes.sort((a, b) => b.linkCount - a.linkCount).slice(0, 10) + .map((n) => `- ${n.label} (${n.type}, ${n.linkCount} links)`), + ] + return { content: [{ type: "text", text: lines.join("\n") }] } + } + + case "wiki_insights": { + const { nodes, edges, communities } = await buildWikiGraph(projectPath) + const maxConn = typeof args.max_connections === "number" ? args.max_connections : 5 + const maxGaps = typeof args.max_gaps === "number" ? args.max_gaps : 8 + const connections = findSurprisingConnections(nodes, edges, communities, maxConn) + const gaps = detectKnowledgeGaps(nodes, edges, communities, maxGaps) + + const lines = [`# Wiki Insights\n`, `## Surprising Connections\n`] + if (connections.length === 0) lines.push("_No surprising connections found yet._\n") + for (const c of connections) { + lines.push(`### ${c.source.label} ↔ ${c.target.label}`) + lines.push(`- Score: ${c.score} | ${c.reasons.join(", ")}\n`) + } + lines.push(`## Knowledge Gaps\n`) + if (gaps.length === 0) lines.push("_No gaps detected._\n") + for (const g of gaps) { + lines.push(`### ${g.title}`) + lines.push(`${g.description}`) + lines.push(`💡 ${g.suggestion}\n`) + } + return { content: [{ type: "text", text: lines.join("\n") }] } + } + + case "wiki_lint": { + const { nodes, edges } = await buildWikiGraph(projectPath) + if (nodes.length === 0) { + return { content: [{ type: "text", text: "No wiki pages found." }] } + } + const edgeTargets = new Set(edges.map((e) => e.target)) + const edgeSources = new Set(edges.map((e) => e.source)) + const allLinked = new Set([...edgeTargets, ...edgeSources]) + const issues: string[] = [] + for (const n of nodes) { + if (n.id === "index" || n.id === "log" || n.id === "overview") continue + if (!allLinked.has(n.id)) issues.push(`[orphan] ${n.label} (${n.id}.md)`) + else if (n.linkCount <= 1) issues.push(`[isolated] ${n.label} — only ${n.linkCount} link(s)`) + } + const text = issues.length === 0 + ? `✓ All ${nodes.length} pages are properly connected.` + : `Found ${issues.length} issue(s) in ${nodes.length} pages:\n\n${issues.join("\n")}` + return { content: [{ type: "text", text: text }] } + } + + default: + throw new McpError(ErrorCode.MethodNotFound, `Unknown tool: ${name}`) + } + } catch (err) { + if (err instanceof McpError) throw err + throw new McpError( + ErrorCode.InternalError, + `Tool '${name}' failed: ${err instanceof Error ? err.message : String(err)}`, + ) + } +}) + +// ── Start server ────────────────────────────────────────────────────────────── +async function main() { + const transport = new StdioServerTransport() + await server.connect(transport) + console.error(`llm-wiki MCP server v${PKG_VERSION} started`) + console.error(`Default wiki path: ${DEFAULT_WIKI_PATH}`) +} + +main().catch((err) => { + console.error("Failed to start MCP server:", err) + process.exit(1) +}) diff --git a/mcp-server/src/lib/graph-insights.ts b/mcp-server/src/lib/graph-insights.ts new file mode 100644 index 00000000..56fa212f --- /dev/null +++ b/mcp-server/src/lib/graph-insights.ts @@ -0,0 +1,150 @@ +import type { GraphNode, GraphEdge, CommunityInfo } from "./wiki-graph" + +export interface SurprisingConnection { + source: GraphNode + target: GraphNode + score: number + reasons: string[] + key: string +} + +export interface KnowledgeGap { + type: "isolated-node" | "sparse-community" | "bridge-node" + title: string + description: string + nodeIds: string[] + suggestion: string +} + +export function findSurprisingConnections( + nodes: GraphNode[], + edges: GraphEdge[], + _communities: CommunityInfo[], + limit: number = 5, +): SurprisingConnection[] { + const nodeMap = new Map(nodes.map((n) => [n.id, n])) + const degreeMap = new Map(nodes.map((n) => [n.id, n.linkCount])) + const maxDegree = Math.max(...nodes.map((n) => n.linkCount), 1) + const STRUCTURAL_IDS = new Set(["index", "log", "overview"]) + const scored: SurprisingConnection[] = [] + + for (const edge of edges) { + const source = nodeMap.get(edge.source) + const target = nodeMap.get(edge.target) + if (!source || !target) continue + if (STRUCTURAL_IDS.has(source.id) || STRUCTURAL_IDS.has(target.id)) continue + + let score = 0 + const reasons: string[] = [] + + if (source.community !== target.community) { + score += 3 + reasons.push("crosses community boundary") + } + if (source.type !== target.type) { + const distantPairs = new Set([ + "source-concept", "concept-source", "source-synthesis", "synthesis-source", + "query-entity", "entity-query", + ]) + if (distantPairs.has(`${source.type}-${target.type}`)) { + score += 2 + reasons.push(`connects ${source.type} to ${target.type}`) + } else { + score += 1 + reasons.push("different types") + } + } + const sourceDeg = degreeMap.get(source.id) ?? 0 + const targetDeg = degreeMap.get(target.id) ?? 0 + const minDeg = Math.min(sourceDeg, targetDeg) + const maxDeg = Math.max(sourceDeg, targetDeg) + if (minDeg <= 2 && maxDeg >= maxDegree * 0.5) { + score += 2 + reasons.push("peripheral node links to hub") + } + if (edge.weight < 2 && edge.weight > 0) { + score += 1 + reasons.push("weak but present connection") + } + if (score >= 3 && reasons.length > 0) { + const key = [source.id, target.id].sort().join(":::") + scored.push({ source, target, score, reasons, key }) + } + } + + scored.sort((a, b) => b.score - a.score) + return scored.slice(0, limit) +} + +export function detectKnowledgeGaps( + nodes: GraphNode[], + edges: GraphEdge[], + communities: CommunityInfo[], + limit: number = 8, +): KnowledgeGap[] { + const gaps: KnowledgeGap[] = [] + const nodeMap = new Map(nodes.map((n) => [n.id, n])) + + // 1. Isolated nodes (degree ≤ 1) + const isolatedNodes = nodes.filter( + (n) => n.linkCount <= 1 && n.type !== "overview" && n.id !== "index" && n.id !== "log", + ) + if (isolatedNodes.length > 0) { + const topIsolated = isolatedNodes.slice(0, 5) + gaps.push({ + type: "isolated-node", + title: `${isolatedNodes.length} isolated page${isolatedNodes.length > 1 ? "s" : ""}`, + description: topIsolated.map((n) => n.label).join(", ") + + (isolatedNodes.length > 5 ? ` and ${isolatedNodes.length - 5} more` : ""), + nodeIds: isolatedNodes.map((n) => n.id), + suggestion: "These pages have few or no connections. Consider adding [[wikilinks]] to related pages.", + }) + } + + // 2. Sparse communities (low cohesion) + for (const comm of communities) { + if (comm.cohesion < 0.15 && comm.nodeCount >= 3) { + gaps.push({ + type: "sparse-community", + title: `Sparse cluster: ${comm.topNodes[0] ?? `Community ${comm.id}`}`, + description: `${comm.nodeCount} pages with cohesion ${comm.cohesion.toFixed(2)} — internal connections are weak.`, + nodeIds: nodes.filter((n) => n.community === comm.id).map((n) => n.id), + suggestion: "This knowledge area lacks internal cross-references. Consider adding links between these pages.", + }) + } + } + + // 3. Bridge nodes (connected to multiple communities) + const communityNeighbors = new Map>() + for (const node of nodes) communityNeighbors.set(node.id, new Set()) + for (const edge of edges) { + const sourceNode = nodeMap.get(edge.source) + const targetNode = nodeMap.get(edge.target) + if (sourceNode && targetNode) { + communityNeighbors.get(edge.source)?.add(targetNode.community) + communityNeighbors.get(edge.target)?.add(sourceNode.community) + } + } + const STRUCTURAL_IDS = new Set(["index", "log", "overview"]) + const bridgeNodes = nodes + .filter((n) => { + if (STRUCTURAL_IDS.has(n.id)) return false + const neighborComms = communityNeighbors.get(n.id) + return neighborComms && neighborComms.size >= 3 + }) + .sort((a, b) => (communityNeighbors.get(b.id)?.size ?? 0) - (communityNeighbors.get(a.id)?.size ?? 0)) + .slice(0, 3) + + for (const bridge of bridgeNodes) { + const commCount = communityNeighbors.get(bridge.id)?.size ?? 0 + gaps.push({ + type: "bridge-node", + title: `Key bridge: ${bridge.label}`, + description: `Connects ${commCount} different knowledge clusters. This is a critical junction in your wiki.`, + nodeIds: [bridge.id], + suggestion: "This page bridges multiple knowledge areas. Ensure it's well-maintained and expanded.", + }) + } + + return gaps.slice(0, limit) +} diff --git a/mcp-server/src/lib/graph-relevance.ts b/mcp-server/src/lib/graph-relevance.ts new file mode 100644 index 00000000..db27e7b1 --- /dev/null +++ b/mcp-server/src/lib/graph-relevance.ts @@ -0,0 +1,229 @@ +import { readFile, listDirectory } from "../shims/fs-node" +import type { FileNode } from "../types/wiki" +import { normalizePath } from "./path-utils" + +export interface RetrievalNode { + readonly id: string + readonly title: string + readonly type: string + readonly path: string + readonly sources: readonly string[] + readonly outLinks: ReadonlySet + readonly inLinks: ReadonlySet +} + +export interface RetrievalGraph { + readonly nodes: ReadonlyMap + readonly dataVersion: number +} + +const WIKILINK_REGEX = /\[\[([^\]|]+?)(?:\|[^\]]+?)?\]\]/g + +const WEIGHTS = { + directLink: 3.0, + sourceOverlap: 4.0, + commonNeighbor: 1.5, + typeAffinity: 1.0, +} as const + +const TYPE_AFFINITY: Record> = { + entity: { concept: 1.2, entity: 0.8, source: 1.0, synthesis: 1.0, query: 0.8 }, + concept: { entity: 1.2, concept: 0.8, source: 1.0, synthesis: 1.2, query: 1.0 }, + source: { entity: 1.0, concept: 1.0, source: 0.5, query: 0.8, synthesis: 1.0 }, + query: { concept: 1.0, entity: 0.8, synthesis: 1.0, source: 0.8, query: 0.5 }, + synthesis: { concept: 1.2, entity: 1.0, source: 1.0, query: 1.0, synthesis: 0.8 }, +} + +let cachedGraph: RetrievalGraph | null = null + +function flattenMdFiles(nodes: readonly FileNode[]): FileNode[] { + const files: FileNode[] = [] + for (const node of nodes) { + if (node.is_dir && node.children) { + files.push(...flattenMdFiles(node.children)) + } else if (!node.is_dir && node.name.endsWith(".md")) { + files.push(node) + } + } + return files +} + +function fileNameToId(fileName: string): string { + return fileName.replace(/\.md$/, "") +} + +function extractFrontmatter(content: string): { title: string; type: string; sources: string[] } { + const fmMatch = content.match(/^---\n([\s\S]*?)\n---/) + const fm = fmMatch ? fmMatch[1] : "" + const titleMatch = fm.match(/^title:\s*["']?(.+?)["']?\s*$/m) + const typeMatch = fm.match(/^type:\s*["']?(.+?)["']?\s*$/m) + const sources: string[] = [] + const sourcesBlockMatch = fm.match(/^sources:\s*\n((?:\s+-\s+.+\n?)*)/m) + if (sourcesBlockMatch) { + const lines = sourcesBlockMatch[1].split("\n") + for (const line of lines) { + const itemMatch = line.match(/^\s+-\s+["']?(.+?)["']?\s*$/) + if (itemMatch) sources.push(itemMatch[1]) + } + } else { + const inlineMatch = fm.match(/^sources:\s*\[([^\]]*)\]/m) + if (inlineMatch) { + const items = inlineMatch[1].split(",") + for (const item of items) { + const trimmed = item.trim().replace(/^["']|["']$/g, "") + if (trimmed) sources.push(trimmed) + } + } + } + let title = titleMatch ? titleMatch[1].trim() : "" + if (!title) { + const headingMatch = content.match(/^#\s+(.+)$/m) + title = headingMatch ? headingMatch[1].trim() : "" + } + return { title, type: typeMatch ? typeMatch[1].trim().toLowerCase() : "other", sources } +} + +function extractWikilinks(content: string): string[] { + const links: string[] = [] + const regex = new RegExp(WIKILINK_REGEX.source, "g") + let match: RegExpExecArray | null + while ((match = regex.exec(content)) !== null) links.push(match[1].trim()) + return links +} + +function resolveTarget(raw: string, nodeIds: ReadonlySet): string | null { + if (nodeIds.has(raw)) return raw + const normalized = raw.toLowerCase().replace(/\s+/g, "-") + for (const id of nodeIds) { + const idLower = id.toLowerCase() + if (idLower === normalized) return id + if (idLower === raw.toLowerCase()) return id + if (idLower.replace(/\s+/g, "-") === normalized) return id + } + return null +} + +function getNeighbors(node: RetrievalNode): ReadonlySet { + const neighbors = new Set() + for (const id of node.outLinks) neighbors.add(id) + for (const id of node.inLinks) neighbors.add(id) + return neighbors +} + +function getNodeDegree(node: RetrievalNode): number { + return node.outLinks.size + node.inLinks.size +} + +export async function buildRetrievalGraph( + projectPath: string, + dataVersion: number = 0, +): Promise { + if (cachedGraph !== null && cachedGraph.dataVersion === dataVersion) return cachedGraph + + const wikiRoot = `${normalizePath(projectPath)}/wiki` + let tree: FileNode[] + try { + tree = await listDirectory(wikiRoot) + } catch { + const emptyGraph: RetrievalGraph = { nodes: new Map(), dataVersion } + cachedGraph = emptyGraph + return emptyGraph + } + + const mdFiles = flattenMdFiles(tree) + const rawNodes: Array<{ + id: string; title: string; type: string; path: string + sources: string[]; rawLinks: string[]; fileName: string + }> = [] + + for (const file of mdFiles) { + const id = fileNameToId(file.name) + let content = "" + try { content = await readFile(file.path) } catch { continue } + const fm = extractFrontmatter(content) + rawNodes.push({ + id, title: fm.title || file.name.replace(/\.md$/, "").replace(/-/g, " "), + type: fm.type, path: file.path, sources: fm.sources, + rawLinks: extractWikilinks(content), fileName: file.name, + }) + } + + const nodeIds = new Set(rawNodes.map((n) => n.id)) + const outLinksMap = new Map>() + const inLinksMap = new Map>() + for (const id of nodeIds) { + outLinksMap.set(id, new Set()) + inLinksMap.set(id, new Set()) + } + + for (const raw of rawNodes) { + for (const linkTarget of raw.rawLinks) { + const resolvedId = resolveTarget(linkTarget, nodeIds) + if (resolvedId === null || resolvedId === raw.id) continue + outLinksMap.get(raw.id)!.add(resolvedId) + inLinksMap.get(resolvedId)!.add(raw.id) + } + } + + const nodes = new Map() + for (const raw of rawNodes) { + nodes.set(raw.id, { + id: raw.id, title: raw.title, type: raw.type, path: raw.path, + sources: Object.freeze([...raw.sources]), + outLinks: Object.freeze(outLinksMap.get(raw.id) ?? new Set()), + inLinks: Object.freeze(inLinksMap.get(raw.id) ?? new Set()), + }) + } + + const graph: RetrievalGraph = { nodes, dataVersion } + cachedGraph = graph + return graph +} + +export function calculateRelevance( + nodeA: RetrievalNode, nodeB: RetrievalNode, graph: RetrievalGraph, +): number { + if (nodeA.id === nodeB.id) return 0 + const forwardLinks = nodeA.outLinks.has(nodeB.id) ? 1 : 0 + const backwardLinks = nodeB.outLinks.has(nodeA.id) ? 1 : 0 + const directLinkScore = (forwardLinks + backwardLinks) * WEIGHTS.directLink + const sourcesA = new Set(nodeA.sources) + let sharedSourceCount = 0 + for (const src of nodeB.sources) { if (sourcesA.has(src)) sharedSourceCount += 1 } + const sourceOverlapScore = sharedSourceCount * WEIGHTS.sourceOverlap + const neighborsA = getNeighbors(nodeA) + const neighborsB = getNeighbors(nodeB) + let adamicAdar = 0 + for (const neighborId of neighborsA) { + if (neighborsB.has(neighborId)) { + const neighbor = graph.nodes.get(neighborId) + if (neighbor) { + const degree = getNodeDegree(neighbor) + adamicAdar += 1 / Math.log(Math.max(degree, 2)) + } + } + } + const commonNeighborScore = adamicAdar * WEIGHTS.commonNeighbor + const affinityMap = TYPE_AFFINITY[nodeA.type] + const typeAffinityScore = (affinityMap?.[nodeB.type] ?? 0.5) * WEIGHTS.typeAffinity + return directLinkScore + sourceOverlapScore + commonNeighborScore + typeAffinityScore +} + +export function getRelatedNodes( + nodeId: string, graph: RetrievalGraph, limit: number = 5, +): ReadonlyArray<{ node: RetrievalNode; relevance: number }> { + const sourceNode = graph.nodes.get(nodeId) + if (!sourceNode) return [] + const scored: Array<{ node: RetrievalNode; relevance: number }> = [] + for (const [id, node] of graph.nodes) { + if (id === nodeId) continue + const relevance = calculateRelevance(sourceNode, node, graph) + if (relevance > 0) scored.push({ node, relevance }) + } + scored.sort((a, b) => b.relevance - a.relevance) + return scored.slice(0, limit) +} + +export function clearGraphCache(): void { + cachedGraph = null +} diff --git a/mcp-server/src/lib/path-utils.ts b/mcp-server/src/lib/path-utils.ts new file mode 100644 index 00000000..3296d0a9 --- /dev/null +++ b/mcp-server/src/lib/path-utils.ts @@ -0,0 +1,38 @@ +export function normalizePath(p: string): string { + return p.replace(/\\/g, "/") +} + +export function joinPath(...segments: string[]): string { + return segments + .map((s) => s.replace(/\\/g, "/")) + .join("/") + .replace(/\/+/g, "/") +} + +export function getFileName(p: string): string { + const normalized = p.replace(/\\/g, "/") + return normalized.split("/").pop() ?? p +} + +export function getFileStem(p: string): string { + const name = getFileName(p) + const lastDot = name.lastIndexOf(".") + return lastDot > 0 ? name.slice(0, lastDot) : name +} + +export function getRelativePath(fullPath: string, basePath: string): string { + const normalFull = normalizePath(fullPath) + const normalBase = normalizePath(basePath).replace(/\/$/, "") + if (normalFull.startsWith(normalBase + "/")) { + return normalFull.slice(normalBase.length + 1) + } + return normalFull +} + +export function isAbsolutePath(p: string): boolean { + if (!p) return false + if (p.startsWith("/")) return true + if (/^[A-Za-z]:[\\/]/.test(p)) return true + if (p.startsWith("\\\\") || p.startsWith("//")) return true + return false +} diff --git a/mcp-server/src/lib/search.ts b/mcp-server/src/lib/search.ts new file mode 100644 index 00000000..d67e4e51 --- /dev/null +++ b/mcp-server/src/lib/search.ts @@ -0,0 +1,228 @@ +import { readFile, listDirectory } from "../shims/fs-node" +import type { FileNode } from "../types/wiki" +import { normalizePath, getFileStem } from "./path-utils" + +export interface ImageRef { + url: string + alt: string +} + +export interface SearchResult { + path: string + title: string + snippet: string + titleMatch: boolean + score: number + images: ImageRef[] +} + +const MAX_RESULTS = 20 +const SNIPPET_CONTEXT = 80 +const RRF_K = 60 +const FILENAME_EXACT_BONUS = 200 +const PHRASE_IN_TITLE_BONUS = 50 +const PHRASE_IN_CONTENT_PER_OCC = 20 +const MAX_PHRASE_OCC_COUNTED = 10 +const TITLE_TOKEN_WEIGHT = 5 +const CONTENT_TOKEN_WEIGHT = 1 + +const STOP_WORDS = new Set([ + "的", "是", "了", "什么", "在", "有", "和", "与", "对", "从", + "the", "is", "a", "an", "what", "how", "are", "was", "were", + "do", "does", "did", "be", "been", "being", "have", "has", "had", + "it", "its", "in", "on", "at", "to", "for", "of", "with", "by", + "this", "that", "these", "those", +]) + +export function tokenizeQuery(query: string): string[] { + const rawTokens = query + .toLowerCase() + .split(/[\s,，。！？、；：""''（）()\-_/\\·~～…]+/) + .filter((t) => t.length > 1) + .filter((t) => !STOP_WORDS.has(t)) + + const tokens: string[] = [] + for (const token of rawTokens) { + const hasCJK = /[\u4e00-\u9fff\u3400-\u4dbf]/.test(token) + if (hasCJK && token.length > 2) { + const chars = [...token] + for (let i = 0; i < chars.length - 1; i++) tokens.push(chars[i] + chars[i + 1]) + for (const ch of chars) { if (!STOP_WORDS.has(ch)) tokens.push(ch) } + tokens.push(token) + } else { + tokens.push(token) + } + } + return [...new Set(tokens)] +} + +function tokenMatchScore(text: string, tokens: readonly string[]): number { + const lower = text.toLowerCase() + let score = 0 + for (const token of tokens) { if (lower.includes(token)) score += 1 } + return score +} + +function countOccurrences(haystackLower: string, needleLower: string): number { + if (!needleLower) return 0 + let count = 0; let pos = 0 + while (true) { + const idx = haystackLower.indexOf(needleLower, pos) + if (idx === -1) break + count++; pos = idx + needleLower.length + } + return count +} + +function flattenMdFiles(nodes: FileNode[]): FileNode[] { + const files: FileNode[] = [] + for (const node of nodes) { + if (node.is_dir && node.children) files.push(...flattenMdFiles(node.children)) + else if (!node.is_dir && node.name.endsWith(".md")) files.push(node) + } + return files +} + +function extractTitle(content: string, fileName: string): string { + const fm = content.match(/^---\n[\s\S]*?^title:\s*["']?(.+?)["']?\s*$/m) + if (fm) return fm[1].trim() + const h = content.match(/^#\s+(.+)$/m) + if (h) return h[1].trim() + return fileName.replace(/\.md$/, "").replace(/-/g, " ") +} + +const IMAGE_REF_RE = /!\[([^\]]*)\]\(([^)\s]+)\)/g + +function extractImageRefs(content: string): ImageRef[] { + const seen = new Set(); const out: ImageRef[] = [] + for (const m of content.matchAll(IMAGE_REF_RE)) { + const url = m[2] + if (seen.has(url)) continue + seen.add(url); out.push({ url, alt: m[1] }) + } + return out +} + +function buildSnippet(content: string, query: string): string { + const lower = content.toLowerCase(); const lowerQuery = query.toLowerCase() + const idx = lower.indexOf(lowerQuery) + if (idx === -1) return content.slice(0, SNIPPET_CONTEXT * 2).replace(/\n/g, " ") + const start = Math.max(0, idx - SNIPPET_CONTEXT) + const end = Math.min(content.length, idx + query.length + SNIPPET_CONTEXT) + let snippet = content.slice(start, end).replace(/\n/g, " ") + if (start > 0) snippet = "..." + snippet + if (end < content.length) snippet = snippet + "..." + return snippet +} + +const TRIM_PUNCT_RE = /^[\s,，。！？、；：""''（）()\-_/\\·~～…]+|[\s,，。！？、；：""''（）()\-_/\\·~～…]+$/g +const SEARCH_READ_CONCURRENCY = 16 + +function scoreFile( + file: FileNode, content: string, tokens: readonly string[], queryPhrase: string, query: string, +): SearchResult | null { + const title = extractTitle(content, file.name) + const titleText = `${title} ${file.name}` + const titleLower = titleText.toLowerCase() + const contentLower = content.toLowerCase() + const fileStem = file.name.replace(/\.md$/, "").toLowerCase() + + const filenameExact = fileStem === queryPhrase + const titleHasPhrase = queryPhrase.length > 0 && titleLower.includes(queryPhrase) + const contentPhraseOcc = Math.min(countOccurrences(contentLower, queryPhrase), MAX_PHRASE_OCC_COUNTED) + const titleTokenScore = tokenMatchScore(titleText, tokens) + const contentTokenScore = tokenMatchScore(content, tokens) + + if (!filenameExact && !titleHasPhrase && contentPhraseOcc === 0 && titleTokenScore === 0 && contentTokenScore === 0) return null + + const score = + (filenameExact ? FILENAME_EXACT_BONUS : 0) + + (titleHasPhrase ? PHRASE_IN_TITLE_BONUS : 0) + + contentPhraseOcc * PHRASE_IN_CONTENT_PER_OCC + + titleTokenScore * TITLE_TOKEN_WEIGHT + + contentTokenScore * CONTENT_TOKEN_WEIGHT + + const snippetAnchor = contentPhraseOcc > 0 ? queryPhrase : (tokens.find((t) => contentLower.includes(t)) ?? query) + return { + path: file.path, title, snippet: buildSnippet(content, snippetAnchor), + titleMatch: titleTokenScore > 0 || titleHasPhrase, score, images: extractImageRefs(content), + } +} + +async function searchFiles( + files: FileNode[], tokens: readonly string[], query: string, results: SearchResult[], +): Promise { + const queryPhrase = query.trim().toLowerCase().replace(TRIM_PUNCT_RE, "") + for (let i = 0; i < files.length; i += SEARCH_READ_CONCURRENCY) { + const batch = files.slice(i, i + SEARCH_READ_CONCURRENCY) + const batchResults = await Promise.all( + batch.map(async (file) => { + let content: string + try { content = await readFile(file.path) } catch { return null } + return scoreFile(file, content, tokens, queryPhrase, query) + }), + ) + for (const r of batchResults) { if (r) results.push(r) } + } +} + +export async function searchWiki(projectPath: string, query: string): Promise { + if (!query.trim()) return [] + const pp = normalizePath(projectPath) + const tokens = tokenizeQuery(query) + const effectiveTokens = tokens.length > 0 ? tokens : [query.trim().toLowerCase()] + const results: SearchResult[] = [] + + try { + const wikiTree = await listDirectory(`${pp}/wiki`) + const wikiFiles = flattenMdFiles(wikiTree) + await searchFiles(wikiFiles, effectiveTokens, query, results) + } catch { /* no wiki directory */ } + + const tokenSorted = [...results].sort((a, b) => b.score - a.score) + const tokenRank = new Map() + tokenSorted.forEach((r, i) => tokenRank.set(normalizePath(r.path), i + 1)) + + // Vector search (optional — gracefully degrades when embedding not configured) + let vectorRank = new Map() + let vectorCount = 0 + try { + const { useWikiStore } = await import("../shims/stores-node") + const embCfg = useWikiStore.getState().embeddingConfig + if (embCfg.enabled && embCfg.model) { + const { searchByEmbedding } = await import("../shims/embedding-stub") + const vectorResults = await searchByEmbedding(pp, query, embCfg, 10) + vectorCount = vectorResults.length + vectorResults.forEach((vr, i) => vectorRank.set(vr.id, i + 1)) + + const knownIds = new Set(results.map((r) => getFileStem(r.path))) + for (const vr of vectorResults) { + if (knownIds.has(vr.id)) continue + const dirs = ["entities", "concepts", "sources", "synthesis", "comparison", "queries"] + for (const dir of dirs) { + const tryPath = `${pp}/wiki/${dir}/${vr.id}.md` + try { + const content = await readFile(tryPath) + const title = extractTitle(content, `${vr.id}.md`) + results.push({ path: tryPath, title, snippet: buildSnippet(content, query), titleMatch: false, score: 0, images: extractImageRefs(content) }) + knownIds.add(vr.id); break + } catch { /* not in this dir */ } + } + } + } + } catch { /* vector search not available */ } + + // RRF fusion + for (const r of results) { + const tRank = tokenRank.get(normalizePath(r.path)) + const vRank = vectorRank.get(getFileStem(r.path)) + let rrf = 0 + if (tRank !== undefined) rrf += 1 / (RRF_K + tRank) + if (vRank !== undefined) rrf += 1 / (RRF_K + vRank) + r.score = rrf + } + + results.sort((a, b) => b.score !== a.score ? b.score - a.score : a.path.localeCompare(b.path)) + console.error(`[search] "${query}" | token:${tokenRank.size} vector:${vectorCount} → ${results.length} results`) + return results.slice(0, MAX_RESULTS) +} diff --git a/mcp-server/src/lib/wiki-graph.ts b/mcp-server/src/lib/wiki-graph.ts new file mode 100644 index 00000000..5160563a --- /dev/null +++ b/mcp-server/src/lib/wiki-graph.ts @@ -0,0 +1,211 @@ +import { readFile, listDirectory } from "../shims/fs-node" +import type { FileNode } from "../types/wiki" +import { buildRetrievalGraph, calculateRelevance } from "./graph-relevance" +import { normalizePath } from "./path-utils" +import Graph from "graphology" +import louvain from "graphology-communities-louvain" + +export interface GraphNode { + id: string + label: string + type: string + path: string + linkCount: number + community: number +} + +export interface GraphEdge { + source: string + target: string + weight: number +} + +export interface CommunityInfo { + id: number + nodeCount: number + cohesion: number + topNodes: string[] +} + +function detectCommunities( + nodes: { id: string; label: string; linkCount: number }[], + edges: GraphEdge[], +): { assignments: Map; communities: CommunityInfo[] } { + if (nodes.length === 0) return { assignments: new Map(), communities: [] } + + const g = new Graph({ type: "undirected" }) + for (const node of nodes) g.addNode(node.id) + for (const edge of edges) { + if (g.hasNode(edge.source) && g.hasNode(edge.target)) { + const key = `${edge.source}->${edge.target}` + if (!g.hasEdge(key) && !g.hasEdge(`${edge.target}->${edge.source}`)) { + g.addEdgeWithKey(key, edge.source, edge.target, { weight: edge.weight }) + } + } + } + + const communityMap: Record = louvain(g, { resolution: 1 }) + const assignments = new Map(Object.entries(communityMap).map(([k, v]) => [k, v as number])) + + const groups = new Map() + for (const [nodeId, commId] of assignments) { + const list = groups.get(commId) ?? [] + list.push(nodeId) + groups.set(commId, list) + } + + const edgeSet = new Set() + for (const edge of edges) { + edgeSet.add(`${edge.source}:::${edge.target}`) + edgeSet.add(`${edge.target}:::${edge.source}`) + } + + const nodeInfo = new Map(nodes.map((n) => [n.id, { label: n.label, linkCount: n.linkCount }])) + const communities: CommunityInfo[] = [] + + for (const [commId, memberIds] of groups) { + const n = memberIds.length + let intraEdges = 0 + for (let i = 0; i < memberIds.length; i++) { + for (let j = i + 1; j < memberIds.length; j++) { + if (edgeSet.has(`${memberIds[i]}:::${memberIds[j]}`)) intraEdges++ + } + } + const possibleEdges = n > 1 ? (n * (n - 1)) / 2 : 1 + const cohesion = intraEdges / possibleEdges + const sorted = [...memberIds].sort( + (a, b) => (nodeInfo.get(b)?.linkCount ?? 0) - (nodeInfo.get(a)?.linkCount ?? 0) + ) + communities.push({ id: commId, nodeCount: n, cohesion, topNodes: sorted.slice(0, 5).map((id) => nodeInfo.get(id)?.label ?? id) }) + } + + communities.sort((a, b) => b.nodeCount - a.nodeCount) + const idRemap = new Map() + communities.forEach((c, idx) => { idRemap.set(c.id, idx); c.id = idx }) + for (const [nodeId, oldId] of assignments) assignments.set(nodeId, idRemap.get(oldId) ?? 0) + + return { assignments, communities } +} + +const WIKILINK_REGEX = /\[\[([^\]|]+?)(?:\|[^\]]+?)?\]\]/g + +function flattenMdFiles(nodes: FileNode[]): FileNode[] { + const files: FileNode[] = [] + for (const node of nodes) { + if (node.is_dir && node.children) files.push(...flattenMdFiles(node.children)) + else if (!node.is_dir && node.name.endsWith(".md")) files.push(node) + } + return files +} + +function extractTitle(content: string, fileName: string): string { + const fm = content.match(/^---\n[\s\S]*?^title:\s*["']?(.+?)["']?\s*$/m) + if (fm) return fm[1].trim() + const h = content.match(/^#\s+(.+)$/m) + if (h) return h[1].trim() + return fileName.replace(/\.md$/, "").replace(/-/g, " ") +} + +function extractType(content: string): string { + const m = content.match(/^---\n[\s\S]*?^type:\s*["']?(.+?)["']?\s*$/m) + return m ? m[1].trim().toLowerCase() : "other" +} + +function extractWikilinks(content: string): string[] { + const links: string[] = [] + const regex = new RegExp(WIKILINK_REGEX.source, "g") + let match: RegExpExecArray | null + while ((match = regex.exec(content)) !== null) links.push(match[1].trim()) + return links +} + +function resolveTarget(raw: string, nodeMap: Map): string | null { + if (nodeMap.has(raw)) return raw + const normalized = raw.toLowerCase().replace(/\s+/g, "-") + for (const id of nodeMap.keys()) { + if (id.toLowerCase() === normalized) return id + if (id.toLowerCase() === raw.toLowerCase()) return id + if (id.toLowerCase().replace(/\s+/g, "-") === normalized) return id + } + return null +} + +export async function buildWikiGraph( + projectPath: string, +): Promise<{ nodes: GraphNode[]; edges: GraphEdge[]; communities: CommunityInfo[] }> { + const wikiRoot = `${normalizePath(projectPath)}/wiki` + let tree: FileNode[] + try { tree = await listDirectory(wikiRoot) } catch { + return { nodes: [], edges: [], communities: [] } + } + + const mdFiles = flattenMdFiles(tree) + if (mdFiles.length === 0) return { nodes: [], edges: [], communities: [] } + + const nodeMap = new Map() + for (const file of mdFiles) { + const id = file.name.replace(/\.md$/, "") + let content = "" + try { content = await readFile(file.path) } catch { continue } + nodeMap.set(id, { id, label: extractTitle(content, file.name), type: extractType(content), path: file.path, links: extractWikilinks(content) }) + } + + const HIDDEN_TYPES = new Set(["query"]) + for (const [id, node] of nodeMap) { + if (HIDDEN_TYPES.has(node.type)) nodeMap.delete(id) + } + + const linkCounts = new Map() + for (const [id] of nodeMap) linkCounts.set(id, 0) + + const rawEdges: GraphEdge[] = [] + for (const [sourceId, nodeData] of nodeMap) { + for (const targetRaw of nodeData.links) { + const targetId = resolveTarget(targetRaw, nodeMap) + if (targetId === null || targetId === sourceId) continue + rawEdges.push({ source: sourceId, target: targetId, weight: 1 }) + linkCounts.set(sourceId, (linkCounts.get(sourceId) ?? 0) + 1) + linkCounts.set(targetId, (linkCounts.get(targetId) ?? 0) + 1) + } + } + + const seenEdges = new Set() + const dedupedEdges: { source: string; target: string }[] = [] + for (const edge of rawEdges) { + const key = `${edge.source}:::${edge.target}` + const reverseKey = `${edge.target}:::${edge.source}` + if (!seenEdges.has(key) && !seenEdges.has(reverseKey)) { + seenEdges.add(key) + dedupedEdges.push(edge) + } + } + + // Try to get retrieval graph for weighted edges (gracefully degrades) + let retrievalGraph: Awaited> | null = null + try { + const { useWikiStore } = await import("../shims/stores-node") + const dv = useWikiStore.getState().dataVersion + retrievalGraph = await buildRetrievalGraph(normalizePath(projectPath), dv) + } catch { /* ignore — weights default to 1 */ } + + const edges: GraphEdge[] = dedupedEdges.map((e) => { + let weight = 1 + if (retrievalGraph) { + const nodeA = retrievalGraph.nodes.get(e.source) + const nodeB = retrievalGraph.nodes.get(e.target) + if (nodeA && nodeB) weight = calculateRelevance(nodeA, nodeB, retrievalGraph) + } + return { source: e.source, target: e.target, weight } + }) + + const prelimNodes = Array.from(nodeMap.values()).map((n) => ({ id: n.id, label: n.label, linkCount: linkCounts.get(n.id) ?? 0 })) + const { assignments, communities } = detectCommunities(prelimNodes, edges) + + const nodes: GraphNode[] = Array.from(nodeMap.values()).map((n) => ({ + id: n.id, label: n.label, type: n.type, path: n.path, + linkCount: linkCounts.get(n.id) ?? 0, + community: assignments.get(n.id) ?? 0, + })) + + return { nodes, edges, communities } +} diff --git a/mcp-server/src/shims/embedding-stub.ts b/mcp-server/src/shims/embedding-stub.ts new file mode 100644 index 00000000..8a2117aa --- /dev/null +++ b/mcp-server/src/shims/embedding-stub.ts @@ -0,0 +1,32 @@ +/** + * Stub for @/lib/embedding — vector search is not available in the skill/MCP layer + * without a running LanceDB instance. Returns empty results so BM25-only search works. + */ +export interface EmbeddingConfig { + enabled: boolean + model: string + apiBase?: string + apiKey?: string +} + +export interface VectorSearchResult { + id: string + score: number + path?: string +} + +export async function searchByEmbedding( + _projectPath: string, + _query: string, + _config: EmbeddingConfig, + _limit: number = 10, +): Promise { + return [] +} + +export async function createEmbedding( + _text: string, + _config: EmbeddingConfig, +): Promise { + return [] +} diff --git a/mcp-server/src/shims/fs-node.ts b/mcp-server/src/shims/fs-node.ts new file mode 100644 index 00000000..de343bda --- /dev/null +++ b/mcp-server/src/shims/fs-node.ts @@ -0,0 +1,101 @@ +/** + * Node.js drop-in replacement for @/commands/fs (nashsu Tauri IPC layer). + * Replaces all invoke("...") calls with standard Node.js fs operations. + */ +import * as fs from "fs" +import * as path from "path" +import type { FileNode } from "../types/wiki" + +export async function readFile(filePath: string): Promise { + return fs.readFileSync(filePath, "utf-8") +} + +export async function writeFile(filePath: string, content: string): Promise { + fs.mkdirSync(path.dirname(filePath), { recursive: true }) + fs.writeFileSync(filePath, content, "utf-8") +} + +export async function listDirectory(dirPath: string): Promise { + function walk(dir: string): FileNode[] { + const entries = fs.readdirSync(dir, { withFileTypes: true }) + return entries.map((entry) => { + const entryPath = path.join(dir, entry.name).replace(/\\/g, "/") + if (entry.isDirectory()) { + return { + name: entry.name, + path: entryPath, + is_dir: true, + children: walk(entryPath), + } + } + return { name: entry.name, path: entryPath, is_dir: false } + }) + } + return walk(dirPath) +} + +export async function copyFile(from: string, to: string): Promise { + fs.mkdirSync(path.dirname(to), { recursive: true }) + fs.copyFileSync(from, to) +} + +export async function deleteFile(filePath: string): Promise { + if (fs.existsSync(filePath)) fs.unlinkSync(filePath) +} + +export async function createDirectory(dirPath: string): Promise { + fs.mkdirSync(dirPath, { recursive: true }) +} + +export async function fileExists(filePath: string): Promise { + return fs.existsSync(filePath) +} + +export async function readFileAsBase64(filePath: string): Promise { + return fs.readFileSync(filePath).toString("base64") +} + +/** + * Text extraction for PDFs/DOCX/etc. + * In Node mode: returns raw file content if text-based, otherwise empty string. + * For real PDF extraction, users should pre-extract with markitdown or pdftotext. + */ +export async function preprocessFile(filePath: string): Promise { + const ext = path.extname(filePath).toLowerCase() + const textExts = [".md", ".txt", ".json", ".yaml", ".yml", ".csv", ".html", ".htm"] + if (textExts.includes(ext)) { + try { + return fs.readFileSync(filePath, "utf-8") + } catch { + return "" + } + } + // For binary files (PDF, DOCX, etc.) return empty — use pre-extracted markdown + console.warn(`[fs-node] preprocessFile: binary format not supported in Node mode: ${filePath}`) + return "" +} + +export async function findRelatedWikiPages( + sourceFile: string, + wikiRoot: string, +): Promise { + const stem = path.basename(sourceFile, path.extname(sourceFile)).toLowerCase() + const results: string[] = [] + function walk(dir: string): void { + if (!fs.existsSync(dir)) return + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name) + if (entry.isDirectory()) walk(full) + else if (entry.name.endsWith(".md")) { + try { + const content = fs.readFileSync(full, "utf-8") + if (content.toLowerCase().includes(stem)) { + results.push(full.replace(/\\/g, "/")) + } + } catch { /* skip */ } + } + } + } + walk(wikiRoot) + return results +} diff --git a/mcp-server/src/shims/stores-node.ts b/mcp-server/src/shims/stores-node.ts new file mode 100644 index 00000000..66fa3773 --- /dev/null +++ b/mcp-server/src/shims/stores-node.ts @@ -0,0 +1,135 @@ +/** + * Node.js drop-in for React zustand stores used by nashsu/llm_wiki lib files. + * Replaces all useXxxStore.getState() calls with module-level state. + */ + +export interface LlmConfig { + provider: string + apiKey: string + model: string + baseUrl?: string + temperature?: number + maxTokens?: number +} + +export interface EmbeddingConfig { + enabled: boolean + model: string + apiBase?: string + apiKey?: string +} + +interface WikiState { + projectPath: string + dataVersion: number + llmConfig: LlmConfig + embeddingConfig: EmbeddingConfig +} + +let wikiState: WikiState = { + projectPath: "", + dataVersion: 0, + llmConfig: { + provider: process.env.LLM_PROVIDER ?? "openai", + apiKey: process.env.OPENAI_API_KEY ?? process.env.LLM_API_KEY ?? "", + model: process.env.LLM_MODEL ?? "gpt-4o-mini", + baseUrl: process.env.LLM_BASE_URL, + }, + embeddingConfig: { + enabled: (process.env.EMBEDDING_ENABLED ?? "false") === "true", + model: process.env.EMBEDDING_MODEL ?? "", + apiBase: process.env.EMBEDDING_BASE_URL, + apiKey: process.env.EMBEDDING_API_KEY ?? process.env.OPENAI_API_KEY, + }, +} + +export const useWikiStore = { + getState: () => ({ ...wikiState }), + setState: (updater: Partial | ((s: WikiState) => Partial)) => { + if (typeof updater === "function") { + wikiState = { ...wikiState, ...updater(wikiState) } + } else { + wikiState = { ...wikiState, ...updater } + } + }, +} + +/** Configure the wiki store from environment variables or explicit config */ +export function configureWikiStore(config: Partial) { + wikiState = { ...wikiState, ...config } +} + +// ── Research store ─────────────────────────────────────────────────────────── +interface ResearchState { + activeProjectPath: string + isResearching: boolean +} + +let researchState: ResearchState = { + activeProjectPath: "", + isResearching: false, +} + +export const useResearchStore = { + getState: () => ({ ...researchState }), + setState: (updater: Partial) => { + researchState = { ...researchState, ...updater } + }, +} + +// ── Activity store (replaces Tauri event system) ───────────────────────────── +export interface ActivityItem { + id: string + type: string + title: string + status: "pending" | "running" | "done" | "error" + detail?: string + filesWritten?: string[] +} + +let activityItems: ActivityItem[] = [] +let activityIdCounter = 0 + +export const useActivityStore = { + getState: () => ({ + items: [...activityItems], + addItem: (item: Omit): string => { + const id = `activity-${++activityIdCounter}` + const newItem: ActivityItem = { id, ...item } + activityItems.push(newItem) + if (process.env.SKILL_VERBOSE === "1") { + console.error(`[activity:${item.type}] ${item.title} — ${item.status}`) + } + return id + }, + updateItem: (id: string, updates: Partial): void => { + const idx = activityItems.findIndex((i) => i.id === id) + if (idx >= 0) { + activityItems[idx] = { ...activityItems[idx], ...updates } + if (process.env.SKILL_VERBOSE === "1") { + const item = activityItems[idx] + console.error(`[activity:update] ${item.title} — ${item.status}: ${item.detail ?? ""}`) + } + } + }, + clearItems: () => { activityItems = [] }, + }), + addItem: (item: Omit): string => { + return useActivityStore.getState().addItem(item) + }, + updateItem: (id: string, updates: Partial): void => { + useActivityStore.getState().updateItem(id, updates) + }, +} + +// ── Chat store ─────────────────────────────────────────────────────────────── +export const useChatStore = { + getState: () => ({ messages: [] as unknown[] }), + setState: (_updater: unknown) => {}, +} + +// ── Review store ───────────────────────────────────────────────────────────── +export const useReviewStore = { + getState: () => ({ queue: [] as unknown[], isProcessing: false }), + setState: (_updater: unknown) => {}, +} diff --git a/mcp-server/src/types/wiki.ts b/mcp-server/src/types/wiki.ts new file mode 100644 index 00000000..a51ace03 --- /dev/null +++ b/mcp-server/src/types/wiki.ts @@ -0,0 +1,18 @@ +export interface WikiProject { + id: string + name: string + path: string +} + +export interface FileNode { + name: string + path: string + is_dir: boolean + children?: FileNode[] +} + +export interface WikiPage { + path: string + content: string + frontmatter: Record +} diff --git a/mcp-server/tsconfig.json b/mcp-server/tsconfig.json new file mode 100644 index 00000000..e37a4324 --- /dev/null +++ b/mcp-server/tsconfig.json @@ -0,0 +1,18 @@ +{ + "compilerOptions": { + "target": "ES2022", + "lib": ["ES2022"], + "module": "CommonJS", + "moduleResolution": "node", + "strict": true, + "esModuleInterop": true, + "allowSyntheticDefaultImports": true, + "skipLibCheck": true, + "outDir": "dist", + "rootDir": "src", + "declaration": true, + "sourceMap": true + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist"] +} diff --git a/skill/package-lock.json b/skill/package-lock.json new file mode 100644 index 00000000..ff78e035 --- /dev/null +++ b/skill/package-lock.json @@ -0,0 +1,1269 @@ +{ + "name": "llm-wiki-nashsu-skill", + "version": "0.4.6-skill.1", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "llm-wiki-nashsu-skill", + "version": "0.4.6-skill.1", + "dependencies": { + "@modelcontextprotocol/sdk": "^1.0.0", + "graphology": "^0.25.4", + "graphology-communities-louvain": "^2.0.0" + }, + "bin": { + "llm-wiki": "dist/cli.js", + "llm-wiki-mcp": "dist/mcp-server.js" + }, + "devDependencies": { + "@types/node": "^20.0.0", + "typescript": "^5.0.0" + } + }, + "node_modules/@hono/node-server": { + "version": "1.19.14", + "resolved": "https://registry.npmjs.org/@hono/node-server/-/node-server-1.19.14.tgz", + "integrity": "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw==", + "license": "MIT", + "engines": { + "node": ">=18.14.1" + }, + "peerDependencies": { + "hono": "^4" + } + }, + "node_modules/@modelcontextprotocol/sdk": { + "version": "1.29.0", + "resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.29.0.tgz", + "integrity": "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ==", + "license": "MIT", + "dependencies": { + "@hono/node-server": "^1.19.9", + "ajv": "^8.17.1", + "ajv-formats": "^3.0.1", + "content-type": "^1.0.5", + "cors": "^2.8.5", + "cross-spawn": "^7.0.5", + "eventsource": "^3.0.2", + "eventsource-parser": "^3.0.0", + "express": "^5.2.1", + "express-rate-limit": "^8.2.1", + "hono": "^4.11.4", + "jose": "^6.1.3", + "json-schema-typed": "^8.0.2", + "pkce-challenge": "^5.0.0", + "raw-body": "^3.0.0", + "zod": "^3.25 || ^4.0", + "zod-to-json-schema": "^3.25.1" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "@cfworker/json-schema": "^4.1.1", + "zod": "^3.25 || ^4.0" + }, + "peerDependenciesMeta": { + "@cfworker/json-schema": { + "optional": true + }, + "zod": { + "optional": false + } + } + }, + "node_modules/@types/node": { + "version": "20.19.39", + "resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.39.tgz", + "integrity": "sha512-orrrD74MBUyK8jOAD/r0+lfa1I2MO6I+vAkmAWzMYbCcgrN4lCrmK52gRFQq/JRxfYPfonkr4b0jcY7Olqdqbw==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": "~6.21.0" + } + }, + "node_modules/accepts": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz", + "integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==", + "license": "MIT", + "dependencies": { + "mime-types": "^3.0.0", + "negotiator": "^1.0.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/ajv": { + "version": "8.20.0", + "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", + "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", + "license": "MIT", + "dependencies": { + "fast-deep-equal": "^3.1.3", + "fast-uri": "^3.0.1", + "json-schema-traverse": "^1.0.0", + "require-from-string": "^2.0.2" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/epoberezkin" + } + }, + "node_modules/ajv-formats": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/ajv-formats/-/ajv-formats-3.0.1.tgz", + "integrity": "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==", + "license": "MIT", + "dependencies": { + "ajv": "^8.0.0" + }, + "peerDependencies": { + "ajv": "^8.0.0" + }, + "peerDependenciesMeta": { + "ajv": { + "optional": true + } + } + }, + "node_modules/body-parser": { + "version": "2.2.2", + "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.2.tgz", + "integrity": "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA==", + "license": "MIT", + "dependencies": { + "bytes": "^3.1.2", + "content-type": "^1.0.5", + "debug": "^4.4.3", + "http-errors": "^2.0.0", + "iconv-lite": "^0.7.0", + "on-finished": "^2.4.1", + "qs": "^6.14.1", + "raw-body": "^3.0.1", + "type-is": "^2.0.1" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/bytes": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", + "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/call-bind-apply-helpers": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", + "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/call-bound": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz", + "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "get-intrinsic": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/content-disposition": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.1.0.tgz", + "integrity": "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/content-type": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz", + "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", + "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie-signature": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz", + "integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==", + "license": "MIT", + "engines": { + "node": ">=6.6.0" + } + }, + "node_modules/cors": { + "version": "2.8.6", + "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.6.tgz", + "integrity": "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==", + "license": "MIT", + "dependencies": { + "object-assign": "^4", + "vary": "^1" + }, + "engines": { + "node": ">= 0.10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/cross-spawn": { + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", + "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", + "license": "MIT", + "dependencies": { + "path-key": "^3.1.0", + "shebang-command": "^2.0.0", + "which": "^2.0.1" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/depd": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", + "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/dunder-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", + "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.1", + "es-errors": "^1.3.0", + "gopd": "^1.2.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/ee-first": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", + "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==", + "license": "MIT" + }, + "node_modules/encodeurl": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", + "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-object-atoms": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz", + "integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/escape-html": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", + "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", + "license": "MIT" + }, + "node_modules/etag": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", + "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/events": { + "version": "3.3.0", + "resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz", + "integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==", + "license": "MIT", + "engines": { + "node": ">=0.8.x" + } + }, + "node_modules/eventsource": { + "version": "3.0.7", + "resolved": "https://registry.npmjs.org/eventsource/-/eventsource-3.0.7.tgz", + "integrity": "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA==", + "license": "MIT", + "dependencies": { + "eventsource-parser": "^3.0.1" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/eventsource-parser": { + "version": "3.0.8", + "resolved": "https://registry.npmjs.org/eventsource-parser/-/eventsource-parser-3.0.8.tgz", + "integrity": "sha512-70QWGkr4snxr0OXLRWsFLeRBIRPuQOvt4s8QYjmUlmlkyTZkRqS7EDVRZtzU3TiyDbXSzaOeF0XUKy8PchzukQ==", + "license": "MIT", + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/express": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", + "integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==", + "license": "MIT", + "dependencies": { + "accepts": "^2.0.0", + "body-parser": "^2.2.1", + "content-disposition": "^1.0.0", + "content-type": "^1.0.5", + "cookie": "^0.7.1", + "cookie-signature": "^1.2.1", + "debug": "^4.4.0", + "depd": "^2.0.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "finalhandler": "^2.1.0", + "fresh": "^2.0.0", + "http-errors": "^2.0.0", + "merge-descriptors": "^2.0.0", + "mime-types": "^3.0.0", + "on-finished": "^2.4.1", + "once": "^1.4.0", + "parseurl": "^1.3.3", + "proxy-addr": "^2.0.7", + "qs": "^6.14.0", + "range-parser": "^1.2.1", + "router": "^2.2.0", + "send": "^1.1.0", + "serve-static": "^2.2.0", + "statuses": "^2.0.1", + "type-is": "^2.0.1", + "vary": "^1.1.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/express-rate-limit": { + "version": "8.4.1", + "resolved": "https://registry.npmjs.org/express-rate-limit/-/express-rate-limit-8.4.1.tgz", + "integrity": "sha512-NGVYwQSAyEQgzxX1iCM978PP9AdO/hW93gMcF6ZwQCm+rFvLsBH6w4xcXWTcliS8La5EPRN3p9wzItqBwJrfNw==", + "license": "MIT", + "dependencies": { + "ip-address": "10.1.0" + }, + "engines": { + "node": ">= 16" + }, + "funding": { + "url": "https://github.com/sponsors/express-rate-limit" + }, + "peerDependencies": { + "express": ">= 4.11" + } + }, + "node_modules/fast-deep-equal": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz", + "integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==", + "license": "MIT" + }, + "node_modules/fast-uri": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.0.tgz", + "integrity": "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "license": "BSD-3-Clause" + }, + "node_modules/finalhandler": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz", + "integrity": "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "on-finished": "^2.4.1", + "parseurl": "^1.3.3", + "statuses": "^2.0.1" + }, + "engines": { + "node": ">= 18.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/forwarded": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", + "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/fresh": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz", + "integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/function-bind": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", + "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-intrinsic": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", + "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "es-define-property": "^1.0.1", + "es-errors": "^1.3.0", + "es-object-atoms": "^1.1.1", + "function-bind": "^1.1.2", + "get-proto": "^1.0.1", + "gopd": "^1.2.0", + "has-symbols": "^1.1.0", + "hasown": "^2.0.2", + "math-intrinsics": "^1.1.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", + "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", + "license": "MIT", + "dependencies": { + "dunder-proto": "^1.0.1", + "es-object-atoms": "^1.0.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/graphology": { + "version": "0.25.4", + "resolved": "https://registry.npmjs.org/graphology/-/graphology-0.25.4.tgz", + "integrity": "sha512-33g0Ol9nkWdD6ulw687viS8YJQBxqG5LWII6FI6nul0pq6iM2t5EKquOTFDbyTblRB3O9I+7KX4xI8u5ffekAQ==", + "license": "MIT", + "dependencies": { + "events": "^3.3.0", + "obliterator": "^2.0.2" + }, + "peerDependencies": { + "graphology-types": ">=0.24.0" + } + }, + "node_modules/graphology-communities-louvain": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/graphology-communities-louvain/-/graphology-communities-louvain-2.0.2.tgz", + "integrity": "sha512-zt+2hHVPYxjEquyecxWXoUoIuN/UvYzsvI7boDdMNz0rRvpESQ7+e+Ejv6wK7AThycbZXuQ6DkG8NPMCq6XwoA==", + "license": "MIT", + "dependencies": { + "graphology-indices": "^0.17.0", + "graphology-utils": "^2.4.4", + "mnemonist": "^0.39.0", + "pandemonium": "^2.4.1" + }, + "peerDependencies": { + "graphology-types": ">=0.19.0" + } + }, + "node_modules/graphology-indices": { + "version": "0.17.0", + "resolved": "https://registry.npmjs.org/graphology-indices/-/graphology-indices-0.17.0.tgz", + "integrity": "sha512-A7RXuKQvdqSWOpn7ZVQo4S33O0vCfPBnUSf7FwE0zNCasqwZVUaCXePuWo5HBpWw68KJcwObZDHpFk6HKH6MYQ==", + "license": "MIT", + "dependencies": { + "graphology-utils": "^2.4.2", + "mnemonist": "^0.39.0" + }, + "peerDependencies": { + "graphology-types": ">=0.20.0" + } + }, + "node_modules/graphology-types": { + "version": "0.24.8", + "resolved": "https://registry.npmjs.org/graphology-types/-/graphology-types-0.24.8.tgz", + "integrity": "sha512-hDRKYXa8TsoZHjgEaysSRyPdT6uB78Ci8WnjgbStlQysz7xR52PInxNsmnB7IBOM1BhikxkNyCVEFgmPKnpx3Q==", + "license": "MIT", + "peer": true + }, + "node_modules/graphology-utils": { + "version": "2.5.2", + "resolved": "https://registry.npmjs.org/graphology-utils/-/graphology-utils-2.5.2.tgz", + "integrity": "sha512-ckHg8MXrXJkOARk56ZaSCM1g1Wihe2d6iTmz1enGOz4W/l831MBCKSayeFQfowgF8wd+PQ4rlch/56Vs/VZLDQ==", + "license": "MIT", + "peerDependencies": { + "graphology-types": ">=0.23.0" + } + }, + "node_modules/has-symbols": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", + "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/hasown": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.3.tgz", + "integrity": "sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg==", + "license": "MIT", + "dependencies": { + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/hono": { + "version": "4.12.16", + "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.16.tgz", + "integrity": "sha512-jN0ZewiNAWSe5khM3EyCmBb250+b40wWbwNILNfEvq84VREWwOIkuUsFONk/3i3nqkz7Oe1PcpM2mwQEK2L9Kg==", + "license": "MIT", + "engines": { + "node": ">=16.9.0" + } + }, + "node_modules/http-errors": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", + "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==", + "license": "MIT", + "dependencies": { + "depd": "~2.0.0", + "inherits": "~2.0.4", + "setprototypeof": "~1.2.0", + "statuses": "~2.0.2", + "toidentifier": "~1.0.1" + }, + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/iconv-lite": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.2.tgz", + "integrity": "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw==", + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", + "license": "ISC" + }, + "node_modules/ip-address": { + "version": "10.1.0", + "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.1.0.tgz", + "integrity": "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q==", + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, + "node_modules/ipaddr.js": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", + "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==", + "license": "MIT", + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/is-promise": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz", + "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==", + "license": "MIT" + }, + "node_modules/isexe": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", + "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", + "license": "ISC" + }, + "node_modules/jose": { + "version": "6.2.3", + "resolved": "https://registry.npmjs.org/jose/-/jose-6.2.3.tgz", + "integrity": "sha512-YYVDInQKFJfR/xa3ojUTl8c2KoTwiL1R5Wg9YCydwH0x0B9grbzlg5HC7mMjCtUJjbQ/YnGEZIhI5tCgfTb4Hw==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/panva" + } + }, + "node_modules/json-schema-traverse": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", + "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", + "license": "MIT" + }, + "node_modules/json-schema-typed": { + "version": "8.0.2", + "resolved": "https://registry.npmjs.org/json-schema-typed/-/json-schema-typed-8.0.2.tgz", + "integrity": "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA==", + "license": "BSD-2-Clause" + }, + "node_modules/math-intrinsics": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", + "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/media-typer": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.0.tgz", + "integrity": "sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/merge-descriptors": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz", + "integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/mime-db": { + "version": "1.54.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", + "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime-types": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", + "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", + "license": "MIT", + "dependencies": { + "mime-db": "^1.54.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/mnemonist": { + "version": "0.39.8", + "resolved": "https://registry.npmjs.org/mnemonist/-/mnemonist-0.39.8.tgz", + "integrity": "sha512-vyWo2K3fjrUw8YeeZ1zF0fy6Mu59RHokURlld8ymdUPjMlD9EC9ov1/YPqTgqRvUN9nTr3Gqfz29LYAmu0PHPQ==", + "license": "MIT", + "dependencies": { + "obliterator": "^2.0.1" + } + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/negotiator": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.0.0.tgz", + "integrity": "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/object-assign": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", + "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/object-inspect": { + "version": "1.13.4", + "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", + "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/obliterator": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/obliterator/-/obliterator-2.0.5.tgz", + "integrity": "sha512-42CPE9AhahZRsMNslczq0ctAEtqk8Eka26QofnqC346BZdHDySk3LWka23LI7ULIw11NmltpiLagIq8gBozxTw==", + "license": "MIT" + }, + "node_modules/on-finished": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", + "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", + "license": "MIT", + "dependencies": { + "ee-first": "1.1.1" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", + "license": "ISC", + "dependencies": { + "wrappy": "1" + } + }, + "node_modules/pandemonium": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/pandemonium/-/pandemonium-2.4.1.tgz", + "integrity": "sha512-wRqjisUyiUfXowgm7MFH2rwJzKIr20rca5FsHXCMNm1W5YPP1hCtrZfgmQ62kP7OZ7Xt+cR858aB28lu5NX55g==", + "license": "MIT", + "dependencies": { + "mnemonist": "^0.39.2" + } + }, + "node_modules/parseurl": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", + "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/path-key": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", + "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/path-to-regexp": { + "version": "8.4.2", + "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.4.2.tgz", + "integrity": "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA==", + "license": "MIT", + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/pkce-challenge": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/pkce-challenge/-/pkce-challenge-5.0.1.tgz", + "integrity": "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ==", + "license": "MIT", + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/proxy-addr": { + "version": "2.0.7", + "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", + "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==", + "license": "MIT", + "dependencies": { + "forwarded": "0.2.0", + "ipaddr.js": "1.9.1" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/qs": { + "version": "6.15.1", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.1.tgz", + "integrity": "sha512-6YHEFRL9mfgcAvql/XhwTvf5jKcOiiupt2FiJxHkiX1z4j7WL8J/jRHYLluORvc1XxB5rV20KoeK00gVJamspg==", + "license": "BSD-3-Clause", + "dependencies": { + "side-channel": "^1.1.0" + }, + "engines": { + "node": ">=0.6" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/range-parser": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", + "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/raw-body": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.2.tgz", + "integrity": "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==", + "license": "MIT", + "dependencies": { + "bytes": "~3.1.2", + "http-errors": "~2.0.1", + "iconv-lite": "~0.7.0", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/require-from-string": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", + "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/router": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz", + "integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.0", + "depd": "^2.0.0", + "is-promise": "^4.0.0", + "parseurl": "^1.3.3", + "path-to-regexp": "^8.0.0" + }, + "engines": { + "node": ">= 18" + } + }, + "node_modules/safer-buffer": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", + "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", + "license": "MIT" + }, + "node_modules/send": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz", + "integrity": "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.3", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "fresh": "^2.0.0", + "http-errors": "^2.0.1", + "mime-types": "^3.0.2", + "ms": "^2.1.3", + "on-finished": "^2.4.1", + "range-parser": "^1.2.1", + "statuses": "^2.0.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/serve-static": { + "version": "2.2.1", + "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz", + "integrity": "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==", + "license": "MIT", + "dependencies": { + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "parseurl": "^1.3.3", + "send": "^1.2.0" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/setprototypeof": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", + "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==", + "license": "ISC" + }, + "node_modules/shebang-command": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", + "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", + "license": "MIT", + "dependencies": { + "shebang-regex": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/shebang-regex": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", + "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/side-channel": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz", + "integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.3", + "side-channel-list": "^1.0.0", + "side-channel-map": "^1.0.1", + "side-channel-weakmap": "^1.0.2" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-list": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz", + "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-map": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz", + "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-weakmap": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz", + "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3", + "side-channel-map": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/statuses": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", + "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/toidentifier": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", + "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==", + "license": "MIT", + "engines": { + "node": ">=0.6" + } + }, + "node_modules/type-is": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.0.1.tgz", + "integrity": "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw==", + "license": "MIT", + "dependencies": { + "content-type": "^1.0.5", + "media-typer": "^1.1.0", + "mime-types": "^3.0.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/typescript": { + "version": "5.9.3", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", + "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/undici-types": { + "version": "6.21.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", + "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/unpipe": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", + "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/vary": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", + "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/which": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", + "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", + "license": "ISC", + "dependencies": { + "isexe": "^2.0.0" + }, + "bin": { + "node-which": "bin/node-which" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/wrappy": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", + "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", + "license": "ISC" + }, + "node_modules/zod": { + "version": "4.4.2", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.4.2.tgz", + "integrity": "sha512-IynmDyxsEsb9RKzO3J9+4SxXnl2FTFSzNBaKKaMV6tsSk0rw9gYw9gs+JFCq/qk2LCZ78KDwyj+Z289TijSkUw==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } + }, + "node_modules/zod-to-json-schema": { + "version": "3.25.2", + "resolved": "https://registry.npmjs.org/zod-to-json-schema/-/zod-to-json-schema-3.25.2.tgz", + "integrity": "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA==", + "license": "ISC", + "peerDependencies": { + "zod": "^3.25.28 || ^4" + } + } + } +} diff --git a/skill/package.json b/skill/package.json index 411610b1..8b8f8da7 100644 --- a/skill/package.json +++ b/skill/package.json @@ -1,20 +1,25 @@ { "name": "llm-wiki-nashsu-skill", - "version": "0.4.6-skill", - "description": "nashsu/llm_wiki backend extracted as Node.js CLI skill (no GUI)", + "version": "0.4.6-skill.1", + "description": "nashsu/llm_wiki backend as Node.js CLI + MCP server (no Tauri/GUI)", "main": "dist/cli.js", + "bin": { + "llm-wiki": "dist/cli.js", + "llm-wiki-mcp": "dist/mcp-server.js" + }, "scripts": { - "build": "tsc -p tsconfig.skill.json", - "dev": "ts-node --project tsconfig.skill.json src/cli.ts", - "cli": "node dist/cli.js" + "build": "tsc -p tsconfig.json", + "typecheck": "tsc -p tsconfig.json --noEmit", + "start": "node dist/cli.js", + "mcp": "node dist/mcp-server.js" }, "dependencies": { "graphology": "^0.25.4", - "graphology-communities-louvain": "^2.0.0" + "graphology-communities-louvain": "^2.0.0", + "@modelcontextprotocol/sdk": "^1.0.0" }, "devDependencies": { "@types/node": "^20.0.0", - "ts-node": "^10.9.0", "typescript": "^5.0.0" } } diff --git a/skill/src/cli.ts b/skill/src/cli.ts index b21d5084..991097ab 100644 --- a/skill/src/cli.ts +++ b/skill/src/cli.ts @@ -1,305 +1,159 @@ #!/usr/bin/env node /** - * llm-wiki-nashsu CLI - * - * Entry point for the nashsu/llm_wiki backend skill (no GUI). - * Replaces Tauri IPC with Node.js fs, React stores with module state. - * - * Usage: - * node cli.js [args...] + * llm-wiki CLI — nashsu/llm_wiki backend as a standalone Node.js tool * * Commands: - * init [topic] [lang] - * graph [--output=graph-data.json] - * insights - * search [--limit=20] - * status - * lint (requires LLM) - * ingest (requires LLM) - * deep-research (requires LLM + search API) - * sweep-reviews (requires LLM) + * graph Build graph JSON (nodes, edges, communities) + * insights Show surprising connections + knowledge gaps + * search BM25+RRF search across wiki pages + * status Page count statistics by type + * init Create wiki directory structure + * lint Structural lint (broken links, orphans) */ - -import * as fs from "fs" import * as path from "path" -// Import core library modules (Tauri deps patched via tsconfig paths) -import { buildWikiGraph } from "../../src/lib/wiki-graph" -import { findSurprisingConnections, detectKnowledgeGaps } from "../../src/lib/graph-insights" -import { searchWiki } from "../../src/lib/search" -import { configureWikiStore } from "./stores-node" - -// --------------------------------------------------------------------------- -// Main dispatch -// --------------------------------------------------------------------------- +import * as fs from "fs" +import { buildWikiGraph } from "./lib/wiki-graph" +import { findSurprisingConnections, detectKnowledgeGaps } from "./lib/graph-insights" +import { searchWiki } from "./lib/search" async function main() { - const [, , command, wikiRoot, ...rest] = process.argv - - if (!command || command === "--help" || command === "-h") { - printHelp() - process.exit(0) - } - - if (command === "--version") { - console.log("0.4.6-skill") - process.exit(0) - } - - if (!wikiRoot) { - console.error("Error: wiki_root is required") - process.exit(1) - } - - const resolvedRoot = path.resolve(wikiRoot) - - // Configure store state (replaces React store initialization) - configureWikiStore({ - projectPath: resolvedRoot, - llmConfig: { - provider: "openai", - apiKey: process.env.OPENAI_API_KEY ?? "", - model: process.env.LLM_MODEL ?? "gpt-4o", - baseUrl: process.env.OPENAI_API_BASE, - }, - embeddingConfig: { - enabled: !!(process.env.EMBEDDING_MODEL), - model: process.env.EMBEDDING_MODEL ?? "", - apiBase: process.env.EMBEDDING_API_BASE, - }, - }) - + const [, , command, ...args] = process.argv + if (!command || command === "help" || command === "--help") { usage(); return } switch (command) { - case "init": - await cmdInit(resolvedRoot, rest[0] ?? "My Knowledge Base", rest[1] ?? "en") - break - - case "graph": - await cmdGraph(resolvedRoot, rest) - break - - case "insights": - await cmdInsights(resolvedRoot) - break - - case "search": - if (!rest[0]) { console.error("Error: query is required"); process.exit(1) } - await cmdSearch(resolvedRoot, rest[0], rest) - break - - case "status": - await cmdStatus(resolvedRoot) - break - + case "graph": return cmdGraph(args) + case "insights": return cmdInsights(args) + case "search": return cmdSearch(args) + case "status": return cmdStatus(args) + case "init": return cmdInit(args) + case "lint": return cmdLint(args) default: console.error(`Unknown command: ${command}`) - console.error("Run with --help for usage") + usage() process.exit(1) } } -// --------------------------------------------------------------------------- -// Commands -// --------------------------------------------------------------------------- - -async function cmdInit(wikiRoot: string, topic: string, lang: string) { - const dirs = ["wiki/entities", "wiki/concepts", "wiki/sources", "wiki/queries", "raw"] - for (const d of dirs) { - fs.mkdirSync(path.join(wikiRoot, d), { recursive: true }) - } - - const indexContent = `--- -type: overview -title: "${topic}" -lang: ${lang} -created: ${new Date().toISOString().slice(0, 10)} ---- - -# ${topic} - -> 这是一个 llm-wiki-nashsu 知识库。 - -## 实体 - -## 概念 - -## 素材来源 -` - const indexPath = path.join(wikiRoot, "wiki", "index.md") - if (!fs.existsSync(indexPath)) { - fs.writeFileSync(indexPath, indexContent, "utf-8") - } +function usage() { + console.log(` +llm-wiki — nashsu/llm_wiki backend skill (no Tauri/GUI) - const configPath = path.join(wikiRoot, ".wiki-config.json") - if (!fs.existsSync(configPath)) { - fs.writeFileSync(configPath, JSON.stringify({ topic, lang, version: "0.4.6-skill" }, null, 2)) - } +USAGE: + llm-wiki [options] - console.log(JSON.stringify({ status: "success", wiki_root: wikiRoot, topic, lang })) +COMMANDS: + graph Build knowledge graph (outputs JSON) + insights Show surprising connections + knowledge gaps + search Keyword search (BM25+RRF) + status Page count and type breakdown + init Initialize wiki directory structure + lint Check for broken links and orphan pages + +ENV VARS: + SKILL_VERBOSE=1 Enable verbose activity logging + WIKI_PATH Default project path for MCP server + +EXAMPLES: + llm-wiki graph ./my-project + llm-wiki search ./my-project "attention mechanism" + llm-wiki insights ./my-project +`.trim()) } -async function cmdGraph(wikiRoot: string, args: string[]) { - const outputArg = args.find((a) => a.startsWith("--output=")) - const outputPath = outputArg - ? outputArg.replace("--output=", "") - : path.join(wikiRoot, "graph-data.json") - - console.error(`[graph] Building wiki graph for: ${wikiRoot}`) - const { nodes, edges, communities } = await buildWikiGraph(wikiRoot) - console.error(`[graph] Found ${nodes.length} nodes, ${edges.length} edges, ${communities.length} communities`) - - const graphData = { - nodes, - edges, - communities, - generated: new Date().toISOString(), - } - - fs.mkdirSync(path.dirname(outputPath), { recursive: true }) - fs.writeFileSync(outputPath, JSON.stringify(graphData, null, 2)) - console.log(JSON.stringify({ status: "success", output: outputPath, ...graphData })) +async function cmdGraph(args: string[]) { + const wikiRoot = args[0] + if (!wikiRoot) { console.error("Usage: graph "); process.exit(1) } + const projectPath = path.resolve(wikiRoot) + console.error(`Building graph: ${projectPath}`) + const result = await buildWikiGraph(projectPath) + process.stdout.write(JSON.stringify(result, null, 2) + "\n") + console.error(`\n✓ ${result.nodes.length} nodes, ${result.edges.length} edges, ${result.communities.length} communities`) } -async function cmdInsights(wikiRoot: string) { - console.error(`[insights] Analyzing graph for: ${wikiRoot}`) - const { nodes, edges, communities } = await buildWikiGraph(wikiRoot) - - const surprising = findSurprisingConnections(nodes, edges, communities, 10) - const gaps = detectKnowledgeGaps(nodes, edges, communities, 10) - - // Format as markdown report - const lines: string[] = [ - "# 图谱洞察报告", - "", - `> 生成时间：${new Date().toISOString().slice(0, 16)}`, - `> 节点总数：${nodes.length}，边总数：${edges.length}，社区总数：${communities.length}`, - "", - "---", - "", - "## 惊人连接（Surprising Connections）", - "", - ] - - if (surprising.length === 0) { - lines.push("_暂无惊人连接。随着知识库增长，跨社区连接会在这里显示。_") - } else { - for (const conn of surprising) { - lines.push(`### ${conn.source.label} ↔ ${conn.target.label}`) - lines.push(`- **惊喜评分**：${conn.score}`) - lines.push(`- **原因**：${conn.reasons.join("；")}`) - lines.push("") - } - } - - lines.push("---", "", "## 知识缺口（Knowledge Gaps）", "") - - if (gaps.length === 0) { - lines.push("_暂无知识缺口。知识库连接良好！_") - } else { - for (const gap of gaps) { - const typeLabel = { - "isolated-node": "🔴 孤立节点", - "sparse-community": "🟡 稀疏社区", - "bridge-node": "🔵 桥节点", - }[gap.type] ?? gap.type - lines.push(`### ${typeLabel}：${gap.title}`) - lines.push(`- **描述**：${gap.description}`) - lines.push(`- **建议**：${gap.suggestion}`) - lines.push("") - } +async function cmdInsights(args: string[]) { + const wikiRoot = args[0] + if (!wikiRoot) { console.error("Usage: insights "); process.exit(1) } + const projectPath = path.resolve(wikiRoot) + const { nodes, edges, communities } = await buildWikiGraph(projectPath) + const connections = findSurprisingConnections(nodes, edges, communities, 10) + const gaps = detectKnowledgeGaps(nodes, edges, communities, 8) + const lines: string[] = ["# Wiki Insights\n", "## Surprising Connections\n"] + if (connections.length === 0) lines.push("_No surprising connections found (need more linked pages)._\n") + for (const c of connections) { + lines.push(`### ${c.source.label} ↔ ${c.target.label}`) + lines.push(`- **Score**: ${c.score} | **Why**: ${c.reasons.join(", ")}\n`) } - - lines.push("---", "", "## 社区凝聚度", "") - for (const comm of communities) { - const warning = comm.cohesion < 0.15 ? " ⚠️ 低凝聚度" : "" - lines.push( - `- **社区 ${comm.id}**（${comm.nodeCount} 个页面）：` + - `凝聚度 ${comm.cohesion.toFixed(2)}${warning}` + - ` — ${comm.topNodes.slice(0, 3).join("、")}`, - ) + lines.push("## Knowledge Gaps\n") + if (gaps.length === 0) lines.push("_No knowledge gaps detected._\n") + for (const g of gaps) { + lines.push(`### ${g.title}`) + lines.push(`**Type**: ${g.type}\n${g.description}`) + lines.push(`💡 ${g.suggestion}\n`) } - - const report = lines.join("\n") - console.log(report) -} - -async function cmdSearch(wikiRoot: string, query: string, args: string[]) { - const limitArg = args.find((a) => a.startsWith("--limit=")) - const _limit = limitArg ? parseInt(limitArg.replace("--limit=", ""), 10) : 20 - - console.error(`[search] Searching "${query}" in: ${wikiRoot}`) - const results = await searchWiki(wikiRoot, query) - - console.log(JSON.stringify({ query, results, total: results.length })) + process.stdout.write(lines.join("\n")) } -async function cmdStatus(wikiRoot: string) { - const wikiDir = path.join(wikiRoot, "wiki") - if (!fs.existsSync(wikiDir)) { - console.log(JSON.stringify({ status: "not_initialized", wiki_root: wikiRoot })) - return +async function cmdSearch(args: string[]) { + const [wikiRoot, ...queryParts] = args + const query = queryParts.join(" ") + if (!wikiRoot || !query) { console.error("Usage: search "); process.exit(1) } + const projectPath = path.resolve(wikiRoot) + const results = await searchWiki(projectPath, query) + if (results.length === 0) { console.log(`No results for: "${query}"`); return } + const lines: string[] = [`# Search: "${query}"\n`] + for (const r of results) { + const relPath = path.relative(projectPath, r.path) + lines.push(`## ${r.title}`) + lines.push(`**Path**: ${relPath} | **Score**: ${r.score.toFixed(4)}`) + lines.push(r.snippet + "\n") } + process.stdout.write(lines.join("\n")) +} - function countMd(dir: string): number { - if (!fs.existsSync(dir)) return 0 - return fs - .readdirSync(dir, { withFileTypes: true }) - .filter((e) => e.isFile() && e.name.endsWith(".md")) - .length +async function cmdStatus(args: string[]) { + const wikiRoot = args[0] + if (!wikiRoot) { console.error("Usage: status "); process.exit(1) } + const projectPath = path.resolve(wikiRoot) + const { nodes, communities } = await buildWikiGraph(projectPath) + const typeCounts: Record = {} + for (const n of nodes) typeCounts[n.type] = (typeCounts[n.type] ?? 0) + 1 + console.log(`Wiki: ${projectPath}\nTotal pages: ${nodes.length}\nCommunities: ${communities.length}`) + for (const [type, count] of Object.entries(typeCounts).sort((a, b) => b[1] - a[1])) { + console.log(` ${type}: ${count}`) } +} - const stats = { - status: "ready", - wiki_root: wikiRoot, - entities: countMd(path.join(wikiDir, "entities")), - concepts: countMd(path.join(wikiDir, "concepts")), - sources: countMd(path.join(wikiDir, "sources")), - queries: countMd(path.join(wikiDir, "queries")), - total: 0, +async function cmdInit(args: string[]) { + const wikiRoot = args[0] + if (!wikiRoot) { console.error("Usage: init "); process.exit(1) } + const projectPath = path.resolve(wikiRoot) + const dirs = ["wiki/entities", "wiki/concepts", "wiki/sources", "wiki/synthesis", "wiki/queries"] + for (const dir of dirs) fs.mkdirSync(path.join(projectPath, dir), { recursive: true }) + const indexPath = path.join(projectPath, "wiki/index.md") + if (!fs.existsSync(indexPath)) { + fs.writeFileSync(indexPath, ["---", "title: Index", "type: overview", "---", "", "# Knowledge Base", "", "Welcome to your wiki.", ""].join("\n")) } - stats.total = stats.entities + stats.concepts + stats.sources + stats.queries - - console.log(JSON.stringify(stats)) + console.log(`✓ Initialized wiki at: ${projectPath}`) } -// --------------------------------------------------------------------------- -// Help -// --------------------------------------------------------------------------- - -function printHelp() { - console.log(` -llm-wiki-nashsu v0.4.6-skill -nashsu/llm_wiki backend extracted as Node.js CLI skill (no GUI) - -USAGE: - node cli.js [options] - -COMMANDS: - init [topic] [lang] Initialize knowledge base - graph [--output=] Build graph data (JSON) - insights Graph insights (markdown) - search [--limit=N] Hybrid BM25+vector search - status Knowledge base statistics - - (Requires LLM config via env vars:) - ingest Ingest document - sweep-reviews Process review queue - deep-research Deep research via web search - -ENVIRONMENT VARIABLES: - OPENAI_API_KEY LLM API key - OPENAI_API_BASE Custom LLM endpoint (Ollama/proxy) - LLM_MODEL Model name (default: gpt-4o) - EMBEDDING_API_BASE Embedding endpoint (enables vector search) - EMBEDDING_MODEL Embedding model - TAVILY_API_KEY Web search API key (for deep-research) -`) +async function cmdLint(args: string[]) { + const wikiRoot = args[0] + if (!wikiRoot) { console.error("Usage: lint "); process.exit(1) } + const projectPath = path.resolve(wikiRoot) + const { nodes, edges } = await buildWikiGraph(projectPath) + if (nodes.length === 0) { console.log("No wiki pages found."); return } + const edgeTargets = new Set(edges.map((e) => e.target)) + const edgeSources = new Set(edges.map((e) => e.source)) + const allLinked = new Set([...edgeTargets, ...edgeSources]) + let issues = 0 + for (const n of nodes) { + if (n.id === "index" || n.id === "log" || n.id === "overview") continue + if (!allLinked.has(n.id)) { console.log(`[orphan] ${n.label} (${n.id}.md)`); issues++ } + else if (n.linkCount <= 1) { console.log(`[isolated] ${n.label} — ${n.linkCount} link(s)`); issues++ } + } + console.log(`\n✓ ${nodes.length} pages checked — ${issues} issue(s)`) } -// --------------------------------------------------------------------------- -// Run -// --------------------------------------------------------------------------- - main().catch((err) => { - console.error("Error:", err.message) + console.error("Error:", err instanceof Error ? err.message : err) process.exit(1) }) diff --git a/skill/src/lib/graph-insights.ts b/skill/src/lib/graph-insights.ts new file mode 100644 index 00000000..56fa212f --- /dev/null +++ b/skill/src/lib/graph-insights.ts @@ -0,0 +1,150 @@ +import type { GraphNode, GraphEdge, CommunityInfo } from "./wiki-graph" + +export interface SurprisingConnection { + source: GraphNode + target: GraphNode + score: number + reasons: string[] + key: string +} + +export interface KnowledgeGap { + type: "isolated-node" | "sparse-community" | "bridge-node" + title: string + description: string + nodeIds: string[] + suggestion: string +} + +export function findSurprisingConnections( + nodes: GraphNode[], + edges: GraphEdge[], + _communities: CommunityInfo[], + limit: number = 5, +): SurprisingConnection[] { + const nodeMap = new Map(nodes.map((n) => [n.id, n])) + const degreeMap = new Map(nodes.map((n) => [n.id, n.linkCount])) + const maxDegree = Math.max(...nodes.map((n) => n.linkCount), 1) + const STRUCTURAL_IDS = new Set(["index", "log", "overview"]) + const scored: SurprisingConnection[] = [] + + for (const edge of edges) { + const source = nodeMap.get(edge.source) + const target = nodeMap.get(edge.target) + if (!source || !target) continue + if (STRUCTURAL_IDS.has(source.id) || STRUCTURAL_IDS.has(target.id)) continue + + let score = 0 + const reasons: string[] = [] + + if (source.community !== target.community) { + score += 3 + reasons.push("crosses community boundary") + } + if (source.type !== target.type) { + const distantPairs = new Set([ + "source-concept", "concept-source", "source-synthesis", "synthesis-source", + "query-entity", "entity-query", + ]) + if (distantPairs.has(`${source.type}-${target.type}`)) { + score += 2 + reasons.push(`connects ${source.type} to ${target.type}`) + } else { + score += 1 + reasons.push("different types") + } + } + const sourceDeg = degreeMap.get(source.id) ?? 0 + const targetDeg = degreeMap.get(target.id) ?? 0 + const minDeg = Math.min(sourceDeg, targetDeg) + const maxDeg = Math.max(sourceDeg, targetDeg) + if (minDeg <= 2 && maxDeg >= maxDegree * 0.5) { + score += 2 + reasons.push("peripheral node links to hub") + } + if (edge.weight < 2 && edge.weight > 0) { + score += 1 + reasons.push("weak but present connection") + } + if (score >= 3 && reasons.length > 0) { + const key = [source.id, target.id].sort().join(":::") + scored.push({ source, target, score, reasons, key }) + } + } + + scored.sort((a, b) => b.score - a.score) + return scored.slice(0, limit) +} + +export function detectKnowledgeGaps( + nodes: GraphNode[], + edges: GraphEdge[], + communities: CommunityInfo[], + limit: number = 8, +): KnowledgeGap[] { + const gaps: KnowledgeGap[] = [] + const nodeMap = new Map(nodes.map((n) => [n.id, n])) + + // 1. Isolated nodes (degree ≤ 1) + const isolatedNodes = nodes.filter( + (n) => n.linkCount <= 1 && n.type !== "overview" && n.id !== "index" && n.id !== "log", + ) + if (isolatedNodes.length > 0) { + const topIsolated = isolatedNodes.slice(0, 5) + gaps.push({ + type: "isolated-node", + title: `${isolatedNodes.length} isolated page${isolatedNodes.length > 1 ? "s" : ""}`, + description: topIsolated.map((n) => n.label).join(", ") + + (isolatedNodes.length > 5 ? ` and ${isolatedNodes.length - 5} more` : ""), + nodeIds: isolatedNodes.map((n) => n.id), + suggestion: "These pages have few or no connections. Consider adding [[wikilinks]] to related pages.", + }) + } + + // 2. Sparse communities (low cohesion) + for (const comm of communities) { + if (comm.cohesion < 0.15 && comm.nodeCount >= 3) { + gaps.push({ + type: "sparse-community", + title: `Sparse cluster: ${comm.topNodes[0] ?? `Community ${comm.id}`}`, + description: `${comm.nodeCount} pages with cohesion ${comm.cohesion.toFixed(2)} — internal connections are weak.`, + nodeIds: nodes.filter((n) => n.community === comm.id).map((n) => n.id), + suggestion: "This knowledge area lacks internal cross-references. Consider adding links between these pages.", + }) + } + } + + // 3. Bridge nodes (connected to multiple communities) + const communityNeighbors = new Map>() + for (const node of nodes) communityNeighbors.set(node.id, new Set()) + for (const edge of edges) { + const sourceNode = nodeMap.get(edge.source) + const targetNode = nodeMap.get(edge.target) + if (sourceNode && targetNode) { + communityNeighbors.get(edge.source)?.add(targetNode.community) + communityNeighbors.get(edge.target)?.add(sourceNode.community) + } + } + const STRUCTURAL_IDS = new Set(["index", "log", "overview"]) + const bridgeNodes = nodes + .filter((n) => { + if (STRUCTURAL_IDS.has(n.id)) return false + const neighborComms = communityNeighbors.get(n.id) + return neighborComms && neighborComms.size >= 3 + }) + .sort((a, b) => (communityNeighbors.get(b.id)?.size ?? 0) - (communityNeighbors.get(a.id)?.size ?? 0)) + .slice(0, 3) + + for (const bridge of bridgeNodes) { + const commCount = communityNeighbors.get(bridge.id)?.size ?? 0 + gaps.push({ + type: "bridge-node", + title: `Key bridge: ${bridge.label}`, + description: `Connects ${commCount} different knowledge clusters. This is a critical junction in your wiki.`, + nodeIds: [bridge.id], + suggestion: "This page bridges multiple knowledge areas. Ensure it's well-maintained and expanded.", + }) + } + + return gaps.slice(0, limit) +} diff --git a/skill/src/lib/graph-relevance.ts b/skill/src/lib/graph-relevance.ts new file mode 100644 index 00000000..db27e7b1 --- /dev/null +++ b/skill/src/lib/graph-relevance.ts @@ -0,0 +1,229 @@ +import { readFile, listDirectory } from "../shims/fs-node" +import type { FileNode } from "../types/wiki" +import { normalizePath } from "./path-utils" + +export interface RetrievalNode { + readonly id: string + readonly title: string + readonly type: string + readonly path: string + readonly sources: readonly string[] + readonly outLinks: ReadonlySet + readonly inLinks: ReadonlySet +} + +export interface RetrievalGraph { + readonly nodes: ReadonlyMap + readonly dataVersion: number +} + +const WIKILINK_REGEX = /\[\[([^\]|]+?)(?:\|[^\]]+?)?\]\]/g + +const WEIGHTS = { + directLink: 3.0, + sourceOverlap: 4.0, + commonNeighbor: 1.5, + typeAffinity: 1.0, +} as const + +const TYPE_AFFINITY: Record> = { + entity: { concept: 1.2, entity: 0.8, source: 1.0, synthesis: 1.0, query: 0.8 }, + concept: { entity: 1.2, concept: 0.8, source: 1.0, synthesis: 1.2, query: 1.0 }, + source: { entity: 1.0, concept: 1.0, source: 0.5, query: 0.8, synthesis: 1.0 }, + query: { concept: 1.0, entity: 0.8, synthesis: 1.0, source: 0.8, query: 0.5 }, + synthesis: { concept: 1.2, entity: 1.0, source: 1.0, query: 1.0, synthesis: 0.8 }, +} + +let cachedGraph: RetrievalGraph | null = null + +function flattenMdFiles(nodes: readonly FileNode[]): FileNode[] { + const files: FileNode[] = [] + for (const node of nodes) { + if (node.is_dir && node.children) { + files.push(...flattenMdFiles(node.children)) + } else if (!node.is_dir && node.name.endsWith(".md")) { + files.push(node) + } + } + return files +} + +function fileNameToId(fileName: string): string { + return fileName.replace(/\.md$/, "") +} + +function extractFrontmatter(content: string): { title: string; type: string; sources: string[] } { + const fmMatch = content.match(/^---\n([\s\S]*?)\n---/) + const fm = fmMatch ? fmMatch[1] : "" + const titleMatch = fm.match(/^title:\s*["']?(.+?)["']?\s*$/m) + const typeMatch = fm.match(/^type:\s*["']?(.+?)["']?\s*$/m) + const sources: string[] = [] + const sourcesBlockMatch = fm.match(/^sources:\s*\n((?:\s+-\s+.+\n?)*)/m) + if (sourcesBlockMatch) { + const lines = sourcesBlockMatch[1].split("\n") + for (const line of lines) { + const itemMatch = line.match(/^\s+-\s+["']?(.+?)["']?\s*$/) + if (itemMatch) sources.push(itemMatch[1]) + } + } else { + const inlineMatch = fm.match(/^sources:\s*\[([^\]]*)\]/m) + if (inlineMatch) { + const items = inlineMatch[1].split(",") + for (const item of items) { + const trimmed = item.trim().replace(/^["']|["']$/g, "") + if (trimmed) sources.push(trimmed) + } + } + } + let title = titleMatch ? titleMatch[1].trim() : "" + if (!title) { + const headingMatch = content.match(/^#\s+(.+)$/m) + title = headingMatch ? headingMatch[1].trim() : "" + } + return { title, type: typeMatch ? typeMatch[1].trim().toLowerCase() : "other", sources } +} + +function extractWikilinks(content: string): string[] { + const links: string[] = [] + const regex = new RegExp(WIKILINK_REGEX.source, "g") + let match: RegExpExecArray | null + while ((match = regex.exec(content)) !== null) links.push(match[1].trim()) + return links +} + +function resolveTarget(raw: string, nodeIds: ReadonlySet): string | null { + if (nodeIds.has(raw)) return raw + const normalized = raw.toLowerCase().replace(/\s+/g, "-") + for (const id of nodeIds) { + const idLower = id.toLowerCase() + if (idLower === normalized) return id + if (idLower === raw.toLowerCase()) return id + if (idLower.replace(/\s+/g, "-") === normalized) return id + } + return null +} + +function getNeighbors(node: RetrievalNode): ReadonlySet { + const neighbors = new Set() + for (const id of node.outLinks) neighbors.add(id) + for (const id of node.inLinks) neighbors.add(id) + return neighbors +} + +function getNodeDegree(node: RetrievalNode): number { + return node.outLinks.size + node.inLinks.size +} + +export async function buildRetrievalGraph( + projectPath: string, + dataVersion: number = 0, +): Promise { + if (cachedGraph !== null && cachedGraph.dataVersion === dataVersion) return cachedGraph + + const wikiRoot = `${normalizePath(projectPath)}/wiki` + let tree: FileNode[] + try { + tree = await listDirectory(wikiRoot) + } catch { + const emptyGraph: RetrievalGraph = { nodes: new Map(), dataVersion } + cachedGraph = emptyGraph + return emptyGraph + } + + const mdFiles = flattenMdFiles(tree) + const rawNodes: Array<{ + id: string; title: string; type: string; path: string + sources: string[]; rawLinks: string[]; fileName: string + }> = [] + + for (const file of mdFiles) { + const id = fileNameToId(file.name) + let content = "" + try { content = await readFile(file.path) } catch { continue } + const fm = extractFrontmatter(content) + rawNodes.push({ + id, title: fm.title || file.name.replace(/\.md$/, "").replace(/-/g, " "), + type: fm.type, path: file.path, sources: fm.sources, + rawLinks: extractWikilinks(content), fileName: file.name, + }) + } + + const nodeIds = new Set(rawNodes.map((n) => n.id)) + const outLinksMap = new Map>() + const inLinksMap = new Map>() + for (const id of nodeIds) { + outLinksMap.set(id, new Set()) + inLinksMap.set(id, new Set()) + } + + for (const raw of rawNodes) { + for (const linkTarget of raw.rawLinks) { + const resolvedId = resolveTarget(linkTarget, nodeIds) + if (resolvedId === null || resolvedId === raw.id) continue + outLinksMap.get(raw.id)!.add(resolvedId) + inLinksMap.get(resolvedId)!.add(raw.id) + } + } + + const nodes = new Map() + for (const raw of rawNodes) { + nodes.set(raw.id, { + id: raw.id, title: raw.title, type: raw.type, path: raw.path, + sources: Object.freeze([...raw.sources]), + outLinks: Object.freeze(outLinksMap.get(raw.id) ?? new Set()), + inLinks: Object.freeze(inLinksMap.get(raw.id) ?? new Set()), + }) + } + + const graph: RetrievalGraph = { nodes, dataVersion } + cachedGraph = graph + return graph +} + +export function calculateRelevance( + nodeA: RetrievalNode, nodeB: RetrievalNode, graph: RetrievalGraph, +): number { + if (nodeA.id === nodeB.id) return 0 + const forwardLinks = nodeA.outLinks.has(nodeB.id) ? 1 : 0 + const backwardLinks = nodeB.outLinks.has(nodeA.id) ? 1 : 0 + const directLinkScore = (forwardLinks + backwardLinks) * WEIGHTS.directLink + const sourcesA = new Set(nodeA.sources) + let sharedSourceCount = 0 + for (const src of nodeB.sources) { if (sourcesA.has(src)) sharedSourceCount += 1 } + const sourceOverlapScore = sharedSourceCount * WEIGHTS.sourceOverlap + const neighborsA = getNeighbors(nodeA) + const neighborsB = getNeighbors(nodeB) + let adamicAdar = 0 + for (const neighborId of neighborsA) { + if (neighborsB.has(neighborId)) { + const neighbor = graph.nodes.get(neighborId) + if (neighbor) { + const degree = getNodeDegree(neighbor) + adamicAdar += 1 / Math.log(Math.max(degree, 2)) + } + } + } + const commonNeighborScore = adamicAdar * WEIGHTS.commonNeighbor + const affinityMap = TYPE_AFFINITY[nodeA.type] + const typeAffinityScore = (affinityMap?.[nodeB.type] ?? 0.5) * WEIGHTS.typeAffinity + return directLinkScore + sourceOverlapScore + commonNeighborScore + typeAffinityScore +} + +export function getRelatedNodes( + nodeId: string, graph: RetrievalGraph, limit: number = 5, +): ReadonlyArray<{ node: RetrievalNode; relevance: number }> { + const sourceNode = graph.nodes.get(nodeId) + if (!sourceNode) return [] + const scored: Array<{ node: RetrievalNode; relevance: number }> = [] + for (const [id, node] of graph.nodes) { + if (id === nodeId) continue + const relevance = calculateRelevance(sourceNode, node, graph) + if (relevance > 0) scored.push({ node, relevance }) + } + scored.sort((a, b) => b.relevance - a.relevance) + return scored.slice(0, limit) +} + +export function clearGraphCache(): void { + cachedGraph = null +} diff --git a/skill/src/lib/path-utils.ts b/skill/src/lib/path-utils.ts new file mode 100644 index 00000000..3296d0a9 --- /dev/null +++ b/skill/src/lib/path-utils.ts @@ -0,0 +1,38 @@ +export function normalizePath(p: string): string { + return p.replace(/\\/g, "/") +} + +export function joinPath(...segments: string[]): string { + return segments + .map((s) => s.replace(/\\/g, "/")) + .join("/") + .replace(/\/+/g, "/") +} + +export function getFileName(p: string): string { + const normalized = p.replace(/\\/g, "/") + return normalized.split("/").pop() ?? p +} + +export function getFileStem(p: string): string { + const name = getFileName(p) + const lastDot = name.lastIndexOf(".") + return lastDot > 0 ? name.slice(0, lastDot) : name +} + +export function getRelativePath(fullPath: string, basePath: string): string { + const normalFull = normalizePath(fullPath) + const normalBase = normalizePath(basePath).replace(/\/$/, "") + if (normalFull.startsWith(normalBase + "/")) { + return normalFull.slice(normalBase.length + 1) + } + return normalFull +} + +export function isAbsolutePath(p: string): boolean { + if (!p) return false + if (p.startsWith("/")) return true + if (/^[A-Za-z]:[\\/]/.test(p)) return true + if (p.startsWith("\\\\") || p.startsWith("//")) return true + return false +} diff --git a/skill/src/lib/search.ts b/skill/src/lib/search.ts new file mode 100644 index 00000000..d67e4e51 --- /dev/null +++ b/skill/src/lib/search.ts @@ -0,0 +1,228 @@ +import { readFile, listDirectory } from "../shims/fs-node" +import type { FileNode } from "../types/wiki" +import { normalizePath, getFileStem } from "./path-utils" + +export interface ImageRef { + url: string + alt: string +} + +export interface SearchResult { + path: string + title: string + snippet: string + titleMatch: boolean + score: number + images: ImageRef[] +} + +const MAX_RESULTS = 20 +const SNIPPET_CONTEXT = 80 +const RRF_K = 60 +const FILENAME_EXACT_BONUS = 200 +const PHRASE_IN_TITLE_BONUS = 50 +const PHRASE_IN_CONTENT_PER_OCC = 20 +const MAX_PHRASE_OCC_COUNTED = 10 +const TITLE_TOKEN_WEIGHT = 5 +const CONTENT_TOKEN_WEIGHT = 1 + +const STOP_WORDS = new Set([ + "的", "是", "了", "什么", "在", "有", "和", "与", "对", "从", + "the", "is", "a", "an", "what", "how", "are", "was", "were", + "do", "does", "did", "be", "been", "being", "have", "has", "had", + "it", "its", "in", "on", "at", "to", "for", "of", "with", "by", + "this", "that", "these", "those", +]) + +export function tokenizeQuery(query: string): string[] { + const rawTokens = query + .toLowerCase() + .split(/[\s,，。！？、；：""''（）()\-_/\\·~～…]+/) + .filter((t) => t.length > 1) + .filter((t) => !STOP_WORDS.has(t)) + + const tokens: string[] = [] + for (const token of rawTokens) { + const hasCJK = /[\u4e00-\u9fff\u3400-\u4dbf]/.test(token) + if (hasCJK && token.length > 2) { + const chars = [...token] + for (let i = 0; i < chars.length - 1; i++) tokens.push(chars[i] + chars[i + 1]) + for (const ch of chars) { if (!STOP_WORDS.has(ch)) tokens.push(ch) } + tokens.push(token) + } else { + tokens.push(token) + } + } + return [...new Set(tokens)] +} + +function tokenMatchScore(text: string, tokens: readonly string[]): number { + const lower = text.toLowerCase() + let score = 0 + for (const token of tokens) { if (lower.includes(token)) score += 1 } + return score +} + +function countOccurrences(haystackLower: string, needleLower: string): number { + if (!needleLower) return 0 + let count = 0; let pos = 0 + while (true) { + const idx = haystackLower.indexOf(needleLower, pos) + if (idx === -1) break + count++; pos = idx + needleLower.length + } + return count +} + +function flattenMdFiles(nodes: FileNode[]): FileNode[] { + const files: FileNode[] = [] + for (const node of nodes) { + if (node.is_dir && node.children) files.push(...flattenMdFiles(node.children)) + else if (!node.is_dir && node.name.endsWith(".md")) files.push(node) + } + return files +} + +function extractTitle(content: string, fileName: string): string { + const fm = content.match(/^---\n[\s\S]*?^title:\s*["']?(.+?)["']?\s*$/m) + if (fm) return fm[1].trim() + const h = content.match(/^#\s+(.+)$/m) + if (h) return h[1].trim() + return fileName.replace(/\.md$/, "").replace(/-/g, " ") +} + +const IMAGE_REF_RE = /!\[([^\]]*)\]\(([^)\s]+)\)/g + +function extractImageRefs(content: string): ImageRef[] { + const seen = new Set(); const out: ImageRef[] = [] + for (const m of content.matchAll(IMAGE_REF_RE)) { + const url = m[2] + if (seen.has(url)) continue + seen.add(url); out.push({ url, alt: m[1] }) + } + return out +} + +function buildSnippet(content: string, query: string): string { + const lower = content.toLowerCase(); const lowerQuery = query.toLowerCase() + const idx = lower.indexOf(lowerQuery) + if (idx === -1) return content.slice(0, SNIPPET_CONTEXT * 2).replace(/\n/g, " ") + const start = Math.max(0, idx - SNIPPET_CONTEXT) + const end = Math.min(content.length, idx + query.length + SNIPPET_CONTEXT) + let snippet = content.slice(start, end).replace(/\n/g, " ") + if (start > 0) snippet = "..." + snippet + if (end < content.length) snippet = snippet + "..." + return snippet +} + +const TRIM_PUNCT_RE = /^[\s,，。！？、；：""''（）()\-_/\\·~～…]+|[\s,，。！？、；：""''（）()\-_/\\·~～…]+$/g +const SEARCH_READ_CONCURRENCY = 16 + +function scoreFile( + file: FileNode, content: string, tokens: readonly string[], queryPhrase: string, query: string, +): SearchResult | null { + const title = extractTitle(content, file.name) + const titleText = `${title} ${file.name}` + const titleLower = titleText.toLowerCase() + const contentLower = content.toLowerCase() + const fileStem = file.name.replace(/\.md$/, "").toLowerCase() + + const filenameExact = fileStem === queryPhrase + const titleHasPhrase = queryPhrase.length > 0 && titleLower.includes(queryPhrase) + const contentPhraseOcc = Math.min(countOccurrences(contentLower, queryPhrase), MAX_PHRASE_OCC_COUNTED) + const titleTokenScore = tokenMatchScore(titleText, tokens) + const contentTokenScore = tokenMatchScore(content, tokens) + + if (!filenameExact && !titleHasPhrase && contentPhraseOcc === 0 && titleTokenScore === 0 && contentTokenScore === 0) return null + + const score = + (filenameExact ? FILENAME_EXACT_BONUS : 0) + + (titleHasPhrase ? PHRASE_IN_TITLE_BONUS : 0) + + contentPhraseOcc * PHRASE_IN_CONTENT_PER_OCC + + titleTokenScore * TITLE_TOKEN_WEIGHT + + contentTokenScore * CONTENT_TOKEN_WEIGHT + + const snippetAnchor = contentPhraseOcc > 0 ? queryPhrase : (tokens.find((t) => contentLower.includes(t)) ?? query) + return { + path: file.path, title, snippet: buildSnippet(content, snippetAnchor), + titleMatch: titleTokenScore > 0 || titleHasPhrase, score, images: extractImageRefs(content), + } +} + +async function searchFiles( + files: FileNode[], tokens: readonly string[], query: string, results: SearchResult[], +): Promise { + const queryPhrase = query.trim().toLowerCase().replace(TRIM_PUNCT_RE, "") + for (let i = 0; i < files.length; i += SEARCH_READ_CONCURRENCY) { + const batch = files.slice(i, i + SEARCH_READ_CONCURRENCY) + const batchResults = await Promise.all( + batch.map(async (file) => { + let content: string + try { content = await readFile(file.path) } catch { return null } + return scoreFile(file, content, tokens, queryPhrase, query) + }), + ) + for (const r of batchResults) { if (r) results.push(r) } + } +} + +export async function searchWiki(projectPath: string, query: string): Promise { + if (!query.trim()) return [] + const pp = normalizePath(projectPath) + const tokens = tokenizeQuery(query) + const effectiveTokens = tokens.length > 0 ? tokens : [query.trim().toLowerCase()] + const results: SearchResult[] = [] + + try { + const wikiTree = await listDirectory(`${pp}/wiki`) + const wikiFiles = flattenMdFiles(wikiTree) + await searchFiles(wikiFiles, effectiveTokens, query, results) + } catch { /* no wiki directory */ } + + const tokenSorted = [...results].sort((a, b) => b.score - a.score) + const tokenRank = new Map() + tokenSorted.forEach((r, i) => tokenRank.set(normalizePath(r.path), i + 1)) + + // Vector search (optional — gracefully degrades when embedding not configured) + let vectorRank = new Map() + let vectorCount = 0 + try { + const { useWikiStore } = await import("../shims/stores-node") + const embCfg = useWikiStore.getState().embeddingConfig + if (embCfg.enabled && embCfg.model) { + const { searchByEmbedding } = await import("../shims/embedding-stub") + const vectorResults = await searchByEmbedding(pp, query, embCfg, 10) + vectorCount = vectorResults.length + vectorResults.forEach((vr, i) => vectorRank.set(vr.id, i + 1)) + + const knownIds = new Set(results.map((r) => getFileStem(r.path))) + for (const vr of vectorResults) { + if (knownIds.has(vr.id)) continue + const dirs = ["entities", "concepts", "sources", "synthesis", "comparison", "queries"] + for (const dir of dirs) { + const tryPath = `${pp}/wiki/${dir}/${vr.id}.md` + try { + const content = await readFile(tryPath) + const title = extractTitle(content, `${vr.id}.md`) + results.push({ path: tryPath, title, snippet: buildSnippet(content, query), titleMatch: false, score: 0, images: extractImageRefs(content) }) + knownIds.add(vr.id); break + } catch { /* not in this dir */ } + } + } + } + } catch { /* vector search not available */ } + + // RRF fusion + for (const r of results) { + const tRank = tokenRank.get(normalizePath(r.path)) + const vRank = vectorRank.get(getFileStem(r.path)) + let rrf = 0 + if (tRank !== undefined) rrf += 1 / (RRF_K + tRank) + if (vRank !== undefined) rrf += 1 / (RRF_K + vRank) + r.score = rrf + } + + results.sort((a, b) => b.score !== a.score ? b.score - a.score : a.path.localeCompare(b.path)) + console.error(`[search] "${query}" | token:${tokenRank.size} vector:${vectorCount} → ${results.length} results`) + return results.slice(0, MAX_RESULTS) +} diff --git a/skill/src/lib/wiki-graph.ts b/skill/src/lib/wiki-graph.ts new file mode 100644 index 00000000..5160563a --- /dev/null +++ b/skill/src/lib/wiki-graph.ts @@ -0,0 +1,211 @@ +import { readFile, listDirectory } from "../shims/fs-node" +import type { FileNode } from "../types/wiki" +import { buildRetrievalGraph, calculateRelevance } from "./graph-relevance" +import { normalizePath } from "./path-utils" +import Graph from "graphology" +import louvain from "graphology-communities-louvain" + +export interface GraphNode { + id: string + label: string + type: string + path: string + linkCount: number + community: number +} + +export interface GraphEdge { + source: string + target: string + weight: number +} + +export interface CommunityInfo { + id: number + nodeCount: number + cohesion: number + topNodes: string[] +} + +function detectCommunities( + nodes: { id: string; label: string; linkCount: number }[], + edges: GraphEdge[], +): { assignments: Map; communities: CommunityInfo[] } { + if (nodes.length === 0) return { assignments: new Map(), communities: [] } + + const g = new Graph({ type: "undirected" }) + for (const node of nodes) g.addNode(node.id) + for (const edge of edges) { + if (g.hasNode(edge.source) && g.hasNode(edge.target)) { + const key = `${edge.source}->${edge.target}` + if (!g.hasEdge(key) && !g.hasEdge(`${edge.target}->${edge.source}`)) { + g.addEdgeWithKey(key, edge.source, edge.target, { weight: edge.weight }) + } + } + } + + const communityMap: Record = louvain(g, { resolution: 1 }) + const assignments = new Map(Object.entries(communityMap).map(([k, v]) => [k, v as number])) + + const groups = new Map() + for (const [nodeId, commId] of assignments) { + const list = groups.get(commId) ?? [] + list.push(nodeId) + groups.set(commId, list) + } + + const edgeSet = new Set() + for (const edge of edges) { + edgeSet.add(`${edge.source}:::${edge.target}`) + edgeSet.add(`${edge.target}:::${edge.source}`) + } + + const nodeInfo = new Map(nodes.map((n) => [n.id, { label: n.label, linkCount: n.linkCount }])) + const communities: CommunityInfo[] = [] + + for (const [commId, memberIds] of groups) { + const n = memberIds.length + let intraEdges = 0 + for (let i = 0; i < memberIds.length; i++) { + for (let j = i + 1; j < memberIds.length; j++) { + if (edgeSet.has(`${memberIds[i]}:::${memberIds[j]}`)) intraEdges++ + } + } + const possibleEdges = n > 1 ? (n * (n - 1)) / 2 : 1 + const cohesion = intraEdges / possibleEdges + const sorted = [...memberIds].sort( + (a, b) => (nodeInfo.get(b)?.linkCount ?? 0) - (nodeInfo.get(a)?.linkCount ?? 0) + ) + communities.push({ id: commId, nodeCount: n, cohesion, topNodes: sorted.slice(0, 5).map((id) => nodeInfo.get(id)?.label ?? id) }) + } + + communities.sort((a, b) => b.nodeCount - a.nodeCount) + const idRemap = new Map() + communities.forEach((c, idx) => { idRemap.set(c.id, idx); c.id = idx }) + for (const [nodeId, oldId] of assignments) assignments.set(nodeId, idRemap.get(oldId) ?? 0) + + return { assignments, communities } +} + +const WIKILINK_REGEX = /\[\[([^\]|]+?)(?:\|[^\]]+?)?\]\]/g + +function flattenMdFiles(nodes: FileNode[]): FileNode[] { + const files: FileNode[] = [] + for (const node of nodes) { + if (node.is_dir && node.children) files.push(...flattenMdFiles(node.children)) + else if (!node.is_dir && node.name.endsWith(".md")) files.push(node) + } + return files +} + +function extractTitle(content: string, fileName: string): string { + const fm = content.match(/^---\n[\s\S]*?^title:\s*["']?(.+?)["']?\s*$/m) + if (fm) return fm[1].trim() + const h = content.match(/^#\s+(.+)$/m) + if (h) return h[1].trim() + return fileName.replace(/\.md$/, "").replace(/-/g, " ") +} + +function extractType(content: string): string { + const m = content.match(/^---\n[\s\S]*?^type:\s*["']?(.+?)["']?\s*$/m) + return m ? m[1].trim().toLowerCase() : "other" +} + +function extractWikilinks(content: string): string[] { + const links: string[] = [] + const regex = new RegExp(WIKILINK_REGEX.source, "g") + let match: RegExpExecArray | null + while ((match = regex.exec(content)) !== null) links.push(match[1].trim()) + return links +} + +function resolveTarget(raw: string, nodeMap: Map): string | null { + if (nodeMap.has(raw)) return raw + const normalized = raw.toLowerCase().replace(/\s+/g, "-") + for (const id of nodeMap.keys()) { + if (id.toLowerCase() === normalized) return id + if (id.toLowerCase() === raw.toLowerCase()) return id + if (id.toLowerCase().replace(/\s+/g, "-") === normalized) return id + } + return null +} + +export async function buildWikiGraph( + projectPath: string, +): Promise<{ nodes: GraphNode[]; edges: GraphEdge[]; communities: CommunityInfo[] }> { + const wikiRoot = `${normalizePath(projectPath)}/wiki` + let tree: FileNode[] + try { tree = await listDirectory(wikiRoot) } catch { + return { nodes: [], edges: [], communities: [] } + } + + const mdFiles = flattenMdFiles(tree) + if (mdFiles.length === 0) return { nodes: [], edges: [], communities: [] } + + const nodeMap = new Map() + for (const file of mdFiles) { + const id = file.name.replace(/\.md$/, "") + let content = "" + try { content = await readFile(file.path) } catch { continue } + nodeMap.set(id, { id, label: extractTitle(content, file.name), type: extractType(content), path: file.path, links: extractWikilinks(content) }) + } + + const HIDDEN_TYPES = new Set(["query"]) + for (const [id, node] of nodeMap) { + if (HIDDEN_TYPES.has(node.type)) nodeMap.delete(id) + } + + const linkCounts = new Map() + for (const [id] of nodeMap) linkCounts.set(id, 0) + + const rawEdges: GraphEdge[] = [] + for (const [sourceId, nodeData] of nodeMap) { + for (const targetRaw of nodeData.links) { + const targetId = resolveTarget(targetRaw, nodeMap) + if (targetId === null || targetId === sourceId) continue + rawEdges.push({ source: sourceId, target: targetId, weight: 1 }) + linkCounts.set(sourceId, (linkCounts.get(sourceId) ?? 0) + 1) + linkCounts.set(targetId, (linkCounts.get(targetId) ?? 0) + 1) + } + } + + const seenEdges = new Set() + const dedupedEdges: { source: string; target: string }[] = [] + for (const edge of rawEdges) { + const key = `${edge.source}:::${edge.target}` + const reverseKey = `${edge.target}:::${edge.source}` + if (!seenEdges.has(key) && !seenEdges.has(reverseKey)) { + seenEdges.add(key) + dedupedEdges.push(edge) + } + } + + // Try to get retrieval graph for weighted edges (gracefully degrades) + let retrievalGraph: Awaited> | null = null + try { + const { useWikiStore } = await import("../shims/stores-node") + const dv = useWikiStore.getState().dataVersion + retrievalGraph = await buildRetrievalGraph(normalizePath(projectPath), dv) + } catch { /* ignore — weights default to 1 */ } + + const edges: GraphEdge[] = dedupedEdges.map((e) => { + let weight = 1 + if (retrievalGraph) { + const nodeA = retrievalGraph.nodes.get(e.source) + const nodeB = retrievalGraph.nodes.get(e.target) + if (nodeA && nodeB) weight = calculateRelevance(nodeA, nodeB, retrievalGraph) + } + return { source: e.source, target: e.target, weight } + }) + + const prelimNodes = Array.from(nodeMap.values()).map((n) => ({ id: n.id, label: n.label, linkCount: linkCounts.get(n.id) ?? 0 })) + const { assignments, communities } = detectCommunities(prelimNodes, edges) + + const nodes: GraphNode[] = Array.from(nodeMap.values()).map((n) => ({ + id: n.id, label: n.label, type: n.type, path: n.path, + linkCount: linkCounts.get(n.id) ?? 0, + community: assignments.get(n.id) ?? 0, + })) + + return { nodes, edges, communities } +} diff --git a/skill/src/mcp-server.ts b/skill/src/mcp-server.ts new file mode 100644 index 00000000..769e6ec0 --- /dev/null +++ b/skill/src/mcp-server.ts @@ -0,0 +1,251 @@ +#!/usr/bin/env node +/** + * llm-wiki MCP Server + * + * Exposes nashsu/llm_wiki backend operations as Model Context Protocol tools. + * Works with Claude Desktop, VS Code Copilot, and any MCP-compatible host. + * + * Tools: + * wiki_status — Page count and type breakdown for a project + * wiki_search — BM25 keyword search (+ optional vector via EMBEDDING_ENABLED) + * wiki_graph — Build knowledge graph (nodes, edges, Louvain communities) + * wiki_insights — Surprising connections and knowledge gaps analysis + * wiki_lint — Structural lint: orphans, no-outlinks, broken links + * + * Usage: + * node dist/mcp-server.js + * WIKI_PATH=/path/to/project node dist/mcp-server.js (default project path) + */ +import { Server } from "@modelcontextprotocol/sdk/server/index.js" +import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js" +import { + CallToolRequestSchema, + ListToolsRequestSchema, + ErrorCode, + McpError, +} from "@modelcontextprotocol/sdk/types.js" + +import * as path from "path" +import { buildWikiGraph } from "./lib/wiki-graph" +import { findSurprisingConnections, detectKnowledgeGaps } from "./lib/graph-insights" +import { searchWiki } from "./lib/search" + +const DEFAULT_WIKI_PATH = process.env.WIKI_PATH ?? process.cwd() +const PKG_VERSION = "0.4.6-mcp" + +const server = new Server( + { name: "llm-wiki", version: PKG_VERSION }, + { capabilities: { tools: {} } }, +) + +// ── Tool definitions ────────────────────────────────────────────────────────── +server.setRequestHandler(ListToolsRequestSchema, async () => ({ + tools: [ + { + name: "wiki_status", + description: "Get page count and type breakdown for a wiki project. Returns statistics about the knowledge base.", + inputSchema: { + type: "object", + properties: { + project_path: { + type: "string", + description: "Absolute path to the wiki project directory (contains wiki/ subdirectory)", + }, + }, + required: [], + }, + }, + { + name: "wiki_search", + description: "Search wiki pages using BM25 keyword matching with optional vector search (RRF fusion). Returns ranked results with snippets.", + inputSchema: { + type: "object", + properties: { + query: { type: "string", description: "Search query (supports Chinese and English)" }, + project_path: { type: "string", description: "Path to wiki project (defaults to WIKI_PATH env var)" }, + limit: { type: "number", description: "Max results to return (default: 10)" }, + }, + required: ["query"], + }, + }, + { + name: "wiki_graph", + description: "Build knowledge graph from wiki pages: wikilinks, type-based edges, Louvain community detection. Returns nodes, edges, and community clusters.", + inputSchema: { + type: "object", + properties: { + project_path: { type: "string", description: "Path to wiki project" }, + format: { + type: "string", + enum: ["json", "summary"], + description: "Output format: 'json' for full graph data, 'summary' for human-readable overview (default: summary)", + }, + }, + required: [], + }, + }, + { + name: "wiki_insights", + description: "Analyze wiki graph structure to find surprising cross-community connections and knowledge gaps (isolated pages, sparse clusters, bridge nodes).", + inputSchema: { + type: "object", + properties: { + project_path: { type: "string", description: "Path to wiki project" }, + max_connections: { type: "number", description: "Max surprising connections to return (default: 5)" }, + max_gaps: { type: "number", description: "Max knowledge gaps to return (default: 8)" }, + }, + required: [], + }, + }, + { + name: "wiki_lint", + description: "Structural lint of wiki pages: find orphaned pages (no links), no-outlinks, and connectivity issues.", + inputSchema: { + type: "object", + properties: { + project_path: { type: "string", description: "Path to wiki project" }, + }, + required: [], + }, + }, + ], +})) + +// ── Tool handlers ───────────────────────────────────────────────────────────── +server.setRequestHandler(CallToolRequestSchema, async (request) => { + const { name, arguments: args = {} } = request.params + const projectPath = path.resolve((args.project_path as string | undefined) ?? DEFAULT_WIKI_PATH) + + try { + switch (name) { + case "wiki_status": { + const { nodes, communities } = await buildWikiGraph(projectPath) + const typeCounts: Record = {} + for (const n of nodes) typeCounts[n.type] = (typeCounts[n.type] ?? 0) + 1 + const summary = [ + `Wiki: ${projectPath}`, + `Total pages: ${nodes.length}`, + `Communities: ${communities.length}`, + ...Object.entries(typeCounts) + .sort((a, b) => b[1] - a[1]) + .map(([t, c]) => ` ${t}: ${c}`), + ].join("\n") + return { content: [{ type: "text", text: summary }] } + } + + case "wiki_search": { + if (!args.query) throw new McpError(ErrorCode.InvalidParams, "query is required") + const results = await searchWiki(projectPath, args.query as string) + const limit = typeof args.limit === "number" ? args.limit : 10 + const top = results.slice(0, limit) + if (top.length === 0) { + return { content: [{ type: "text", text: `No results for: "${args.query}"` }] } + } + const lines = [`# Search: "${args.query}"\n`] + for (const r of top) { + const relPath = path.relative(projectPath, r.path) + lines.push(`## ${r.title}`) + lines.push(`**Path**: ${relPath} | **Score**: ${r.score.toFixed(4)}`) + lines.push(r.snippet) + lines.push("") + } + return { content: [{ type: "text", text: lines.join("\n") }] } + } + + case "wiki_graph": { + const graphData = await buildWikiGraph(projectPath) + const format = (args.format as string | undefined) ?? "summary" + if (format === "json") { + return { content: [{ type: "text", text: JSON.stringify(graphData, null, 2) }] } + } + // Summary format + const { nodes, edges, communities } = graphData + const typeCounts: Record = {} + for (const n of nodes) typeCounts[n.type] = (typeCounts[n.type] ?? 0) + 1 + const lines = [ + `# Knowledge Graph Summary`, + ``, + `**Nodes**: ${nodes.length} | **Edges**: ${edges.length} | **Communities**: ${communities.length}`, + ``, + `## Node Types`, + ...Object.entries(typeCounts).sort((a, b) => b[1] - a[1]).map(([t, c]) => `- ${t}: ${c}`), + ``, + `## Top Communities`, + ...communities.slice(0, 5).map((c, i) => + `### Community ${i + 1} (${c.nodeCount} pages, cohesion: ${c.cohesion.toFixed(2)})\nKey pages: ${c.topNodes.join(", ")}` + ), + ``, + `## Top Hubs (by link count)`, + ...nodes.sort((a, b) => b.linkCount - a.linkCount).slice(0, 10) + .map((n) => `- ${n.label} (${n.type}, ${n.linkCount} links)`), + ] + return { content: [{ type: "text", text: lines.join("\n") }] } + } + + case "wiki_insights": { + const { nodes, edges, communities } = await buildWikiGraph(projectPath) + const maxConn = typeof args.max_connections === "number" ? args.max_connections : 5 + const maxGaps = typeof args.max_gaps === "number" ? args.max_gaps : 8 + const connections = findSurprisingConnections(nodes, edges, communities, maxConn) + const gaps = detectKnowledgeGaps(nodes, edges, communities, maxGaps) + + const lines = [`# Wiki Insights\n`, `## Surprising Connections\n`] + if (connections.length === 0) lines.push("_No surprising connections found yet._\n") + for (const c of connections) { + lines.push(`### ${c.source.label} ↔ ${c.target.label}`) + lines.push(`- Score: ${c.score} | ${c.reasons.join(", ")}\n`) + } + lines.push(`## Knowledge Gaps\n`) + if (gaps.length === 0) lines.push("_No gaps detected._\n") + for (const g of gaps) { + lines.push(`### ${g.title}`) + lines.push(`${g.description}`) + lines.push(`💡 ${g.suggestion}\n`) + } + return { content: [{ type: "text", text: lines.join("\n") }] } + } + + case "wiki_lint": { + const { nodes, edges } = await buildWikiGraph(projectPath) + if (nodes.length === 0) { + return { content: [{ type: "text", text: "No wiki pages found." }] } + } + const edgeTargets = new Set(edges.map((e) => e.target)) + const edgeSources = new Set(edges.map((e) => e.source)) + const allLinked = new Set([...edgeTargets, ...edgeSources]) + const issues: string[] = [] + for (const n of nodes) { + if (n.id === "index" || n.id === "log" || n.id === "overview") continue + if (!allLinked.has(n.id)) issues.push(`[orphan] ${n.label} (${n.id}.md)`) + else if (n.linkCount <= 1) issues.push(`[isolated] ${n.label} — only ${n.linkCount} link(s)`) + } + const text = issues.length === 0 + ? `✓ All ${nodes.length} pages are properly connected.` + : `Found ${issues.length} issue(s) in ${nodes.length} pages:\n\n${issues.join("\n")}` + return { content: [{ type: "text", text: text }] } + } + + default: + throw new McpError(ErrorCode.MethodNotFound, `Unknown tool: ${name}`) + } + } catch (err) { + if (err instanceof McpError) throw err + throw new McpError( + ErrorCode.InternalError, + `Tool '${name}' failed: ${err instanceof Error ? err.message : String(err)}`, + ) + } +}) + +// ── Start server ────────────────────────────────────────────────────────────── +async function main() { + const transport = new StdioServerTransport() + await server.connect(transport) + console.error(`llm-wiki MCP server v${PKG_VERSION} started`) + console.error(`Default wiki path: ${DEFAULT_WIKI_PATH}`) +} + +main().catch((err) => { + console.error("Failed to start MCP server:", err) + process.exit(1) +}) diff --git a/skill/src/shims/embedding-stub.ts b/skill/src/shims/embedding-stub.ts new file mode 100644 index 00000000..8a2117aa --- /dev/null +++ b/skill/src/shims/embedding-stub.ts @@ -0,0 +1,32 @@ +/** + * Stub for @/lib/embedding — vector search is not available in the skill/MCP layer + * without a running LanceDB instance. Returns empty results so BM25-only search works. + */ +export interface EmbeddingConfig { + enabled: boolean + model: string + apiBase?: string + apiKey?: string +} + +export interface VectorSearchResult { + id: string + score: number + path?: string +} + +export async function searchByEmbedding( + _projectPath: string, + _query: string, + _config: EmbeddingConfig, + _limit: number = 10, +): Promise { + return [] +} + +export async function createEmbedding( + _text: string, + _config: EmbeddingConfig, +): Promise { + return [] +} diff --git a/skill/src/shims/fs-node.ts b/skill/src/shims/fs-node.ts new file mode 100644 index 00000000..de343bda --- /dev/null +++ b/skill/src/shims/fs-node.ts @@ -0,0 +1,101 @@ +/** + * Node.js drop-in replacement for @/commands/fs (nashsu Tauri IPC layer). + * Replaces all invoke("...") calls with standard Node.js fs operations. + */ +import * as fs from "fs" +import * as path from "path" +import type { FileNode } from "../types/wiki" + +export async function readFile(filePath: string): Promise { + return fs.readFileSync(filePath, "utf-8") +} + +export async function writeFile(filePath: string, content: string): Promise { + fs.mkdirSync(path.dirname(filePath), { recursive: true }) + fs.writeFileSync(filePath, content, "utf-8") +} + +export async function listDirectory(dirPath: string): Promise { + function walk(dir: string): FileNode[] { + const entries = fs.readdirSync(dir, { withFileTypes: true }) + return entries.map((entry) => { + const entryPath = path.join(dir, entry.name).replace(/\\/g, "/") + if (entry.isDirectory()) { + return { + name: entry.name, + path: entryPath, + is_dir: true, + children: walk(entryPath), + } + } + return { name: entry.name, path: entryPath, is_dir: false } + }) + } + return walk(dirPath) +} + +export async function copyFile(from: string, to: string): Promise { + fs.mkdirSync(path.dirname(to), { recursive: true }) + fs.copyFileSync(from, to) +} + +export async function deleteFile(filePath: string): Promise { + if (fs.existsSync(filePath)) fs.unlinkSync(filePath) +} + +export async function createDirectory(dirPath: string): Promise { + fs.mkdirSync(dirPath, { recursive: true }) +} + +export async function fileExists(filePath: string): Promise { + return fs.existsSync(filePath) +} + +export async function readFileAsBase64(filePath: string): Promise { + return fs.readFileSync(filePath).toString("base64") +} + +/** + * Text extraction for PDFs/DOCX/etc. + * In Node mode: returns raw file content if text-based, otherwise empty string. + * For real PDF extraction, users should pre-extract with markitdown or pdftotext. + */ +export async function preprocessFile(filePath: string): Promise { + const ext = path.extname(filePath).toLowerCase() + const textExts = [".md", ".txt", ".json", ".yaml", ".yml", ".csv", ".html", ".htm"] + if (textExts.includes(ext)) { + try { + return fs.readFileSync(filePath, "utf-8") + } catch { + return "" + } + } + // For binary files (PDF, DOCX, etc.) return empty — use pre-extracted markdown + console.warn(`[fs-node] preprocessFile: binary format not supported in Node mode: ${filePath}`) + return "" +} + +export async function findRelatedWikiPages( + sourceFile: string, + wikiRoot: string, +): Promise { + const stem = path.basename(sourceFile, path.extname(sourceFile)).toLowerCase() + const results: string[] = [] + function walk(dir: string): void { + if (!fs.existsSync(dir)) return + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name) + if (entry.isDirectory()) walk(full) + else if (entry.name.endsWith(".md")) { + try { + const content = fs.readFileSync(full, "utf-8") + if (content.toLowerCase().includes(stem)) { + results.push(full.replace(/\\/g, "/")) + } + } catch { /* skip */ } + } + } + } + walk(wikiRoot) + return results +} diff --git a/skill/src/shims/stores-node.ts b/skill/src/shims/stores-node.ts new file mode 100644 index 00000000..66fa3773 --- /dev/null +++ b/skill/src/shims/stores-node.ts @@ -0,0 +1,135 @@ +/** + * Node.js drop-in for React zustand stores used by nashsu/llm_wiki lib files. + * Replaces all useXxxStore.getState() calls with module-level state. + */ + +export interface LlmConfig { + provider: string + apiKey: string + model: string + baseUrl?: string + temperature?: number + maxTokens?: number +} + +export interface EmbeddingConfig { + enabled: boolean + model: string + apiBase?: string + apiKey?: string +} + +interface WikiState { + projectPath: string + dataVersion: number + llmConfig: LlmConfig + embeddingConfig: EmbeddingConfig +} + +let wikiState: WikiState = { + projectPath: "", + dataVersion: 0, + llmConfig: { + provider: process.env.LLM_PROVIDER ?? "openai", + apiKey: process.env.OPENAI_API_KEY ?? process.env.LLM_API_KEY ?? "", + model: process.env.LLM_MODEL ?? "gpt-4o-mini", + baseUrl: process.env.LLM_BASE_URL, + }, + embeddingConfig: { + enabled: (process.env.EMBEDDING_ENABLED ?? "false") === "true", + model: process.env.EMBEDDING_MODEL ?? "", + apiBase: process.env.EMBEDDING_BASE_URL, + apiKey: process.env.EMBEDDING_API_KEY ?? process.env.OPENAI_API_KEY, + }, +} + +export const useWikiStore = { + getState: () => ({ ...wikiState }), + setState: (updater: Partial | ((s: WikiState) => Partial)) => { + if (typeof updater === "function") { + wikiState = { ...wikiState, ...updater(wikiState) } + } else { + wikiState = { ...wikiState, ...updater } + } + }, +} + +/** Configure the wiki store from environment variables or explicit config */ +export function configureWikiStore(config: Partial) { + wikiState = { ...wikiState, ...config } +} + +// ── Research store ─────────────────────────────────────────────────────────── +interface ResearchState { + activeProjectPath: string + isResearching: boolean +} + +let researchState: ResearchState = { + activeProjectPath: "", + isResearching: false, +} + +export const useResearchStore = { + getState: () => ({ ...researchState }), + setState: (updater: Partial) => { + researchState = { ...researchState, ...updater } + }, +} + +// ── Activity store (replaces Tauri event system) ───────────────────────────── +export interface ActivityItem { + id: string + type: string + title: string + status: "pending" | "running" | "done" | "error" + detail?: string + filesWritten?: string[] +} + +let activityItems: ActivityItem[] = [] +let activityIdCounter = 0 + +export const useActivityStore = { + getState: () => ({ + items: [...activityItems], + addItem: (item: Omit): string => { + const id = `activity-${++activityIdCounter}` + const newItem: ActivityItem = { id, ...item } + activityItems.push(newItem) + if (process.env.SKILL_VERBOSE === "1") { + console.error(`[activity:${item.type}] ${item.title} — ${item.status}`) + } + return id + }, + updateItem: (id: string, updates: Partial): void => { + const idx = activityItems.findIndex((i) => i.id === id) + if (idx >= 0) { + activityItems[idx] = { ...activityItems[idx], ...updates } + if (process.env.SKILL_VERBOSE === "1") { + const item = activityItems[idx] + console.error(`[activity:update] ${item.title} — ${item.status}: ${item.detail ?? ""}`) + } + } + }, + clearItems: () => { activityItems = [] }, + }), + addItem: (item: Omit): string => { + return useActivityStore.getState().addItem(item) + }, + updateItem: (id: string, updates: Partial): void => { + useActivityStore.getState().updateItem(id, updates) + }, +} + +// ── Chat store ─────────────────────────────────────────────────────────────── +export const useChatStore = { + getState: () => ({ messages: [] as unknown[] }), + setState: (_updater: unknown) => {}, +} + +// ── Review store ───────────────────────────────────────────────────────────── +export const useReviewStore = { + getState: () => ({ queue: [] as unknown[], isProcessing: false }), + setState: (_updater: unknown) => {}, +} diff --git a/skill/src/types/wiki.ts b/skill/src/types/wiki.ts new file mode 100644 index 00000000..a51ace03 --- /dev/null +++ b/skill/src/types/wiki.ts @@ -0,0 +1,18 @@ +export interface WikiProject { + id: string + name: string + path: string +} + +export interface FileNode { + name: string + path: string + is_dir: boolean + children?: FileNode[] +} + +export interface WikiPage { + path: string + content: string + frontmatter: Record +} diff --git a/skill/tsconfig.json b/skill/tsconfig.json new file mode 100644 index 00000000..54aad9ba --- /dev/null +++ b/skill/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compilerOptions": { + "target": "ES2022", + "lib": ["ES2022"], + "module": "CommonJS", + "moduleResolution": "node", + "strict": true, + "esModuleInterop": true, + "allowSyntheticDefaultImports": true, + "skipLibCheck": true, + "outDir": "dist", + "rootDir": "src", + "declaration": true, + "sourceMap": true, + "noEmit": false + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist"] +} From 7e01db8ee6eda7f7ab094befe477f81c4e0c782d Mon Sep 17 00:00:00 2001 From: toughhou Date: Sat, 2 May 2026 02:58:16 -0700 Subject: [PATCH 3/6] feat(skill): add ingest dependencies, LLM client, docs - Port llm-client.ts: OpenAI-compatible SSE streaming (native fetch) - Port detect-language.ts: Unicode script-based language detection - Port output-language.ts: LLM output language directive builder - Port frontmatter.ts: js-yaml frontmatter parser - Port sources-merge.ts: frontmatter array field union merging - Port page-merge.ts: LLM-assisted wiki page merging - Port ingest-sanitize.ts: LLM output sanitization - Port ingest-cache.ts: SHA256 incremental ingest cache (Node.js crypto) - Port project-mutex.ts: per-project async mutex - Port web-search.ts: Tavily API web search wrapper - Add skill/docs/skill-mcp-progress.md: project plan and progress doc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- SKILL.md | 19 ++- skill/docs/skill-mcp-progress.md | 248 +++++++++++++++++++++++++++++++ skill/src/lib/detect-language.ts | 52 +++++++ skill/src/lib/frontmatter.ts | 70 +++++++++ skill/src/lib/ingest-cache.ts | 76 ++++++++++ skill/src/lib/ingest-sanitize.ts | 44 ++++++ skill/src/lib/llm-client.ts | 165 ++++++++++++++++++++ skill/src/lib/output-language.ts | 26 ++++ skill/src/lib/page-merge.ts | 93 ++++++++++++ skill/src/lib/project-mutex.ts | 30 ++++ skill/src/lib/sources-merge.ts | 80 ++++++++++ skill/src/lib/web-search.ts | 57 +++++++ 12 files changed, 953 insertions(+), 7 deletions(-) create mode 100644 skill/docs/skill-mcp-progress.md create mode 100644 skill/src/lib/detect-language.ts create mode 100644 skill/src/lib/frontmatter.ts create mode 100644 skill/src/lib/ingest-cache.ts create mode 100644 skill/src/lib/ingest-sanitize.ts create mode 100644 skill/src/lib/llm-client.ts create mode 100644 skill/src/lib/output-language.ts create mode 100644 skill/src/lib/page-merge.ts create mode 100644 skill/src/lib/project-mutex.ts create mode 100644 skill/src/lib/sources-merge.ts create mode 100644 skill/src/lib/web-search.ts diff --git a/SKILL.md b/SKILL.md index cc27b77f..267bd302 100644 --- a/SKILL.md +++ b/SKILL.md @@ -31,13 +31,16 @@ metadata: | 能力 | 本技能 | llm-wiki-skill | |------|-------|----------------| -| **图谱相关度** | 4 信号模型（直接链接×3 + 来源重叠×4 + Adamic-Adar×1.5 + 类型亲和×1）| 仅 wikilink，无权重 | -| **社区检测** | Louvain 算法 + 凝聚度评分 | 主题页→社区（启发式）| -| **图谱洞察** | 惊人连接 + 知识缺口 + 桥节点检测 | 无 | -| **搜索** | RRF 混合（BM25 + 向量） | Grep 关键词 | +| **图谱相关度** | 4 信号模型（直接链接×3 + 来源重叠×4 + Adamic-Adar×1.5 + 类型亲和×1）| 3 信号模型（共引强度 + 来源重叠 + 类型亲和度）| +| **社区检测** | Louvain 算法 + 凝聚度评分 | Louvain 算法（graph-analysis.js）| +| **图谱洞察** | 惊人连接 + 知识缺口 + 桥节点检测 | 惊人连接 + 桥节点 + 孤立节点 + 稀疏社区（大图自动降级）| +| **搜索** | RRF 混合（BM25 + 向量） | Grep + 别名展开 + 段落上限 | | **深度研究** | 网络搜索→LLM 综合→自动消化 | 无 | | **审核队列** | 异步异步 sweep-reviews 系统 | 无 | | **图像处理** | 视觉 API 图像标注管线 | 无 | +| **数字山水可视化** | 无（sigma.js 通用图谱）| ✅ 东方编辑部 × 数字山水风交互式 HTML | +| **置信度标注** | 无 | ✅ EXTRACTED / INFERRED / AMBIGUOUS / UNVERIFIED | +| **SessionStart hook** | 无 | ✅ 会话自动注入 wiki 上下文 | --- @@ -272,11 +275,13 @@ node ${SKILL_DIR}/skill/cli.js status | 场景 | 推荐方案 | |------|---------| -| 日常 ingest（速度优先）| llm-wiki-skill（Shell，零开销）| -| 图谱质量分析 | 本技能（graph + insights 命令）| +| 日常 ingest（速度优先）| llm-wiki-skill（Shell，零开销，SHA256 缓存）| +| 高精度图谱分析（Adamic-Adar） | 本技能（graph + insights 命令，4 信号模型）| +| RRF 混合搜索 | 本技能（search 命令，BM25+向量）| | 深度研究专项 | 本技能（deep-research 命令）| +| 基础图谱分析与可视化 | llm-wiki-skill（3 信号 + Louvain + 数字山水 HTML）| | 中文内容源（微信/知乎/小红书）| llm-wiki-skill | -| Hermes Runtime 集成 | llm-wiki-skill（已有 HERMES.md）| +| Hermes Runtime 集成 | llm-wiki-skill（已有 HERMES.md + SessionStart hook）| | 本技能 Hermes 集成 | 参见 HERMES.md（需手动适配）| --- diff --git a/skill/docs/skill-mcp-progress.md b/skill/docs/skill-mcp-progress.md new file mode 100644 index 00000000..e30a9b8c --- /dev/null +++ b/skill/docs/skill-mcp-progress.md @@ -0,0 +1,248 @@ +# llm_wiki Node.js Skill + MCP Server — 方案与进度 + +> 文档生成日期：2026-05-02 +> 状态：**进行中** — ingest / deep-research 实现中，PR 待更新 + +--- + +## 一、背景与目标 + +### 项目来源 + +[nashsu/llm_wiki](https://github.com/nashsu/llm_wiki) 是一个基于 Tauri v2（Rust + React/TypeScript）的桌面应用，核心功能是把本地源文件（Markdown/PDF/DOCX）通过 LLM 自动整理成结构化 Wiki。 + +### 需求 + +bid-sys 项目需要其后台核心逻辑，但 **不需要 GUI（Tauri 桌面应用）**，目标是： + +1. **Node.js Skill** — 纯命令行可调用的 wiki 管理工具 +2. **MCP Server** — 将 wiki 操作暴露为 AI 可调用的工具（供 Claude Desktop / VS Code Copilot Chat 使用） +3. **贡献 MCP** — 向 nashsu/llm_wiki 提交 PR，将 MCP 服务器作为官方插件 + +--- + +## 二、架构分析 + +### nashsu/llm_wiki 技术栈 + +``` +llm_wiki/ +├── src/ # React + TypeScript 前端（GUI 层） +│ ├── lib/ # 核心业务逻辑（纯 TypeScript）⬅ 我们需要的 +│ ├── stores/ # Zustand React 状态管理 +│ └── commands/ # Tauri IPC 桥接层 +├── src-tauri/ # Rust 后端（文件 I/O、PDF 提取、系统集成） +``` + +### 两个注入点 + +所有 `src/lib/*.ts` 通过两个抽象层与 Tauri 交互： + +| 原始导入 | 功能 | Node.js 替代 | +|---------|------|-------------| +| `@/commands/fs` | 文件读写/列举 | `shims/fs-node.ts` | +| `@/stores/*` | 应用状态（LLM 配置等）| `shims/stores-node.ts` | + +Tauri HTTP 代理（`tauri-fetch.ts`）已内置 `isNodeEnv` 检测，直接降级到 `globalThis.fetch`，无需额外适配。 + +--- + +## 三、实现方案 + +### 方案选择：自包含副本（Self-Contained Copy） + +将 `src/lib/*.ts` 复制并修补到 `skill/src/lib/`，修改所有 `@/` 路径别名为相对路径，完全独立于原始项目结构。 + +**优点：** 不依赖 tsconfig 路径别名，构建简单，易于移植 +**缺点：** 需手工同步上游更新 + +### 目录结构 + +``` +skill/ +├── src/ +│ ├── cli.ts # CLI 入口（8 个命令） +│ ├── mcp-server.ts # MCP 服务器（7 个工具） +│ ├── lib/ # 从 nashsu/llm_wiki 移植的核心库 +│ │ ├── graph-relevance.ts +│ │ ├── wiki-graph.ts +│ │ ├── graph-insights.ts +│ │ ├── search.ts +│ │ ├── path-utils.ts +│ │ ├── llm-client.ts # LLM SSE 流式调用 +│ │ ├── detect-language.ts # Unicode 脚本语言检测 +│ │ ├── output-language.ts # 输出语言指令构建 +│ │ ├── frontmatter.ts # YAML frontmatter 解析器 +│ │ ├── sources-merge.ts # Frontmatter 数组字段合并 +│ │ ├── page-merge.ts # Wiki 页面内容合并（LLM） +│ │ ├── ingest-sanitize.ts # LLM 输出清理 +│ │ ├── ingest-cache.ts # SHA256 内容缓存 +│ │ ├── project-mutex.ts # 按项目路径的异步互斥锁 +│ │ ├── ingest.ts # 核心 ingest 流水线（待完成） +│ │ └── web-search.ts # Tavily 搜索 API +│ ├── shims/ # Tauri → Node.js 适配层 +│ │ ├── fs-node.ts +│ │ ├── stores-node.ts +│ │ └── embedding-stub.ts +│ └── types/ +│ └── wiki.ts +├── package.json +└── tsconfig.json + +mcp-server/ # 独立 MCP 包（用于 PR 提交） +├── src/index.ts +├── package.json +└── README.md +``` + +--- + +## 四、功能清单 + +### CLI 命令 + +| 命令 | 状态 | 说明 | +|------|------|------| +| `status` | ✅ | 统计 wiki 页面数量/类型 | +| `search ` | ✅ | BM25+RRF 全文搜索 | +| `graph` | ✅ | 构建并输出知识图谱（Louvain 社区检测）| +| `insights` | ✅ | 发现意外关联 + 知识盲点 | +| `lint` | ✅ | 检测孤立页面/断链/缺失字段 | +| `init` | ✅ | 初始化 wiki 目录结构 | +| `ingest ` | 🔄 | LLM 自动摄入源文件 → wiki 页面 | +| `deep-research ` | 🔄 | 网络搜索 → LLM 综合 → 自动摄入 | + +### MCP 工具 + +| 工具 | 状态 | 说明 | +|------|------|------| +| `wiki_status` | ✅ | 获取 wiki 统计 | +| `wiki_search` | ✅ | 搜索 wiki 页面 | +| `wiki_graph` | ✅ | 获取知识图谱 | +| `wiki_insights` | ✅ | 获取 AI 见解 | +| `wiki_lint` | ✅ | 检查 wiki 健康度 | +| `wiki_ingest` | 🔄 | 摄入源文件 | +| `wiki_deep_research` | 🔄 | 深度研究 | + +--- + +## 五、环境变量配置 + +```bash +# LLM 配置（ingest / deep-research 必需） +export LLM_PROVIDER=openai # openai | anthropic | ollama | deepseek +export OPENAI_API_KEY=sk-... +export LLM_MODEL=gpt-4o-mini +export LLM_BASE_URL= # 自定义端点（可选） + +# 网络搜索（deep-research 必需） +export TAVILY_API_KEY=tvly-... + +# 输出语言（可选，默认 auto 自动检测） +export WIKI_OUTPUT_LANGUAGE=auto # auto | English | Chinese | Japanese | ... + +# 调试 +export SKILL_VERBOSE=1 # 输出详细日志到 stderr +``` + +--- + +## 六、依赖 + +```json +{ + "dependencies": { + "graphology": "^0.25.4", + "graphology-communities-louvain": "^2.0.0", + "@modelcontextprotocol/sdk": "^1.1.0", + "js-yaml": "^4.1.0" + } +} +``` + +--- + +## 七、开发进度 + +### 已完成 + +- [x] 分析 nashsu/llm_wiki 架构，识别 Tauri 注入点 +- [x] 创建 `shims/fs-node.ts` — Tauri IPC → Node.js fs 适配 +- [x] 创建 `shims/stores-node.ts` — Zustand → 模块级状态，支持 env 配置 LLM +- [x] 创建 `shims/embedding-stub.ts` — 向量搜索优雅降级 +- [x] 移植并修补所有图谱库（graph-relevance, wiki-graph, graph-insights） +- [x] 移植搜索库（BM25+RRF，向量可选） +- [x] 移植 path-utils（纯工具函数） +- [x] 实现 CLI 6 个命令：status/search/graph/insights/lint/init +- [x] 实现 MCP 服务器 5 个工具 +- [x] npm install + tsc 构建通过 +- [x] 端到端测试：合成 wiki 数据验证所有命令 +- [x] Fork nashsu/llm_wiki → toughhou/llm_wiki +- [x] 移植 llm-client.ts（OpenAI 兼容 SSE 流式调用） +- [x] 移植 detect-language.ts（Unicode 脚本检测） +- [x] 移植 output-language.ts +- [x] 移植 frontmatter.ts（js-yaml 解析） +- [x] 移植 sources-merge.ts（数组字段合并） +- [x] 移植 page-merge.ts（LLM 辅助页面合并） +- [x] 移植 ingest-sanitize.ts（LLM 输出清洗） +- [x] 移植 ingest-cache.ts（SHA256 增量缓存） +- [x] 移植 project-mutex.ts（并发保护） +- [x] 移植 web-search.ts（Tavily API） +- [x] 提交 PR #117 到 nashsu/llm_wiki + +### 进行中 + +- [ ] 完成 ingest.ts — 两阶段 LLM 流水线（分析 → 生成 → 写文件） +- [ ] 完成 deep-research.ts — 网络搜索 → LLM 综合 → auto-ingest +- [ ] CLI 添加 ingest / deep-research 命令 +- [ ] MCP 服务器添加 wiki_ingest / wiki_deep_research 工具 +- [ ] 端到端测试（需真实 LLM API Key） +- [ ] 更新 PR #117 + +### 待完成 + +- [ ] sweep-reviews（批量审核 wiki 页面） +- [ ] 嵌入向量搜索（可选，需 embedding API） + +--- + +## 八、PR 提交记录 + +| PR | 仓库 | 分支 | 状态 | +|----|------|------|------| +| #117 | nashsu/llm_wiki | feat/mcp-server | 开放中，待更新 | + +--- + +## 九、本地测试方法 + +```bash +cd skill && npm install && npm run build + +# 测试基础命令 +node dist/cli.js status /path/to/wiki-project +node dist/cli.js search "machine learning" /path/to/wiki-project +node dist/cli.js graph /path/to/wiki-project +node dist/cli.js insights /path/to/wiki-project +node dist/cli.js lint /path/to/wiki-project + +# 测试 ingest（需 LLM API Key） +export OPENAI_API_KEY=sk-xxx +node dist/cli.js ingest /path/to/source.md /path/to/wiki-project + +# 测试 deep-research（需 LLM + Tavily） +export TAVILY_API_KEY=tvly-xxx +node dist/cli.js deep-research "transformer architecture" /path/to/wiki-project + +# 启动 MCP 服务器 +node dist/mcp-server.js +``` + +--- + +## 十、相关资源 + +- 上游仓库：https://github.com/nashsu/llm_wiki +- 本仓库（fork）：https://github.com/toughhou/llm_wiki +- PR #117：https://github.com/nashsu/llm_wiki/pull/117 +- bid-sys 项目：https://github.com/toughhou/bid-sys diff --git a/skill/src/lib/detect-language.ts b/skill/src/lib/detect-language.ts new file mode 100644 index 00000000..925c7798 --- /dev/null +++ b/skill/src/lib/detect-language.ts @@ -0,0 +1,52 @@ +/** + * Language detection — ported from nashsu/llm_wiki src/lib/detect-language.ts + * Pure function, no external dependencies. + */ +export function detectLanguage(text: string): string { + const counts: Record = {} + for (const ch of text) { + const cp = ch.codePointAt(0) + if (!cp || cp < 0x80) continue + const script = getScript(cp) + if (script) counts[script] = (counts[script] ?? 0) + 1 + } + + if ((counts.Japanese ?? 0) > 0 && (counts.Chinese ?? 0) > 0) return "Japanese" + + let maxScript = ""; let maxCount = 0 + for (const [script, count] of Object.entries(counts)) { + if (count > maxCount) { maxScript = script; maxCount = count } + } + if (maxScript && maxCount >= 2) return maxScript + + const latinLang = detectLatinLanguage(text) + if (latinLang) return latinLang + return "English" +} + +function getScript(cp: number): string | null { + if ((cp >= 0x4E00 && cp <= 0x9FFF) || (cp >= 0x3400 && cp <= 0x4DBF) || (cp >= 0x20000 && cp <= 0x2A6DF) || (cp >= 0xF900 && cp <= 0xFAFF)) return "Chinese" + if ((cp >= 0x3040 && cp <= 0x309F) || (cp >= 0x30A0 && cp <= 0x30FF) || (cp >= 0x31F0 && cp <= 0x31FF) || (cp >= 0xFF65 && cp <= 0xFF9F)) return "Japanese" + if ((cp >= 0xAC00 && cp <= 0xD7AF) || (cp >= 0x1100 && cp <= 0x11FF) || (cp >= 0x3130 && cp <= 0x318F)) return "Korean" + if ((cp >= 0x0600 && cp <= 0x06FF) || (cp >= 0x0750 && cp <= 0x077F) || (cp >= 0x08A0 && cp <= 0x08FF) || (cp >= 0xFB50 && cp <= 0xFDFF) || (cp >= 0xFE70 && cp <= 0xFEFF)) return "Arabic" + if ((cp >= 0x0590 && cp <= 0x05FF) || (cp >= 0xFB1D && cp <= 0xFB4F)) return "Hebrew" + if (cp >= 0x0E00 && cp <= 0x0E7F) return "Thai" + if (cp >= 0x0900 && cp <= 0x097F) return "Hindi" + if ((cp >= 0x0400 && cp <= 0x04FF) || (cp >= 0x0500 && cp <= 0x052F)) return "Russian" + if ((cp >= 0x0370 && cp <= 0x03FF) || (cp >= 0x1F00 && cp <= 0x1FFF)) return "Greek" + return null +} + +function detectLatinLanguage(text: string): string | null { + const lower = text.toLowerCase() + if (/[ảạắằẳẵặấầẩẫậđẻẽẹếềểễệỉĩịỏọốồổỗộơớờởỡợủũụưứừửữựỷỹỵ]/.test(lower)) return "Vietnamese" + if (/[ğış]/.test(lower) && /\b(bir|ve|için|ile|bu|da|de|değil|ama)\b/.test(lower)) return "Turkish" + if (/[ąćęłńóśźż]/.test(lower)) return "Polish" + if (/[ěšžřďťňů]/.test(lower)) return "Czech" + if (/[äöüß]/.test(lower) && /\b(und|der|die|das|ist)\b/.test(lower)) return "German" + if (/[àâçéèêëïîôùûüÿœæ]/.test(lower) && /\b(le|la|les|est|une|des)\b/.test(lower)) return "French" + if (/[ãõç]/.test(lower) && /\b(o|a|os|as|de|do|da|é|em|um|uma|não|que)\b/.test(lower)) return "Portuguese" + if ((/[áéíóúñ¿¡]/.test(lower) || /\b(el|la|los|las|de|del|es|en)\b/.test(lower)) && (/\b(el|los|las|del|por)\b/.test(lower) || /[ñ¿¡]/.test(lower))) return "Spanish" + if (/\b(il|della|gli|che|è)\b/.test(lower)) return "Italian" + return null +} diff --git a/skill/src/lib/frontmatter.ts b/skill/src/lib/frontmatter.ts new file mode 100644 index 00000000..b452e220 --- /dev/null +++ b/skill/src/lib/frontmatter.ts @@ -0,0 +1,70 @@ +/** + * Frontmatter parser — Node.js port. + * Uses js-yaml for YAML parsing. + */ +import yaml from "js-yaml" + +export type FrontmatterValue = string | string[] + +export interface FrontmatterParseResult { + frontmatter: Record | null + body: string + rawBlock: string +} + +const FM_BLOCK_STRICT_RE = /^---\s*\r?\n([\s\S]*?)\r?\n---\s*(?:\r?\n|$)/ + +export function parseFrontmatter(content: string): FrontmatterParseResult { + const strict = content.match(FM_BLOCK_STRICT_RE) + if (!strict) return { frontmatter: null, body: content, rawBlock: "" } + + const yamlPayload = strict[1] + const rawBlock = strict[0] + const body = content.slice(rawBlock.length) + + let parsed: unknown + try { + parsed = yaml.load(yamlPayload, { schema: yaml.JSON_SCHEMA }) + } catch { + try { + parsed = yaml.load(repairWikilinkLists(yamlPayload), { schema: yaml.JSON_SCHEMA }) + } catch { + return { frontmatter: null, body, rawBlock } + } + } + + return { frontmatter: normalize(parsed), body, rawBlock } +} + +function repairWikilinkLists(payload: string): string { + return payload + .split("\n") + .map((line) => { + const m = line.match(/^(\s*[A-Za-z_][\w-]*\s*:\s*)(\[\[[^\]]+\]\](?:\s*,\s*\[\[[^\]]+\]\])+)\s*$/) + if (!m) return line + const items = m[2].split(",").map((s) => s.trim()).filter(Boolean).map((s) => `"${s}"`).join(", ") + return `${m[1]}[${items}]` + }) + .join("\n") +} + +function normalize(parsed: unknown): Record | null { + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return null + const out: Record = {} + for (const [key, value] of Object.entries(parsed as Record)) { + if (Array.isArray(value)) { + out[key] = value.map((v) => stringifyScalar(v)) + continue + } + out[key] = stringifyScalar(value) + } + return out +} + +function stringifyScalar(v: unknown): string { + if (v === null || v === undefined) return "" + if (typeof v === "string") return v + if (typeof v === "number" || typeof v === "boolean") return String(v) + if (v instanceof Date) return v.toISOString().slice(0, 10) + try { return JSON.stringify(v) } catch { return String(v) } +} diff --git a/skill/src/lib/ingest-cache.ts b/skill/src/lib/ingest-cache.ts new file mode 100644 index 00000000..55ad2d23 --- /dev/null +++ b/skill/src/lib/ingest-cache.ts @@ -0,0 +1,76 @@ +/** + * SHA256-based ingest cache — Node.js port. + * Uses Node.js crypto instead of WebCrypto. + */ +import * as crypto from "crypto" +import { readFile, writeFile, fileExists } from "../shims/fs-node" + +interface CacheEntry { + hash: string + timestamp: number + filesWritten: string[] +} + +interface CacheData { + entries: Record +} + +function sha256(content: string): string { + return crypto.createHash("sha256").update(content).digest("hex") +} + +function cachePath(projectPath: string): string { + return `${projectPath}/.llm-wiki/ingest-cache.json` +} + +async function loadCache(projectPath: string): Promise { + try { + const raw = await readFile(cachePath(projectPath)) + return JSON.parse(raw) as CacheData + } catch { + return { entries: {} } + } +} + +async function saveCache(projectPath: string, cache: CacheData): Promise { + try { + await writeFile(cachePath(projectPath), JSON.stringify(cache, null, 2)) + } catch { /* non-critical */ } +} + +export async function checkIngestCache( + projectPath: string, + sourceFileName: string, + sourceContent: string, +): Promise { + const cache = await loadCache(projectPath) + const entry = cache.entries[sourceFileName] + if (!entry) return null + + const currentHash = sha256(sourceContent) + if (entry.hash !== currentHash) return null + + for (const filePath of entry.filesWritten) { + try { + if (!(await fileExists(filePath.startsWith("/") ? filePath : `${projectPath}/${filePath}`))) { + return null + } + } catch { return null } + } + return entry.filesWritten +} + +export async function saveIngestCache( + projectPath: string, + sourceFileName: string, + sourceContent: string, + filesWritten: string[], +): Promise { + const cache = await loadCache(projectPath) + cache.entries[sourceFileName] = { + hash: sha256(sourceContent), + timestamp: Date.now(), + filesWritten, + } + await saveCache(projectPath, cache) +} diff --git a/skill/src/lib/ingest-sanitize.ts b/skill/src/lib/ingest-sanitize.ts new file mode 100644 index 00000000..960b96f2 --- /dev/null +++ b/skill/src/lib/ingest-sanitize.ts @@ -0,0 +1,44 @@ +/** + * Sanitize LLM-generated wiki page content. + * Ported from nashsu/llm_wiki — pure functions, no deps. + */ +export function sanitizeIngestedFileContent(content: string): string { + let cleaned = content + cleaned = stripOuterCodeFence(cleaned) + cleaned = stripFrontmatterKeyPrefix(cleaned) + cleaned = repairWikilinkListsInFrontmatter(cleaned) + return cleaned +} + +function stripOuterCodeFence(content: string): string { + const open = content.match(/^[ \t]*```(?:yaml|md|markdown)?[ \t]*\r?\n/) + if (!open) return content + const afterOpen = content.slice(open[0].length) + const close = afterOpen.match(/\r?\n[ \t]*```[ \t]*\r?\n?\s*$/) + if (!close) return content + return afterOpen.slice(0, close.index) +} + +function stripFrontmatterKeyPrefix(content: string): string { + const m = content.match(/^[ \t]*frontmatter\s*:\s*\r?\n(?=[ \t]*---\s*\r?\n)/) + if (!m) return content + return content.slice(m[0].length) +} + +function repairWikilinkListsInFrontmatter(content: string): string { + const fmRe = /^---\s*\r?\n([\s\S]*?)\r?\n---\s*(\r?\n|$)/ + const m = content.match(fmRe) + if (!m) return content + const repairedPayload = m[1] + .split("\n") + .map((line) => { + const lm = line.match( + /^(\s*[A-Za-z_][\w-]*\s*:\s*)(\[\[[^\]]+\]\](?:\s*,\s*\[\[[^\]]+\]\])+)\s*$/, + ) + if (!lm) return line + const items = lm[2].split(",").map((s) => s.trim()).filter(Boolean).map((s) => `"${s}"`).join(", ") + return `${lm[1]}[${items}]` + }) + .join("\n") + return content.slice(0, m.index! + 4) + repairedPayload + content.slice(m.index! + 4 + m[1].length) +} diff --git a/skill/src/lib/llm-client.ts b/skill/src/lib/llm-client.ts new file mode 100644 index 00000000..536b51eb --- /dev/null +++ b/skill/src/lib/llm-client.ts @@ -0,0 +1,165 @@ +/** + * Node.js llm-client — replaces Tauri HTTP plugin-based LLM streaming. + * Uses Node.js native fetch for HTTP requests. + */ +import type { LlmConfig } from "../shims/stores-node" + +export interface ChatMessage { + role: "system" | "user" | "assistant" + content: string +} + +export interface StreamCallbacks { + onToken: (token: string) => void + onDone: () => void + onError: (error: Error) => void +} + +export interface RequestOverrides { + temperature?: number + max_tokens?: number +} + +/** + * Build the request body for OpenAI-compatible APIs. + */ +function buildBody(config: LlmConfig, messages: ChatMessage[], overrides?: RequestOverrides) { + const body: Record = { + model: config.model, + messages: messages.map((m) => ({ role: m.role, content: m.content })), + stream: true, + } + if (overrides?.temperature !== undefined) body.temperature = overrides.temperature + if (overrides?.max_tokens !== undefined) body.max_tokens = overrides.max_tokens + return body +} + +/** + * Get the API endpoint URL. + */ +function getUrl(config: LlmConfig): string { + if (config.baseUrl) { + const base = config.baseUrl.replace(/\/+$/, "") + if (base.endsWith("/chat/completions")) return base + if (base.endsWith("/v1")) return `${base}/chat/completions` + return `${base}/v1/chat/completions` + } + switch (config.provider) { + case "openai": return "https://api.openai.com/v1/chat/completions" + case "anthropic": return "https://api.anthropic.com/v1/messages" + default: return "https://api.openai.com/v1/chat/completions" + } +} + +/** + * Get the API headers. + */ +function getHeaders(config: LlmConfig): Record { + const headers: Record = { "Content-Type": "application/json" } + if (config.apiKey) { + headers["Authorization"] = `Bearer ${config.apiKey}` + } + return headers +} + +/** + * Stream chat completion from an OpenAI-compatible API. + * Replaces Tauri-based streamChat with Node.js native fetch + SSE parsing. + */ +export async function streamChat( + config: LlmConfig, + messages: ChatMessage[], + callbacks: StreamCallbacks, + signal?: AbortSignal, + requestOverrides?: RequestOverrides, +): Promise { + const { onToken, onDone, onError } = callbacks + + const url = getUrl(config) + const headers = getHeaders(config) + const body = buildBody(config, messages, requestOverrides) + + let response: Response + try { + response = await fetch(url, { + method: "POST", + headers, + body: JSON.stringify(body), + signal, + }) + } catch (err) { + if (signal?.aborted) { onDone(); return } + onError(err instanceof Error ? err : new Error(String(err))) + return + } + + if (!response.ok) { + let errorDetail = `HTTP ${response.status}: ${response.statusText}` + try { + const body = await response.text() + if (body) errorDetail += ` — ${body.slice(0, 500)}` + } catch { /* ignore */ } + onError(new Error(errorDetail)) + return + } + + if (!response.body) { + onError(new Error("Response body is null")) + return + } + + const reader = response.body.getReader() + const decoder = new TextDecoder() + let lineBuffer = "" + + try { + while (true) { + const { done, value } = await reader.read() + if (done) { + if (lineBuffer.trim()) { + const token = parseSSELine(lineBuffer.trim()) + if (token !== null) onToken(token) + } + break + } + + const text = lineBuffer + decoder.decode(value, { stream: true }) + const lines = text.split("\n") + lineBuffer = lines.pop() ?? "" + + for (const line of lines) { + const trimmed = line.trim() + if (!trimmed) continue + const token = parseSSELine(trimmed) + if (token !== null) onToken(token) + } + } + onDone() + } catch (err) { + if (signal?.aborted) { onDone(); return } + onError(err instanceof Error ? err : new Error(String(err))) + } finally { + reader.releaseLock() + } +} + +/** + * Parse a single SSE line and extract the content delta token. + */ +function parseSSELine(line: string): string | null { + if (!line.startsWith("data: ")) return null + const data = line.slice(6).trim() + if (data === "[DONE]") return null + + try { + const parsed = JSON.parse(data) + // OpenAI format + const delta = parsed.choices?.[0]?.delta + if (delta?.content) return delta.content + // Some providers use different field names + if (typeof parsed.content === "string") return parsed.content + return null + } catch { + return null + } +} diff --git a/skill/src/lib/output-language.ts b/skill/src/lib/output-language.ts new file mode 100644 index 00000000..89865e20 --- /dev/null +++ b/skill/src/lib/output-language.ts @@ -0,0 +1,26 @@ +/** + * Output language directive builder. + * Ported from nashsu/llm_wiki — uses configurable language or auto-detection. + */ +import { detectLanguage } from "./detect-language" + +export function getOutputLanguage(fallbackText: string = ""): string { + const configured = process.env.WIKI_OUTPUT_LANGUAGE + if (configured && configured !== "auto") return configured + return detectLanguage(fallbackText || "English") +} + +export function buildLanguageDirective(fallbackText: string = ""): string { + const lang = getOutputLanguage(fallbackText) + return [ + `## ⚠️ MANDATORY OUTPUT LANGUAGE: ${lang}`, + "", + `You MUST write your entire response in **${lang}**.`, + `The source material may be in a different language, but generate everything in ${lang} only.`, + ].join("\n") +} + +export function buildLanguageReminder(fallbackText: string = ""): string { + const lang = getOutputLanguage(fallbackText) + return `REMINDER: All output must be in ${lang}.` +} diff --git a/skill/src/lib/page-merge.ts b/skill/src/lib/page-merge.ts new file mode 100644 index 00000000..40bb55fc --- /dev/null +++ b/skill/src/lib/page-merge.ts @@ -0,0 +1,93 @@ +/** + * Page merge — merges new wiki page content with existing page on disk. + * Ported from nashsu/llm_wiki — pure logic, LLM call injected as parameter. + */ +import { parseFrontmatter } from "./frontmatter" +import { mergeArrayFieldsIntoContent } from "./sources-merge" + +const UNION_FIELDS = ["sources", "tags", "related"] as const +const BODY_SHRINK_THRESHOLD = 0.7 + +export interface MergeFn { + (existingContent: string, incomingContent: string, sourceFileName: string, signal?: AbortSignal): Promise +} + +export interface MergePageOptions { + sourceFileName: string + pagePath: string + signal?: AbortSignal + backup?: (existingContent: string) => Promise + today?: () => string +} + +export async function mergePageContent( + newContent: string, + existingContent: string | null, + merger: MergeFn, + opts: MergePageOptions, +): Promise { + if (!existingContent) return newContent + if (newContent === existingContent) return existingContent + + const arrayMerged = mergeArrayFieldsIntoContent(newContent, existingContent, [...UNION_FIELDS]) + + const oldParsed = parseFrontmatter(existingContent) + const arrayMergedParsed = parseFrontmatter(arrayMerged) + if (oldParsed.body.trim() === arrayMergedParsed.body.trim()) return arrayMerged + + let llmOutput: string + try { + llmOutput = await merger(existingContent, arrayMerged, opts.sourceFileName, opts.signal) + } catch (err) { + console.warn(`[page-merge] LLM merge failed for ${opts.pagePath}, falling back: ${err instanceof Error ? err.message : err}`) + await tryBackup(opts, existingContent) + return arrayMerged + } + + const llmParsed = parseFrontmatter(llmOutput) + if (llmParsed.frontmatter === null) { + console.warn(`[page-merge] LLM output for ${opts.pagePath} has no frontmatter — rejecting`) + await tryBackup(opts, existingContent) + return arrayMerged + } + + const oldBodyLen = oldParsed.body.length + const newBodyLen = arrayMergedParsed.body.length + const llmBodyLen = llmParsed.body.length + const minThreshold = Math.max(oldBodyLen, newBodyLen) * BODY_SHRINK_THRESHOLD + if (llmBodyLen < minThreshold) { + console.warn(`[page-merge] LLM merge for ${opts.pagePath} produced ${llmBodyLen} chars below threshold ${minThreshold.toFixed(0)} — rejecting`) + await tryBackup(opts, existingContent) + return arrayMerged + } + + let final = llmOutput + for (const field of ["type", "title", "created"] as const) { + const existingValue = oldParsed.frontmatter?.[field] + if (typeof existingValue === "string" && existingValue !== "") { + final = setFrontmatterScalar(final, field, existingValue) + } + } + final = mergeArrayFieldsIntoContent(final, arrayMerged, [...UNION_FIELDS]) + final = setFrontmatterScalar(final, "updated", (opts.today ?? (() => new Date().toISOString().slice(0, 10)))()) + + return final +} + +async function tryBackup(opts: MergePageOptions, existingContent: string): Promise { + if (!opts.backup) return + try { await opts.backup(existingContent) } catch { /* ignore */ } +} + +function setFrontmatterScalar(content: string, fieldName: string, value: string): string { + const fmMatch = content.match(/^(---\n)([\s\S]*?)(\n---)/) + if (!fmMatch) return content + const [, openDelim, fmBody, closeDelim] = fmMatch + const escapedName = fieldName.replace(/[.*+?^${}()|[\]\\]/g, "\\$&") + const newLine = `${fieldName}: ${value}` + const lineRe = new RegExp(`^${escapedName}:\\s*(?!\\[)([^\\n]*)`, "m") + if (lineRe.test(fmBody)) { + return `${openDelim}${fmBody.replace(lineRe, newLine)}${closeDelim}${content.slice(fmMatch[0].length)}` + } + return `${openDelim}${fmBody}\n${newLine}${closeDelim}${content.slice(fmMatch[0].length)}` +} diff --git a/skill/src/lib/project-mutex.ts b/skill/src/lib/project-mutex.ts new file mode 100644 index 00000000..04bc7938 --- /dev/null +++ b/skill/src/lib/project-mutex.ts @@ -0,0 +1,30 @@ +/** + * Per-project async mutex — Node.js port. + * Ensures only one ingest runs at a time per project path. + */ +const locks = new Map>() + +export async function withProjectLock( + projectPath: string, + fn: () => Promise, +): Promise { + const prev = locks.get(projectPath) ?? Promise.resolve() + let release!: () => void + const next = new Promise((resolve) => { release = resolve }) + locks.set(projectPath, prev.then(() => next)) + + try { + await prev.catch(() => {}) + return await fn() + } finally { + release() + if (locks.size > 128) { + const tail = locks.get(projectPath) + if (tail) { + Promise.resolve().then(() => { + if (locks.get(projectPath) === tail) locks.delete(projectPath) + }) + } + } + } +} diff --git a/skill/src/lib/sources-merge.ts b/skill/src/lib/sources-merge.ts new file mode 100644 index 00000000..98ddadda --- /dev/null +++ b/skill/src/lib/sources-merge.ts @@ -0,0 +1,80 @@ +/** + * Frontmatter array-field merging during ingest. + * Ported from nashsu/llm_wiki — pure functions. + */ + +export function parseFrontmatterArray(content: string, fieldName: string): string[] { + const fmMatch = content.match(/^---\n([\s\S]*?)\n---/) + if (!fmMatch) return [] + const fm = fmMatch[1] + const escapedName = fieldName.replace(/[.*+?^${}()|[\]\\]/g, "\\$&") + + const blockRe = new RegExp(`^${escapedName}:\\s*\\n((?:[ \\t]+-\\s+.+\\n?)+)`, "m") + const block = fm.match(blockRe) + if (block) { + const out: string[] = [] + for (const line of block[1].split("\n")) { + const m = line.match(/^\s+-\s+["']?(.+?)["']?\s*$/) + if (m && m[1]) out.push(m[1].trim()) + } + return out + } + + const inlineRe = new RegExp(`^${escapedName}:\\s*\\[([^\\]]*)\\]`, "m") + const inline = fm.match(inlineRe) + if (!inline) return [] + const body = inline[1].trim() + if (body === "") return [] + return body.split(",").map((s) => s.trim().replace(/^["']|["']$/g, "")).filter((s) => s.length > 0) +} + +export function writeFrontmatterArray(content: string, fieldName: string, values: string[]): string { + const fmMatch = content.match(/^(---\n)([\s\S]*?)(\n---)/) + if (!fmMatch) return content + const [, openDelim, fmBody, closeDelim] = fmMatch + const escapedName = fieldName.replace(/[.*+?^${}()|[\]\\]/g, "\\$&") + const serialized = values.map((s) => `"${s}"`).join(", ") + const newLine = `${fieldName}: [${serialized}]` + + const inlineRe = new RegExp(`^${escapedName}:\\s*\\[[^\\]]*\\]`, "m") + if (inlineRe.test(fmBody)) { + return `${openDelim}${fmBody.replace(inlineRe, newLine)}${closeDelim}${content.slice(fmMatch[0].length)}` + } + + const blockRe = new RegExp(`^${escapedName}:\\s*\\n((?:[ \\t]+-\\s+.+\\n?)+)`, "m") + if (blockRe.test(fmBody)) { + return `${openDelim}${fmBody.replace(blockRe, newLine)}${closeDelim}${content.slice(fmMatch[0].length)}` + } + + return `${openDelim}${fmBody}\n${newLine}${closeDelim}${content.slice(fmMatch[0].length)}` +} + +function mergeLists(existing: readonly string[], incoming: readonly string[]): string[] { + const seen = new Set() + const out: string[] = [] + for (const s of [...existing, ...incoming]) { + const key = s.toLowerCase() + if (seen.has(key)) continue + seen.add(key) + out.push(s) + } + return out +} + +export function mergeArrayFieldsIntoContent(newContent: string, existingContent: string | null, fields: readonly string[]): string { + if (!existingContent) return newContent + if (!/^---\n/.test(existingContent)) return newContent + + let result = newContent + let changed = false + for (const field of fields) { + const oldValues = parseFrontmatterArray(existingContent, field) + if (oldValues.length === 0) continue + const newValues = parseFrontmatterArray(result, field) + const merged = mergeLists(oldValues, newValues) + if (merged.length === newValues.length && merged.every((s, i) => s === newValues[i])) continue + result = writeFrontmatterArray(result, field, merged) + changed = true + } + return changed ? result : newContent +} diff --git a/skill/src/lib/web-search.ts b/skill/src/lib/web-search.ts new file mode 100644 index 00000000..4eb8f408 --- /dev/null +++ b/skill/src/lib/web-search.ts @@ -0,0 +1,57 @@ +/** + * Web search via Tavily API — Node.js port. + * No Tauri dependencies, uses native fetch. + */ + +export interface SearchResult { + title: string + url: string + content: string + score: number +} + +export interface SearchResponse { + results: SearchResult[] + query: string +} + +export async function webSearch(query: string, maxResults: number = 5): Promise { + const apiKey = process.env.TAVILY_API_KEY + if (!apiKey) { + console.warn("[web-search] TAVILY_API_KEY not set, returning empty results") + return { results: [], query } + } + + try { + const response = await fetch("https://api.tavily.com/search", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + api_key: apiKey, + query, + max_results: maxResults, + search_depth: "advanced", + include_answer: false, + }), + }) + + if (!response.ok) { + const errText = await response.text().catch(() => "") + console.error(`[web-search] Tavily API error ${response.status}: ${errText.slice(0, 200)}`) + return { results: [], query } + } + + const data = await response.json() as Record + const results = ((data.results ?? []) as Array>).map((r) => ({ + title: (r.title as string) || "", + url: (r.url as string) || "", + content: (r.content as string) || "", + score: (r.score as number) || 0, + })) + + return { results, query } + } catch (err) { + console.error(`[web-search] failed: ${err instanceof Error ? err.message : err}`) + return { results: [], query } + } +} From 04b3202d93589a5388dcbde7642fa63d11548ea4 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 2 May 2026 10:32:11 +0000 Subject: [PATCH 4/6] Add ingest/deep-research lib, CLI/MCP commands, real e2e test harness Agent-Logs-Url: https://github.com/toughhou/llm_wiki/sessions/63c92ae1-4eba-4083-9087-deb3a07b750c Co-authored-by: toughhou <7472236+toughhou@users.noreply.github.com> --- mcp-server/README.md | 78 -- mcp-server/package-lock.json | 1269 ---------------------- mcp-server/package.json | 26 - mcp-server/src/index.ts | 251 ----- mcp-server/src/lib/graph-insights.ts | 150 --- mcp-server/src/lib/graph-relevance.ts | 229 ---- mcp-server/src/lib/path-utils.ts | 38 - mcp-server/src/lib/search.ts | 228 ---- mcp-server/src/lib/wiki-graph.ts | 211 ---- mcp-server/src/shims/embedding-stub.ts | 32 - mcp-server/src/shims/fs-node.ts | 101 -- mcp-server/src/shims/stores-node.ts | 135 --- mcp-server/src/types/wiki.ts | 18 - mcp-server/tsconfig.json | 18 - skill/package-lock.json | 28 +- skill/package.json | 4 +- skill/src/cli.ts | 101 +- skill/src/fs-node.ts | 156 --- skill/src/lib/deep-research.ts | 222 ++++ skill/src/lib/ingest.ts | 616 +++++++++++ skill/src/lib/web-search.ts | 3 +- skill/src/mcp-server.ts | 84 ++ skill/src/stores-node.ts | 160 --- skill/src/test-server/e2e.ts | 641 +++++++++++ skill/src/test-server/fake-llm-server.ts | 167 +++ 25 files changed, 1851 insertions(+), 3115 deletions(-) delete mode 100644 mcp-server/README.md delete mode 100644 mcp-server/package-lock.json delete mode 100644 mcp-server/package.json delete mode 100644 mcp-server/src/index.ts delete mode 100644 mcp-server/src/lib/graph-insights.ts delete mode 100644 mcp-server/src/lib/graph-relevance.ts delete mode 100644 mcp-server/src/lib/path-utils.ts delete mode 100644 mcp-server/src/lib/search.ts delete mode 100644 mcp-server/src/lib/wiki-graph.ts delete mode 100644 mcp-server/src/shims/embedding-stub.ts delete mode 100644 mcp-server/src/shims/fs-node.ts delete mode 100644 mcp-server/src/shims/stores-node.ts delete mode 100644 mcp-server/src/types/wiki.ts delete mode 100644 mcp-server/tsconfig.json delete mode 100644 skill/src/fs-node.ts create mode 100644 skill/src/lib/deep-research.ts create mode 100644 skill/src/lib/ingest.ts delete mode 100644 skill/src/stores-node.ts create mode 100644 skill/src/test-server/e2e.ts create mode 100644 skill/src/test-server/fake-llm-server.ts diff --git a/mcp-server/README.md b/mcp-server/README.md deleted file mode 100644 index bcdeb72f..00000000 --- a/mcp-server/README.md +++ /dev/null @@ -1,78 +0,0 @@ -# llm_wiki MCP Server - -An [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server that exposes **llm_wiki** backend operations as AI-callable tools. - -Use it with Claude Desktop, VS Code Copilot Chat, Cursor, or any MCP-compatible host to give your AI assistant direct access to your wiki knowledge base. - -## Tools - -| Tool | Description | -|------|-------------| -| `wiki_status` | Page count and type breakdown | -| `wiki_search` | BM25 keyword search (+ optional vector via `EMBEDDING_ENABLED=true`) | -| `wiki_graph` | Build Louvain knowledge graph — nodes, edges, community clusters | -| `wiki_insights` | Find surprising cross-community connections + knowledge gaps | -| `wiki_lint` | Structural lint: orphaned pages, isolated nodes, broken links | - -## Quick Start - -```bash -cd mcp-server -npm install -npm run build -``` - -### Claude Desktop - -Add to `~/Library/Application Support/Claude/claude_desktop_config.json`: - -```json -{ - "mcpServers": { - "llm-wiki": { - "command": "node", - "args": ["/path/to/llm_wiki/mcp-server/dist/index.js"], - "env": { - "WIKI_PATH": "/path/to/your/wiki-project" - } - } - } -} -``` - -### VS Code Copilot (`.vscode/mcp.json`) - -```json -{ - "servers": { - "llm-wiki": { - "type": "stdio", - "command": "node", - "args": ["${workspaceFolder}/mcp-server/dist/index.js"], - "env": { "WIKI_PATH": "${workspaceFolder}" } - } - } -} -``` - -## Environment Variables - -| Variable | Description | Default | -|----------|-------------|---------| -| `WIKI_PATH` | Default project path (used when `project_path` not specified) | `process.cwd()` | -| `EMBEDDING_ENABLED` | Enable vector search via LanceDB | `false` | -| `EMBEDDING_MODEL` | Embedding model name (e.g. `text-embedding-3-small`) | — | -| `OPENAI_API_KEY` | API key for LLM + embedding calls | — | -| `SKILL_VERBOSE` | Set to `1` for verbose activity logging | — | - -## Architecture - -The MCP server runs entirely in Node.js without the Tauri desktop app. It replaces the Tauri IPC layer (`@/commands/fs`) with standard Node.js `fs` operations, making it suitable for headless server and CI/CD environments. - -**Capabilities without Tauri**: -- ✅ `wiki_search` — BM25 keyword search -- ✅ `wiki_graph` — Louvain community detection -- ✅ `wiki_insights` — Surprising connections + knowledge gaps -- ✅ `wiki_lint` — Structural lint -- ⚠️ `wiki_search` with vector — requires `EMBEDDING_ENABLED=true` + configured API -- ❌ `ingest` — PDF/DOCX extraction not supported (use pre-converted Markdown) diff --git a/mcp-server/package-lock.json b/mcp-server/package-lock.json deleted file mode 100644 index 68e090ca..00000000 --- a/mcp-server/package-lock.json +++ /dev/null @@ -1,1269 +0,0 @@ -{ - "name": "llm-wiki-mcp-server", - "version": "0.4.6", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "name": "llm-wiki-mcp-server", - "version": "0.4.6", - "license": "MIT", - "dependencies": { - "@modelcontextprotocol/sdk": "^1.0.0", - "graphology": "^0.25.4", - "graphology-communities-louvain": "^2.0.0" - }, - "bin": { - "llm-wiki-mcp": "dist/index.js" - }, - "devDependencies": { - "@types/node": "^20.0.0", - "typescript": "^5.0.0" - } - }, - "node_modules/@hono/node-server": { - "version": "1.19.14", - "resolved": "https://registry.npmjs.org/@hono/node-server/-/node-server-1.19.14.tgz", - "integrity": "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw==", - "license": "MIT", - "engines": { - "node": ">=18.14.1" - }, - "peerDependencies": { - "hono": "^4" - } - }, - "node_modules/@modelcontextprotocol/sdk": { - "version": "1.29.0", - "resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.29.0.tgz", - "integrity": "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ==", - "license": "MIT", - "dependencies": { - "@hono/node-server": "^1.19.9", - "ajv": "^8.17.1", - "ajv-formats": "^3.0.1", - "content-type": "^1.0.5", - "cors": "^2.8.5", - "cross-spawn": "^7.0.5", - "eventsource": "^3.0.2", - "eventsource-parser": "^3.0.0", - "express": "^5.2.1", - "express-rate-limit": "^8.2.1", - "hono": "^4.11.4", - "jose": "^6.1.3", - "json-schema-typed": "^8.0.2", - "pkce-challenge": "^5.0.0", - "raw-body": "^3.0.0", - "zod": "^3.25 || ^4.0", - "zod-to-json-schema": "^3.25.1" - }, - "engines": { - "node": ">=18" - }, - "peerDependencies": { - "@cfworker/json-schema": "^4.1.1", - "zod": "^3.25 || ^4.0" - }, - "peerDependenciesMeta": { - "@cfworker/json-schema": { - "optional": true - }, - "zod": { - "optional": false - } - } - }, - "node_modules/@types/node": { - "version": "20.19.39", - "resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.39.tgz", - "integrity": "sha512-orrrD74MBUyK8jOAD/r0+lfa1I2MO6I+vAkmAWzMYbCcgrN4lCrmK52gRFQq/JRxfYPfonkr4b0jcY7Olqdqbw==", - "dev": true, - "license": "MIT", - "dependencies": { - "undici-types": "~6.21.0" - } - }, - "node_modules/accepts": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz", - "integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==", - "license": "MIT", - "dependencies": { - "mime-types": "^3.0.0", - "negotiator": "^1.0.0" - }, - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/ajv": { - "version": "8.20.0", - "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", - "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", - "license": "MIT", - "dependencies": { - "fast-deep-equal": "^3.1.3", - "fast-uri": "^3.0.1", - "json-schema-traverse": "^1.0.0", - "require-from-string": "^2.0.2" - }, - "funding": { - "type": "github", - "url": "https://github.com/sponsors/epoberezkin" - } - }, - "node_modules/ajv-formats": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/ajv-formats/-/ajv-formats-3.0.1.tgz", - "integrity": "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==", - "license": "MIT", - "dependencies": { - "ajv": "^8.0.0" - }, - "peerDependencies": { - "ajv": "^8.0.0" - }, - "peerDependenciesMeta": { - "ajv": { - "optional": true - } - } - }, - "node_modules/body-parser": { - "version": "2.2.2", - "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.2.tgz", - "integrity": "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA==", - "license": "MIT", - "dependencies": { - "bytes": "^3.1.2", - "content-type": "^1.0.5", - "debug": "^4.4.3", - "http-errors": "^2.0.0", - "iconv-lite": "^0.7.0", - "on-finished": "^2.4.1", - "qs": "^6.14.1", - "raw-body": "^3.0.1", - "type-is": "^2.0.1" - }, - "engines": { - "node": ">=18" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/bytes": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", - "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/call-bind-apply-helpers": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", - "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "function-bind": "^1.1.2" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/call-bound": { - "version": "1.0.4", - "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz", - "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==", - "license": "MIT", - "dependencies": { - "call-bind-apply-helpers": "^1.0.2", - "get-intrinsic": "^1.3.0" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/content-disposition": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.1.0.tgz", - "integrity": "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g==", - "license": "MIT", - "engines": { - "node": ">=18" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/content-type": { - "version": "1.0.5", - "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz", - "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/cookie": { - "version": "0.7.2", - "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", - "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/cookie-signature": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz", - "integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==", - "license": "MIT", - "engines": { - "node": ">=6.6.0" - } - }, - "node_modules/cors": { - "version": "2.8.6", - "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.6.tgz", - "integrity": "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==", - "license": "MIT", - "dependencies": { - "object-assign": "^4", - "vary": "^1" - }, - "engines": { - "node": ">= 0.10" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/cross-spawn": { - "version": "7.0.6", - "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", - "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", - "license": "MIT", - "dependencies": { - "path-key": "^3.1.0", - "shebang-command": "^2.0.0", - "which": "^2.0.1" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/depd": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", - "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/dunder-proto": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", - "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", - "license": "MIT", - "dependencies": { - "call-bind-apply-helpers": "^1.0.1", - "es-errors": "^1.3.0", - "gopd": "^1.2.0" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/ee-first": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", - "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==", - "license": "MIT" - }, - "node_modules/encodeurl": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", - "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/es-define-property": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", - "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/es-errors": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", - "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/es-object-atoms": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz", - "integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/escape-html": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", - "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", - "license": "MIT" - }, - "node_modules/etag": { - "version": "1.8.1", - "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", - "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/events": { - "version": "3.3.0", - "resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz", - "integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==", - "license": "MIT", - "engines": { - "node": ">=0.8.x" - } - }, - "node_modules/eventsource": { - "version": "3.0.7", - "resolved": "https://registry.npmjs.org/eventsource/-/eventsource-3.0.7.tgz", - "integrity": "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA==", - "license": "MIT", - "dependencies": { - "eventsource-parser": "^3.0.1" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "node_modules/eventsource-parser": { - "version": "3.0.8", - "resolved": "https://registry.npmjs.org/eventsource-parser/-/eventsource-parser-3.0.8.tgz", - "integrity": "sha512-70QWGkr4snxr0OXLRWsFLeRBIRPuQOvt4s8QYjmUlmlkyTZkRqS7EDVRZtzU3TiyDbXSzaOeF0XUKy8PchzukQ==", - "license": "MIT", - "engines": { - "node": ">=18.0.0" - } - }, - "node_modules/express": { - "version": "5.2.1", - "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", - "integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==", - "license": "MIT", - "dependencies": { - "accepts": "^2.0.0", - "body-parser": "^2.2.1", - "content-disposition": "^1.0.0", - "content-type": "^1.0.5", - "cookie": "^0.7.1", - "cookie-signature": "^1.2.1", - "debug": "^4.4.0", - "depd": "^2.0.0", - "encodeurl": "^2.0.0", - "escape-html": "^1.0.3", - "etag": "^1.8.1", - "finalhandler": "^2.1.0", - "fresh": "^2.0.0", - "http-errors": "^2.0.0", - "merge-descriptors": "^2.0.0", - "mime-types": "^3.0.0", - "on-finished": "^2.4.1", - "once": "^1.4.0", - "parseurl": "^1.3.3", - "proxy-addr": "^2.0.7", - "qs": "^6.14.0", - "range-parser": "^1.2.1", - "router": "^2.2.0", - "send": "^1.1.0", - "serve-static": "^2.2.0", - "statuses": "^2.0.1", - "type-is": "^2.0.1", - "vary": "^1.1.2" - }, - "engines": { - "node": ">= 18" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/express-rate-limit": { - "version": "8.4.1", - "resolved": "https://registry.npmjs.org/express-rate-limit/-/express-rate-limit-8.4.1.tgz", - "integrity": "sha512-NGVYwQSAyEQgzxX1iCM978PP9AdO/hW93gMcF6ZwQCm+rFvLsBH6w4xcXWTcliS8La5EPRN3p9wzItqBwJrfNw==", - "license": "MIT", - "dependencies": { - "ip-address": "10.1.0" - }, - "engines": { - "node": ">= 16" - }, - "funding": { - "url": "https://github.com/sponsors/express-rate-limit" - }, - "peerDependencies": { - "express": ">= 4.11" - } - }, - "node_modules/fast-deep-equal": { - "version": "3.1.3", - "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz", - "integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==", - "license": "MIT" - }, - "node_modules/fast-uri": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.0.tgz", - "integrity": "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/fastify" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/fastify" - } - ], - "license": "BSD-3-Clause" - }, - "node_modules/finalhandler": { - "version": "2.1.1", - "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz", - "integrity": "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==", - "license": "MIT", - "dependencies": { - "debug": "^4.4.0", - "encodeurl": "^2.0.0", - "escape-html": "^1.0.3", - "on-finished": "^2.4.1", - "parseurl": "^1.3.3", - "statuses": "^2.0.1" - }, - "engines": { - "node": ">= 18.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/forwarded": { - "version": "0.2.0", - "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", - "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/fresh": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz", - "integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/function-bind": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", - "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/get-intrinsic": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", - "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", - "license": "MIT", - "dependencies": { - "call-bind-apply-helpers": "^1.0.2", - "es-define-property": "^1.0.1", - "es-errors": "^1.3.0", - "es-object-atoms": "^1.1.1", - "function-bind": "^1.1.2", - "get-proto": "^1.0.1", - "gopd": "^1.2.0", - "has-symbols": "^1.1.0", - "hasown": "^2.0.2", - "math-intrinsics": "^1.1.0" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/get-proto": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", - "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", - "license": "MIT", - "dependencies": { - "dunder-proto": "^1.0.1", - "es-object-atoms": "^1.0.0" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/gopd": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", - "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/graphology": { - "version": "0.25.4", - "resolved": "https://registry.npmjs.org/graphology/-/graphology-0.25.4.tgz", - "integrity": "sha512-33g0Ol9nkWdD6ulw687viS8YJQBxqG5LWII6FI6nul0pq6iM2t5EKquOTFDbyTblRB3O9I+7KX4xI8u5ffekAQ==", - "license": "MIT", - "dependencies": { - "events": "^3.3.0", - "obliterator": "^2.0.2" - }, - "peerDependencies": { - "graphology-types": ">=0.24.0" - } - }, - "node_modules/graphology-communities-louvain": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/graphology-communities-louvain/-/graphology-communities-louvain-2.0.2.tgz", - "integrity": "sha512-zt+2hHVPYxjEquyecxWXoUoIuN/UvYzsvI7boDdMNz0rRvpESQ7+e+Ejv6wK7AThycbZXuQ6DkG8NPMCq6XwoA==", - "license": "MIT", - "dependencies": { - "graphology-indices": "^0.17.0", - "graphology-utils": "^2.4.4", - "mnemonist": "^0.39.0", - "pandemonium": "^2.4.1" - }, - "peerDependencies": { - "graphology-types": ">=0.19.0" - } - }, - "node_modules/graphology-indices": { - "version": "0.17.0", - "resolved": "https://registry.npmjs.org/graphology-indices/-/graphology-indices-0.17.0.tgz", - "integrity": "sha512-A7RXuKQvdqSWOpn7ZVQo4S33O0vCfPBnUSf7FwE0zNCasqwZVUaCXePuWo5HBpWw68KJcwObZDHpFk6HKH6MYQ==", - "license": "MIT", - "dependencies": { - "graphology-utils": "^2.4.2", - "mnemonist": "^0.39.0" - }, - "peerDependencies": { - "graphology-types": ">=0.20.0" - } - }, - "node_modules/graphology-types": { - "version": "0.24.8", - "resolved": "https://registry.npmjs.org/graphology-types/-/graphology-types-0.24.8.tgz", - "integrity": "sha512-hDRKYXa8TsoZHjgEaysSRyPdT6uB78Ci8WnjgbStlQysz7xR52PInxNsmnB7IBOM1BhikxkNyCVEFgmPKnpx3Q==", - "license": "MIT", - "peer": true - }, - "node_modules/graphology-utils": { - "version": "2.5.2", - "resolved": "https://registry.npmjs.org/graphology-utils/-/graphology-utils-2.5.2.tgz", - "integrity": "sha512-ckHg8MXrXJkOARk56ZaSCM1g1Wihe2d6iTmz1enGOz4W/l831MBCKSayeFQfowgF8wd+PQ4rlch/56Vs/VZLDQ==", - "license": "MIT", - "peerDependencies": { - "graphology-types": ">=0.23.0" - } - }, - "node_modules/has-symbols": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", - "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/hasown": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.3.tgz", - "integrity": "sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg==", - "license": "MIT", - "dependencies": { - "function-bind": "^1.1.2" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/hono": { - "version": "4.12.16", - "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.16.tgz", - "integrity": "sha512-jN0ZewiNAWSe5khM3EyCmBb250+b40wWbwNILNfEvq84VREWwOIkuUsFONk/3i3nqkz7Oe1PcpM2mwQEK2L9Kg==", - "license": "MIT", - "engines": { - "node": ">=16.9.0" - } - }, - "node_modules/http-errors": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", - "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==", - "license": "MIT", - "dependencies": { - "depd": "~2.0.0", - "inherits": "~2.0.4", - "setprototypeof": "~1.2.0", - "statuses": "~2.0.2", - "toidentifier": "~1.0.1" - }, - "engines": { - "node": ">= 0.8" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/iconv-lite": { - "version": "0.7.2", - "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.2.tgz", - "integrity": "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw==", - "license": "MIT", - "dependencies": { - "safer-buffer": ">= 2.1.2 < 3.0.0" - }, - "engines": { - "node": ">=0.10.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/inherits": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", - "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", - "license": "ISC" - }, - "node_modules/ip-address": { - "version": "10.1.0", - "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.1.0.tgz", - "integrity": "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q==", - "license": "MIT", - "engines": { - "node": ">= 12" - } - }, - "node_modules/ipaddr.js": { - "version": "1.9.1", - "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", - "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==", - "license": "MIT", - "engines": { - "node": ">= 0.10" - } - }, - "node_modules/is-promise": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz", - "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==", - "license": "MIT" - }, - "node_modules/isexe": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", - "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", - "license": "ISC" - }, - "node_modules/jose": { - "version": "6.2.3", - "resolved": "https://registry.npmjs.org/jose/-/jose-6.2.3.tgz", - "integrity": "sha512-YYVDInQKFJfR/xa3ojUTl8c2KoTwiL1R5Wg9YCydwH0x0B9grbzlg5HC7mMjCtUJjbQ/YnGEZIhI5tCgfTb4Hw==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/panva" - } - }, - "node_modules/json-schema-traverse": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", - "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", - "license": "MIT" - }, - "node_modules/json-schema-typed": { - "version": "8.0.2", - "resolved": "https://registry.npmjs.org/json-schema-typed/-/json-schema-typed-8.0.2.tgz", - "integrity": "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA==", - "license": "BSD-2-Clause" - }, - "node_modules/math-intrinsics": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", - "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/media-typer": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.0.tgz", - "integrity": "sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/merge-descriptors": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz", - "integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==", - "license": "MIT", - "engines": { - "node": ">=18" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/mime-db": { - "version": "1.54.0", - "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", - "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/mime-types": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", - "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", - "license": "MIT", - "dependencies": { - "mime-db": "^1.54.0" - }, - "engines": { - "node": ">=18" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/mnemonist": { - "version": "0.39.8", - "resolved": "https://registry.npmjs.org/mnemonist/-/mnemonist-0.39.8.tgz", - "integrity": "sha512-vyWo2K3fjrUw8YeeZ1zF0fy6Mu59RHokURlld8ymdUPjMlD9EC9ov1/YPqTgqRvUN9nTr3Gqfz29LYAmu0PHPQ==", - "license": "MIT", - "dependencies": { - "obliterator": "^2.0.1" - } - }, - "node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/negotiator": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.0.0.tgz", - "integrity": "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/object-assign": { - "version": "4.1.1", - "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", - "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/object-inspect": { - "version": "1.13.4", - "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", - "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/obliterator": { - "version": "2.0.5", - "resolved": "https://registry.npmjs.org/obliterator/-/obliterator-2.0.5.tgz", - "integrity": "sha512-42CPE9AhahZRsMNslczq0ctAEtqk8Eka26QofnqC346BZdHDySk3LWka23LI7ULIw11NmltpiLagIq8gBozxTw==", - "license": "MIT" - }, - "node_modules/on-finished": { - "version": "2.4.1", - "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", - "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", - "license": "MIT", - "dependencies": { - "ee-first": "1.1.1" - }, - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/once": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", - "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", - "license": "ISC", - "dependencies": { - "wrappy": "1" - } - }, - "node_modules/pandemonium": { - "version": "2.4.1", - "resolved": "https://registry.npmjs.org/pandemonium/-/pandemonium-2.4.1.tgz", - "integrity": "sha512-wRqjisUyiUfXowgm7MFH2rwJzKIr20rca5FsHXCMNm1W5YPP1hCtrZfgmQ62kP7OZ7Xt+cR858aB28lu5NX55g==", - "license": "MIT", - "dependencies": { - "mnemonist": "^0.39.2" - } - }, - "node_modules/parseurl": { - "version": "1.3.3", - "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", - "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/path-key": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", - "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/path-to-regexp": { - "version": "8.4.2", - "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.4.2.tgz", - "integrity": "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA==", - "license": "MIT", - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/pkce-challenge": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/pkce-challenge/-/pkce-challenge-5.0.1.tgz", - "integrity": "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ==", - "license": "MIT", - "engines": { - "node": ">=16.20.0" - } - }, - "node_modules/proxy-addr": { - "version": "2.0.7", - "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", - "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==", - "license": "MIT", - "dependencies": { - "forwarded": "0.2.0", - "ipaddr.js": "1.9.1" - }, - "engines": { - "node": ">= 0.10" - } - }, - "node_modules/qs": { - "version": "6.15.1", - "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.1.tgz", - "integrity": "sha512-6YHEFRL9mfgcAvql/XhwTvf5jKcOiiupt2FiJxHkiX1z4j7WL8J/jRHYLluORvc1XxB5rV20KoeK00gVJamspg==", - "license": "BSD-3-Clause", - "dependencies": { - "side-channel": "^1.1.0" - }, - "engines": { - "node": ">=0.6" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/range-parser": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", - "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/raw-body": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.2.tgz", - "integrity": "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==", - "license": "MIT", - "dependencies": { - "bytes": "~3.1.2", - "http-errors": "~2.0.1", - "iconv-lite": "~0.7.0", - "unpipe": "~1.0.0" - }, - "engines": { - "node": ">= 0.10" - } - }, - "node_modules/require-from-string": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", - "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/router": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz", - "integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==", - "license": "MIT", - "dependencies": { - "debug": "^4.4.0", - "depd": "^2.0.0", - "is-promise": "^4.0.0", - "parseurl": "^1.3.3", - "path-to-regexp": "^8.0.0" - }, - "engines": { - "node": ">= 18" - } - }, - "node_modules/safer-buffer": { - "version": "2.1.2", - "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", - "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", - "license": "MIT" - }, - "node_modules/send": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz", - "integrity": "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==", - "license": "MIT", - "dependencies": { - "debug": "^4.4.3", - "encodeurl": "^2.0.0", - "escape-html": "^1.0.3", - "etag": "^1.8.1", - "fresh": "^2.0.0", - "http-errors": "^2.0.1", - "mime-types": "^3.0.2", - "ms": "^2.1.3", - "on-finished": "^2.4.1", - "range-parser": "^1.2.1", - "statuses": "^2.0.2" - }, - "engines": { - "node": ">= 18" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/serve-static": { - "version": "2.2.1", - "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz", - "integrity": "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==", - "license": "MIT", - "dependencies": { - "encodeurl": "^2.0.0", - "escape-html": "^1.0.3", - "parseurl": "^1.3.3", - "send": "^1.2.0" - }, - "engines": { - "node": ">= 18" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/setprototypeof": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", - "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==", - "license": "ISC" - }, - "node_modules/shebang-command": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", - "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", - "license": "MIT", - "dependencies": { - "shebang-regex": "^3.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/shebang-regex": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", - "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/side-channel": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz", - "integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "object-inspect": "^1.13.3", - "side-channel-list": "^1.0.0", - "side-channel-map": "^1.0.1", - "side-channel-weakmap": "^1.0.2" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/side-channel-list": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz", - "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "object-inspect": "^1.13.4" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/side-channel-map": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz", - "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==", - "license": "MIT", - "dependencies": { - "call-bound": "^1.0.2", - "es-errors": "^1.3.0", - "get-intrinsic": "^1.2.5", - "object-inspect": "^1.13.3" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/side-channel-weakmap": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz", - "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==", - "license": "MIT", - "dependencies": { - "call-bound": "^1.0.2", - "es-errors": "^1.3.0", - "get-intrinsic": "^1.2.5", - "object-inspect": "^1.13.3", - "side-channel-map": "^1.0.1" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/statuses": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", - "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/toidentifier": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", - "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==", - "license": "MIT", - "engines": { - "node": ">=0.6" - } - }, - "node_modules/type-is": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.0.1.tgz", - "integrity": "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw==", - "license": "MIT", - "dependencies": { - "content-type": "^1.0.5", - "media-typer": "^1.1.0", - "mime-types": "^3.0.0" - }, - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/typescript": { - "version": "5.9.3", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", - "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", - "dev": true, - "license": "Apache-2.0", - "bin": { - "tsc": "bin/tsc", - "tsserver": "bin/tsserver" - }, - "engines": { - "node": ">=14.17" - } - }, - "node_modules/undici-types": { - "version": "6.21.0", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", - "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", - "dev": true, - "license": "MIT" - }, - "node_modules/unpipe": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", - "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/vary": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", - "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==", - "license": "MIT", - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/which": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", - "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", - "license": "ISC", - "dependencies": { - "isexe": "^2.0.0" - }, - "bin": { - "node-which": "bin/node-which" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/wrappy": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", - "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", - "license": "ISC" - }, - "node_modules/zod": { - "version": "4.4.2", - "resolved": "https://registry.npmjs.org/zod/-/zod-4.4.2.tgz", - "integrity": "sha512-IynmDyxsEsb9RKzO3J9+4SxXnl2FTFSzNBaKKaMV6tsSk0rw9gYw9gs+JFCq/qk2LCZ78KDwyj+Z289TijSkUw==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/colinhacks" - } - }, - "node_modules/zod-to-json-schema": { - "version": "3.25.2", - "resolved": "https://registry.npmjs.org/zod-to-json-schema/-/zod-to-json-schema-3.25.2.tgz", - "integrity": "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA==", - "license": "ISC", - "peerDependencies": { - "zod": "^3.25.28 || ^4" - } - } - } -} diff --git a/mcp-server/package.json b/mcp-server/package.json deleted file mode 100644 index 523cd426..00000000 --- a/mcp-server/package.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "name": "llm-wiki-mcp-server", - "version": "0.4.6", - "description": "MCP server for llm_wiki — exposes wiki graph, search, insights, and lint as Model Context Protocol tools", - "main": "dist/index.js", - "bin": { - "llm-wiki-mcp": "dist/index.js" - }, - "scripts": { - "build": "tsc -p tsconfig.json", - "typecheck": "tsc -p tsconfig.json --noEmit", - "start": "node dist/index.js", - "dev": "ts-node src/index.ts" - }, - "keywords": ["mcp", "wiki", "knowledge-graph", "llm", "model-context-protocol"], - "license": "MIT", - "dependencies": { - "graphology": "^0.25.4", - "graphology-communities-louvain": "^2.0.0", - "@modelcontextprotocol/sdk": "^1.0.0" - }, - "devDependencies": { - "@types/node": "^20.0.0", - "typescript": "^5.0.0" - } -} diff --git a/mcp-server/src/index.ts b/mcp-server/src/index.ts deleted file mode 100644 index b07ee8df..00000000 --- a/mcp-server/src/index.ts +++ /dev/null @@ -1,251 +0,0 @@ -#!/usr/bin/env node -/** - * llm-wiki MCP Server - * - * Exposes nashsu/llm_wiki backend operations as Model Context Protocol tools. - * Works with Claude Desktop, VS Code Copilot, and any MCP-compatible host. - * - * Tools: - * wiki_status — Page count and type breakdown for a project - * wiki_search — BM25 keyword search (+ optional vector via EMBEDDING_ENABLED) - * wiki_graph — Build knowledge graph (nodes, edges, Louvain communities) - * wiki_insights — Surprising connections and knowledge gaps analysis - * wiki_lint — Structural lint: orphans, no-outlinks, broken links - * - * Usage: - * node dist/index.js - * WIKI_PATH=/path/to/project node dist/index.js (default project path) - */ -import { Server } from "@modelcontextprotocol/sdk/server/index.js" -import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js" -import { - CallToolRequestSchema, - ListToolsRequestSchema, - ErrorCode, - McpError, -} from "@modelcontextprotocol/sdk/types.js" - -import * as path from "path" -import { buildWikiGraph } from "./lib/wiki-graph" -import { findSurprisingConnections, detectKnowledgeGaps } from "./lib/graph-insights" -import { searchWiki } from "./lib/search" - -const DEFAULT_WIKI_PATH = process.env.WIKI_PATH ?? process.cwd() -const PKG_VERSION = "0.4.6-mcp" - -const server = new Server( - { name: "llm-wiki", version: PKG_VERSION }, - { capabilities: { tools: {} } }, -) - -// ── Tool definitions ────────────────────────────────────────────────────────── -server.setRequestHandler(ListToolsRequestSchema, async () => ({ - tools: [ - { - name: "wiki_status", - description: "Get page count and type breakdown for a wiki project. Returns statistics about the knowledge base.", - inputSchema: { - type: "object", - properties: { - project_path: { - type: "string", - description: "Absolute path to the wiki project directory (contains wiki/ subdirectory)", - }, - }, - required: [], - }, - }, - { - name: "wiki_search", - description: "Search wiki pages using BM25 keyword matching with optional vector search (RRF fusion). Returns ranked results with snippets.", - inputSchema: { - type: "object", - properties: { - query: { type: "string", description: "Search query (supports Chinese and English)" }, - project_path: { type: "string", description: "Path to wiki project (defaults to WIKI_PATH env var)" }, - limit: { type: "number", description: "Max results to return (default: 10)" }, - }, - required: ["query"], - }, - }, - { - name: "wiki_graph", - description: "Build knowledge graph from wiki pages: wikilinks, type-based edges, Louvain community detection. Returns nodes, edges, and community clusters.", - inputSchema: { - type: "object", - properties: { - project_path: { type: "string", description: "Path to wiki project" }, - format: { - type: "string", - enum: ["json", "summary"], - description: "Output format: 'json' for full graph data, 'summary' for human-readable overview (default: summary)", - }, - }, - required: [], - }, - }, - { - name: "wiki_insights", - description: "Analyze wiki graph structure to find surprising cross-community connections and knowledge gaps (isolated pages, sparse clusters, bridge nodes).", - inputSchema: { - type: "object", - properties: { - project_path: { type: "string", description: "Path to wiki project" }, - max_connections: { type: "number", description: "Max surprising connections to return (default: 5)" }, - max_gaps: { type: "number", description: "Max knowledge gaps to return (default: 8)" }, - }, - required: [], - }, - }, - { - name: "wiki_lint", - description: "Structural lint of wiki pages: find orphaned pages (no links), no-outlinks, and connectivity issues.", - inputSchema: { - type: "object", - properties: { - project_path: { type: "string", description: "Path to wiki project" }, - }, - required: [], - }, - }, - ], -})) - -// ── Tool handlers ───────────────────────────────────────────────────────────── -server.setRequestHandler(CallToolRequestSchema, async (request) => { - const { name, arguments: args = {} } = request.params - const projectPath = path.resolve((args.project_path as string | undefined) ?? DEFAULT_WIKI_PATH) - - try { - switch (name) { - case "wiki_status": { - const { nodes, communities } = await buildWikiGraph(projectPath) - const typeCounts: Record = {} - for (const n of nodes) typeCounts[n.type] = (typeCounts[n.type] ?? 0) + 1 - const summary = [ - `Wiki: ${projectPath}`, - `Total pages: ${nodes.length}`, - `Communities: ${communities.length}`, - ...Object.entries(typeCounts) - .sort((a, b) => b[1] - a[1]) - .map(([t, c]) => ` ${t}: ${c}`), - ].join("\n") - return { content: [{ type: "text", text: summary }] } - } - - case "wiki_search": { - if (!args.query) throw new McpError(ErrorCode.InvalidParams, "query is required") - const results = await searchWiki(projectPath, args.query as string) - const limit = typeof args.limit === "number" ? args.limit : 10 - const top = results.slice(0, limit) - if (top.length === 0) { - return { content: [{ type: "text", text: `No results for: "${args.query}"` }] } - } - const lines = [`# Search: "${args.query}"\n`] - for (const r of top) { - const relPath = path.relative(projectPath, r.path) - lines.push(`## ${r.title}`) - lines.push(`**Path**: ${relPath} | **Score**: ${r.score.toFixed(4)}`) - lines.push(r.snippet) - lines.push("") - } - return { content: [{ type: "text", text: lines.join("\n") }] } - } - - case "wiki_graph": { - const graphData = await buildWikiGraph(projectPath) - const format = (args.format as string | undefined) ?? "summary" - if (format === "json") { - return { content: [{ type: "text", text: JSON.stringify(graphData, null, 2) }] } - } - // Summary format - const { nodes, edges, communities } = graphData - const typeCounts: Record = {} - for (const n of nodes) typeCounts[n.type] = (typeCounts[n.type] ?? 0) + 1 - const lines = [ - `# Knowledge Graph Summary`, - ``, - `**Nodes**: ${nodes.length} | **Edges**: ${edges.length} | **Communities**: ${communities.length}`, - ``, - `## Node Types`, - ...Object.entries(typeCounts).sort((a, b) => b[1] - a[1]).map(([t, c]) => `- ${t}: ${c}`), - ``, - `## Top Communities`, - ...communities.slice(0, 5).map((c, i) => - `### Community ${i + 1} (${c.nodeCount} pages, cohesion: ${c.cohesion.toFixed(2)})\nKey pages: ${c.topNodes.join(", ")}` - ), - ``, - `## Top Hubs (by link count)`, - ...nodes.sort((a, b) => b.linkCount - a.linkCount).slice(0, 10) - .map((n) => `- ${n.label} (${n.type}, ${n.linkCount} links)`), - ] - return { content: [{ type: "text", text: lines.join("\n") }] } - } - - case "wiki_insights": { - const { nodes, edges, communities } = await buildWikiGraph(projectPath) - const maxConn = typeof args.max_connections === "number" ? args.max_connections : 5 - const maxGaps = typeof args.max_gaps === "number" ? args.max_gaps : 8 - const connections = findSurprisingConnections(nodes, edges, communities, maxConn) - const gaps = detectKnowledgeGaps(nodes, edges, communities, maxGaps) - - const lines = [`# Wiki Insights\n`, `## Surprising Connections\n`] - if (connections.length === 0) lines.push("_No surprising connections found yet._\n") - for (const c of connections) { - lines.push(`### ${c.source.label} ↔ ${c.target.label}`) - lines.push(`- Score: ${c.score} | ${c.reasons.join(", ")}\n`) - } - lines.push(`## Knowledge Gaps\n`) - if (gaps.length === 0) lines.push("_No gaps detected._\n") - for (const g of gaps) { - lines.push(`### ${g.title}`) - lines.push(`${g.description}`) - lines.push(`💡 ${g.suggestion}\n`) - } - return { content: [{ type: "text", text: lines.join("\n") }] } - } - - case "wiki_lint": { - const { nodes, edges } = await buildWikiGraph(projectPath) - if (nodes.length === 0) { - return { content: [{ type: "text", text: "No wiki pages found." }] } - } - const edgeTargets = new Set(edges.map((e) => e.target)) - const edgeSources = new Set(edges.map((e) => e.source)) - const allLinked = new Set([...edgeTargets, ...edgeSources]) - const issues: string[] = [] - for (const n of nodes) { - if (n.id === "index" || n.id === "log" || n.id === "overview") continue - if (!allLinked.has(n.id)) issues.push(`[orphan] ${n.label} (${n.id}.md)`) - else if (n.linkCount <= 1) issues.push(`[isolated] ${n.label} — only ${n.linkCount} link(s)`) - } - const text = issues.length === 0 - ? `✓ All ${nodes.length} pages are properly connected.` - : `Found ${issues.length} issue(s) in ${nodes.length} pages:\n\n${issues.join("\n")}` - return { content: [{ type: "text", text: text }] } - } - - default: - throw new McpError(ErrorCode.MethodNotFound, `Unknown tool: ${name}`) - } - } catch (err) { - if (err instanceof McpError) throw err - throw new McpError( - ErrorCode.InternalError, - `Tool '${name}' failed: ${err instanceof Error ? err.message : String(err)}`, - ) - } -}) - -// ── Start server ────────────────────────────────────────────────────────────── -async function main() { - const transport = new StdioServerTransport() - await server.connect(transport) - console.error(`llm-wiki MCP server v${PKG_VERSION} started`) - console.error(`Default wiki path: ${DEFAULT_WIKI_PATH}`) -} - -main().catch((err) => { - console.error("Failed to start MCP server:", err) - process.exit(1) -}) diff --git a/mcp-server/src/lib/graph-insights.ts b/mcp-server/src/lib/graph-insights.ts deleted file mode 100644 index 56fa212f..00000000 --- a/mcp-server/src/lib/graph-insights.ts +++ /dev/null @@ -1,150 +0,0 @@ -import type { GraphNode, GraphEdge, CommunityInfo } from "./wiki-graph" - -export interface SurprisingConnection { - source: GraphNode - target: GraphNode - score: number - reasons: string[] - key: string -} - -export interface KnowledgeGap { - type: "isolated-node" | "sparse-community" | "bridge-node" - title: string - description: string - nodeIds: string[] - suggestion: string -} - -export function findSurprisingConnections( - nodes: GraphNode[], - edges: GraphEdge[], - _communities: CommunityInfo[], - limit: number = 5, -): SurprisingConnection[] { - const nodeMap = new Map(nodes.map((n) => [n.id, n])) - const degreeMap = new Map(nodes.map((n) => [n.id, n.linkCount])) - const maxDegree = Math.max(...nodes.map((n) => n.linkCount), 1) - const STRUCTURAL_IDS = new Set(["index", "log", "overview"]) - const scored: SurprisingConnection[] = [] - - for (const edge of edges) { - const source = nodeMap.get(edge.source) - const target = nodeMap.get(edge.target) - if (!source || !target) continue - if (STRUCTURAL_IDS.has(source.id) || STRUCTURAL_IDS.has(target.id)) continue - - let score = 0 - const reasons: string[] = [] - - if (source.community !== target.community) { - score += 3 - reasons.push("crosses community boundary") - } - if (source.type !== target.type) { - const distantPairs = new Set([ - "source-concept", "concept-source", "source-synthesis", "synthesis-source", - "query-entity", "entity-query", - ]) - if (distantPairs.has(`${source.type}-${target.type}`)) { - score += 2 - reasons.push(`connects ${source.type} to ${target.type}`) - } else { - score += 1 - reasons.push("different types") - } - } - const sourceDeg = degreeMap.get(source.id) ?? 0 - const targetDeg = degreeMap.get(target.id) ?? 0 - const minDeg = Math.min(sourceDeg, targetDeg) - const maxDeg = Math.max(sourceDeg, targetDeg) - if (minDeg <= 2 && maxDeg >= maxDegree * 0.5) { - score += 2 - reasons.push("peripheral node links to hub") - } - if (edge.weight < 2 && edge.weight > 0) { - score += 1 - reasons.push("weak but present connection") - } - if (score >= 3 && reasons.length > 0) { - const key = [source.id, target.id].sort().join(":::") - scored.push({ source, target, score, reasons, key }) - } - } - - scored.sort((a, b) => b.score - a.score) - return scored.slice(0, limit) -} - -export function detectKnowledgeGaps( - nodes: GraphNode[], - edges: GraphEdge[], - communities: CommunityInfo[], - limit: number = 8, -): KnowledgeGap[] { - const gaps: KnowledgeGap[] = [] - const nodeMap = new Map(nodes.map((n) => [n.id, n])) - - // 1. Isolated nodes (degree ≤ 1) - const isolatedNodes = nodes.filter( - (n) => n.linkCount <= 1 && n.type !== "overview" && n.id !== "index" && n.id !== "log", - ) - if (isolatedNodes.length > 0) { - const topIsolated = isolatedNodes.slice(0, 5) - gaps.push({ - type: "isolated-node", - title: `${isolatedNodes.length} isolated page${isolatedNodes.length > 1 ? "s" : ""}`, - description: topIsolated.map((n) => n.label).join(", ") + - (isolatedNodes.length > 5 ? ` and ${isolatedNodes.length - 5} more` : ""), - nodeIds: isolatedNodes.map((n) => n.id), - suggestion: "These pages have few or no connections. Consider adding [[wikilinks]] to related pages.", - }) - } - - // 2. Sparse communities (low cohesion) - for (const comm of communities) { - if (comm.cohesion < 0.15 && comm.nodeCount >= 3) { - gaps.push({ - type: "sparse-community", - title: `Sparse cluster: ${comm.topNodes[0] ?? `Community ${comm.id}`}`, - description: `${comm.nodeCount} pages with cohesion ${comm.cohesion.toFixed(2)} — internal connections are weak.`, - nodeIds: nodes.filter((n) => n.community === comm.id).map((n) => n.id), - suggestion: "This knowledge area lacks internal cross-references. Consider adding links between these pages.", - }) - } - } - - // 3. Bridge nodes (connected to multiple communities) - const communityNeighbors = new Map>() - for (const node of nodes) communityNeighbors.set(node.id, new Set()) - for (const edge of edges) { - const sourceNode = nodeMap.get(edge.source) - const targetNode = nodeMap.get(edge.target) - if (sourceNode && targetNode) { - communityNeighbors.get(edge.source)?.add(targetNode.community) - communityNeighbors.get(edge.target)?.add(sourceNode.community) - } - } - const STRUCTURAL_IDS = new Set(["index", "log", "overview"]) - const bridgeNodes = nodes - .filter((n) => { - if (STRUCTURAL_IDS.has(n.id)) return false - const neighborComms = communityNeighbors.get(n.id) - return neighborComms && neighborComms.size >= 3 - }) - .sort((a, b) => (communityNeighbors.get(b.id)?.size ?? 0) - (communityNeighbors.get(a.id)?.size ?? 0)) - .slice(0, 3) - - for (const bridge of bridgeNodes) { - const commCount = communityNeighbors.get(bridge.id)?.size ?? 0 - gaps.push({ - type: "bridge-node", - title: `Key bridge: ${bridge.label}`, - description: `Connects ${commCount} different knowledge clusters. This is a critical junction in your wiki.`, - nodeIds: [bridge.id], - suggestion: "This page bridges multiple knowledge areas. Ensure it's well-maintained and expanded.", - }) - } - - return gaps.slice(0, limit) -} diff --git a/mcp-server/src/lib/graph-relevance.ts b/mcp-server/src/lib/graph-relevance.ts deleted file mode 100644 index db27e7b1..00000000 --- a/mcp-server/src/lib/graph-relevance.ts +++ /dev/null @@ -1,229 +0,0 @@ -import { readFile, listDirectory } from "../shims/fs-node" -import type { FileNode } from "../types/wiki" -import { normalizePath } from "./path-utils" - -export interface RetrievalNode { - readonly id: string - readonly title: string - readonly type: string - readonly path: string - readonly sources: readonly string[] - readonly outLinks: ReadonlySet - readonly inLinks: ReadonlySet -} - -export interface RetrievalGraph { - readonly nodes: ReadonlyMap - readonly dataVersion: number -} - -const WIKILINK_REGEX = /\[\[([^\]|]+?)(?:\|[^\]]+?)?\]\]/g - -const WEIGHTS = { - directLink: 3.0, - sourceOverlap: 4.0, - commonNeighbor: 1.5, - typeAffinity: 1.0, -} as const - -const TYPE_AFFINITY: Record> = { - entity: { concept: 1.2, entity: 0.8, source: 1.0, synthesis: 1.0, query: 0.8 }, - concept: { entity: 1.2, concept: 0.8, source: 1.0, synthesis: 1.2, query: 1.0 }, - source: { entity: 1.0, concept: 1.0, source: 0.5, query: 0.8, synthesis: 1.0 }, - query: { concept: 1.0, entity: 0.8, synthesis: 1.0, source: 0.8, query: 0.5 }, - synthesis: { concept: 1.2, entity: 1.0, source: 1.0, query: 1.0, synthesis: 0.8 }, -} - -let cachedGraph: RetrievalGraph | null = null - -function flattenMdFiles(nodes: readonly FileNode[]): FileNode[] { - const files: FileNode[] = [] - for (const node of nodes) { - if (node.is_dir && node.children) { - files.push(...flattenMdFiles(node.children)) - } else if (!node.is_dir && node.name.endsWith(".md")) { - files.push(node) - } - } - return files -} - -function fileNameToId(fileName: string): string { - return fileName.replace(/\.md$/, "") -} - -function extractFrontmatter(content: string): { title: string; type: string; sources: string[] } { - const fmMatch = content.match(/^---\n([\s\S]*?)\n---/) - const fm = fmMatch ? fmMatch[1] : "" - const titleMatch = fm.match(/^title:\s*["']?(.+?)["']?\s*$/m) - const typeMatch = fm.match(/^type:\s*["']?(.+?)["']?\s*$/m) - const sources: string[] = [] - const sourcesBlockMatch = fm.match(/^sources:\s*\n((?:\s+-\s+.+\n?)*)/m) - if (sourcesBlockMatch) { - const lines = sourcesBlockMatch[1].split("\n") - for (const line of lines) { - const itemMatch = line.match(/^\s+-\s+["']?(.+?)["']?\s*$/) - if (itemMatch) sources.push(itemMatch[1]) - } - } else { - const inlineMatch = fm.match(/^sources:\s*\[([^\]]*)\]/m) - if (inlineMatch) { - const items = inlineMatch[1].split(",") - for (const item of items) { - const trimmed = item.trim().replace(/^["']|["']$/g, "") - if (trimmed) sources.push(trimmed) - } - } - } - let title = titleMatch ? titleMatch[1].trim() : "" - if (!title) { - const headingMatch = content.match(/^#\s+(.+)$/m) - title = headingMatch ? headingMatch[1].trim() : "" - } - return { title, type: typeMatch ? typeMatch[1].trim().toLowerCase() : "other", sources } -} - -function extractWikilinks(content: string): string[] { - const links: string[] = [] - const regex = new RegExp(WIKILINK_REGEX.source, "g") - let match: RegExpExecArray | null - while ((match = regex.exec(content)) !== null) links.push(match[1].trim()) - return links -} - -function resolveTarget(raw: string, nodeIds: ReadonlySet): string | null { - if (nodeIds.has(raw)) return raw - const normalized = raw.toLowerCase().replace(/\s+/g, "-") - for (const id of nodeIds) { - const idLower = id.toLowerCase() - if (idLower === normalized) return id - if (idLower === raw.toLowerCase()) return id - if (idLower.replace(/\s+/g, "-") === normalized) return id - } - return null -} - -function getNeighbors(node: RetrievalNode): ReadonlySet { - const neighbors = new Set() - for (const id of node.outLinks) neighbors.add(id) - for (const id of node.inLinks) neighbors.add(id) - return neighbors -} - -function getNodeDegree(node: RetrievalNode): number { - return node.outLinks.size + node.inLinks.size -} - -export async function buildRetrievalGraph( - projectPath: string, - dataVersion: number = 0, -): Promise { - if (cachedGraph !== null && cachedGraph.dataVersion === dataVersion) return cachedGraph - - const wikiRoot = `${normalizePath(projectPath)}/wiki` - let tree: FileNode[] - try { - tree = await listDirectory(wikiRoot) - } catch { - const emptyGraph: RetrievalGraph = { nodes: new Map(), dataVersion } - cachedGraph = emptyGraph - return emptyGraph - } - - const mdFiles = flattenMdFiles(tree) - const rawNodes: Array<{ - id: string; title: string; type: string; path: string - sources: string[]; rawLinks: string[]; fileName: string - }> = [] - - for (const file of mdFiles) { - const id = fileNameToId(file.name) - let content = "" - try { content = await readFile(file.path) } catch { continue } - const fm = extractFrontmatter(content) - rawNodes.push({ - id, title: fm.title || file.name.replace(/\.md$/, "").replace(/-/g, " "), - type: fm.type, path: file.path, sources: fm.sources, - rawLinks: extractWikilinks(content), fileName: file.name, - }) - } - - const nodeIds = new Set(rawNodes.map((n) => n.id)) - const outLinksMap = new Map>() - const inLinksMap = new Map>() - for (const id of nodeIds) { - outLinksMap.set(id, new Set()) - inLinksMap.set(id, new Set()) - } - - for (const raw of rawNodes) { - for (const linkTarget of raw.rawLinks) { - const resolvedId = resolveTarget(linkTarget, nodeIds) - if (resolvedId === null || resolvedId === raw.id) continue - outLinksMap.get(raw.id)!.add(resolvedId) - inLinksMap.get(resolvedId)!.add(raw.id) - } - } - - const nodes = new Map() - for (const raw of rawNodes) { - nodes.set(raw.id, { - id: raw.id, title: raw.title, type: raw.type, path: raw.path, - sources: Object.freeze([...raw.sources]), - outLinks: Object.freeze(outLinksMap.get(raw.id) ?? new Set()), - inLinks: Object.freeze(inLinksMap.get(raw.id) ?? new Set()), - }) - } - - const graph: RetrievalGraph = { nodes, dataVersion } - cachedGraph = graph - return graph -} - -export function calculateRelevance( - nodeA: RetrievalNode, nodeB: RetrievalNode, graph: RetrievalGraph, -): number { - if (nodeA.id === nodeB.id) return 0 - const forwardLinks = nodeA.outLinks.has(nodeB.id) ? 1 : 0 - const backwardLinks = nodeB.outLinks.has(nodeA.id) ? 1 : 0 - const directLinkScore = (forwardLinks + backwardLinks) * WEIGHTS.directLink - const sourcesA = new Set(nodeA.sources) - let sharedSourceCount = 0 - for (const src of nodeB.sources) { if (sourcesA.has(src)) sharedSourceCount += 1 } - const sourceOverlapScore = sharedSourceCount * WEIGHTS.sourceOverlap - const neighborsA = getNeighbors(nodeA) - const neighborsB = getNeighbors(nodeB) - let adamicAdar = 0 - for (const neighborId of neighborsA) { - if (neighborsB.has(neighborId)) { - const neighbor = graph.nodes.get(neighborId) - if (neighbor) { - const degree = getNodeDegree(neighbor) - adamicAdar += 1 / Math.log(Math.max(degree, 2)) - } - } - } - const commonNeighborScore = adamicAdar * WEIGHTS.commonNeighbor - const affinityMap = TYPE_AFFINITY[nodeA.type] - const typeAffinityScore = (affinityMap?.[nodeB.type] ?? 0.5) * WEIGHTS.typeAffinity - return directLinkScore + sourceOverlapScore + commonNeighborScore + typeAffinityScore -} - -export function getRelatedNodes( - nodeId: string, graph: RetrievalGraph, limit: number = 5, -): ReadonlyArray<{ node: RetrievalNode; relevance: number }> { - const sourceNode = graph.nodes.get(nodeId) - if (!sourceNode) return [] - const scored: Array<{ node: RetrievalNode; relevance: number }> = [] - for (const [id, node] of graph.nodes) { - if (id === nodeId) continue - const relevance = calculateRelevance(sourceNode, node, graph) - if (relevance > 0) scored.push({ node, relevance }) - } - scored.sort((a, b) => b.relevance - a.relevance) - return scored.slice(0, limit) -} - -export function clearGraphCache(): void { - cachedGraph = null -} diff --git a/mcp-server/src/lib/path-utils.ts b/mcp-server/src/lib/path-utils.ts deleted file mode 100644 index 3296d0a9..00000000 --- a/mcp-server/src/lib/path-utils.ts +++ /dev/null @@ -1,38 +0,0 @@ -export function normalizePath(p: string): string { - return p.replace(/\\/g, "/") -} - -export function joinPath(...segments: string[]): string { - return segments - .map((s) => s.replace(/\\/g, "/")) - .join("/") - .replace(/\/+/g, "/") -} - -export function getFileName(p: string): string { - const normalized = p.replace(/\\/g, "/") - return normalized.split("/").pop() ?? p -} - -export function getFileStem(p: string): string { - const name = getFileName(p) - const lastDot = name.lastIndexOf(".") - return lastDot > 0 ? name.slice(0, lastDot) : name -} - -export function getRelativePath(fullPath: string, basePath: string): string { - const normalFull = normalizePath(fullPath) - const normalBase = normalizePath(basePath).replace(/\/$/, "") - if (normalFull.startsWith(normalBase + "/")) { - return normalFull.slice(normalBase.length + 1) - } - return normalFull -} - -export function isAbsolutePath(p: string): boolean { - if (!p) return false - if (p.startsWith("/")) return true - if (/^[A-Za-z]:[\\/]/.test(p)) return true - if (p.startsWith("\\\\") || p.startsWith("//")) return true - return false -} diff --git a/mcp-server/src/lib/search.ts b/mcp-server/src/lib/search.ts deleted file mode 100644 index d67e4e51..00000000 --- a/mcp-server/src/lib/search.ts +++ /dev/null @@ -1,228 +0,0 @@ -import { readFile, listDirectory } from "../shims/fs-node" -import type { FileNode } from "../types/wiki" -import { normalizePath, getFileStem } from "./path-utils" - -export interface ImageRef { - url: string - alt: string -} - -export interface SearchResult { - path: string - title: string - snippet: string - titleMatch: boolean - score: number - images: ImageRef[] -} - -const MAX_RESULTS = 20 -const SNIPPET_CONTEXT = 80 -const RRF_K = 60 -const FILENAME_EXACT_BONUS = 200 -const PHRASE_IN_TITLE_BONUS = 50 -const PHRASE_IN_CONTENT_PER_OCC = 20 -const MAX_PHRASE_OCC_COUNTED = 10 -const TITLE_TOKEN_WEIGHT = 5 -const CONTENT_TOKEN_WEIGHT = 1 - -const STOP_WORDS = new Set([ - "的", "是", "了", "什么", "在", "有", "和", "与", "对", "从", - "the", "is", "a", "an", "what", "how", "are", "was", "were", - "do", "does", "did", "be", "been", "being", "have", "has", "had", - "it", "its", "in", "on", "at", "to", "for", "of", "with", "by", - "this", "that", "these", "those", -]) - -export function tokenizeQuery(query: string): string[] { - const rawTokens = query - .toLowerCase() - .split(/[\s,，。！？、；：""''（）()\-_/\\·~～…]+/) - .filter((t) => t.length > 1) - .filter((t) => !STOP_WORDS.has(t)) - - const tokens: string[] = [] - for (const token of rawTokens) { - const hasCJK = /[\u4e00-\u9fff\u3400-\u4dbf]/.test(token) - if (hasCJK && token.length > 2) { - const chars = [...token] - for (let i = 0; i < chars.length - 1; i++) tokens.push(chars[i] + chars[i + 1]) - for (const ch of chars) { if (!STOP_WORDS.has(ch)) tokens.push(ch) } - tokens.push(token) - } else { - tokens.push(token) - } - } - return [...new Set(tokens)] -} - -function tokenMatchScore(text: string, tokens: readonly string[]): number { - const lower = text.toLowerCase() - let score = 0 - for (const token of tokens) { if (lower.includes(token)) score += 1 } - return score -} - -function countOccurrences(haystackLower: string, needleLower: string): number { - if (!needleLower) return 0 - let count = 0; let pos = 0 - while (true) { - const idx = haystackLower.indexOf(needleLower, pos) - if (idx === -1) break - count++; pos = idx + needleLower.length - } - return count -} - -function flattenMdFiles(nodes: FileNode[]): FileNode[] { - const files: FileNode[] = [] - for (const node of nodes) { - if (node.is_dir && node.children) files.push(...flattenMdFiles(node.children)) - else if (!node.is_dir && node.name.endsWith(".md")) files.push(node) - } - return files -} - -function extractTitle(content: string, fileName: string): string { - const fm = content.match(/^---\n[\s\S]*?^title:\s*["']?(.+?)["']?\s*$/m) - if (fm) return fm[1].trim() - const h = content.match(/^#\s+(.+)$/m) - if (h) return h[1].trim() - return fileName.replace(/\.md$/, "").replace(/-/g, " ") -} - -const IMAGE_REF_RE = /!\[([^\]]*)\]\(([^)\s]+)\)/g - -function extractImageRefs(content: string): ImageRef[] { - const seen = new Set(); const out: ImageRef[] = [] - for (const m of content.matchAll(IMAGE_REF_RE)) { - const url = m[2] - if (seen.has(url)) continue - seen.add(url); out.push({ url, alt: m[1] }) - } - return out -} - -function buildSnippet(content: string, query: string): string { - const lower = content.toLowerCase(); const lowerQuery = query.toLowerCase() - const idx = lower.indexOf(lowerQuery) - if (idx === -1) return content.slice(0, SNIPPET_CONTEXT * 2).replace(/\n/g, " ") - const start = Math.max(0, idx - SNIPPET_CONTEXT) - const end = Math.min(content.length, idx + query.length + SNIPPET_CONTEXT) - let snippet = content.slice(start, end).replace(/\n/g, " ") - if (start > 0) snippet = "..." + snippet - if (end < content.length) snippet = snippet + "..." - return snippet -} - -const TRIM_PUNCT_RE = /^[\s,，。！？、；：""''（）()\-_/\\·~～…]+|[\s,，。！？、；：""''（）()\-_/\\·~～…]+$/g -const SEARCH_READ_CONCURRENCY = 16 - -function scoreFile( - file: FileNode, content: string, tokens: readonly string[], queryPhrase: string, query: string, -): SearchResult | null { - const title = extractTitle(content, file.name) - const titleText = `${title} ${file.name}` - const titleLower = titleText.toLowerCase() - const contentLower = content.toLowerCase() - const fileStem = file.name.replace(/\.md$/, "").toLowerCase() - - const filenameExact = fileStem === queryPhrase - const titleHasPhrase = queryPhrase.length > 0 && titleLower.includes(queryPhrase) - const contentPhraseOcc = Math.min(countOccurrences(contentLower, queryPhrase), MAX_PHRASE_OCC_COUNTED) - const titleTokenScore = tokenMatchScore(titleText, tokens) - const contentTokenScore = tokenMatchScore(content, tokens) - - if (!filenameExact && !titleHasPhrase && contentPhraseOcc === 0 && titleTokenScore === 0 && contentTokenScore === 0) return null - - const score = - (filenameExact ? FILENAME_EXACT_BONUS : 0) + - (titleHasPhrase ? PHRASE_IN_TITLE_BONUS : 0) + - contentPhraseOcc * PHRASE_IN_CONTENT_PER_OCC + - titleTokenScore * TITLE_TOKEN_WEIGHT + - contentTokenScore * CONTENT_TOKEN_WEIGHT - - const snippetAnchor = contentPhraseOcc > 0 ? queryPhrase : (tokens.find((t) => contentLower.includes(t)) ?? query) - return { - path: file.path, title, snippet: buildSnippet(content, snippetAnchor), - titleMatch: titleTokenScore > 0 || titleHasPhrase, score, images: extractImageRefs(content), - } -} - -async function searchFiles( - files: FileNode[], tokens: readonly string[], query: string, results: SearchResult[], -): Promise { - const queryPhrase = query.trim().toLowerCase().replace(TRIM_PUNCT_RE, "") - for (let i = 0; i < files.length; i += SEARCH_READ_CONCURRENCY) { - const batch = files.slice(i, i + SEARCH_READ_CONCURRENCY) - const batchResults = await Promise.all( - batch.map(async (file) => { - let content: string - try { content = await readFile(file.path) } catch { return null } - return scoreFile(file, content, tokens, queryPhrase, query) - }), - ) - for (const r of batchResults) { if (r) results.push(r) } - } -} - -export async function searchWiki(projectPath: string, query: string): Promise { - if (!query.trim()) return [] - const pp = normalizePath(projectPath) - const tokens = tokenizeQuery(query) - const effectiveTokens = tokens.length > 0 ? tokens : [query.trim().toLowerCase()] - const results: SearchResult[] = [] - - try { - const wikiTree = await listDirectory(`${pp}/wiki`) - const wikiFiles = flattenMdFiles(wikiTree) - await searchFiles(wikiFiles, effectiveTokens, query, results) - } catch { /* no wiki directory */ } - - const tokenSorted = [...results].sort((a, b) => b.score - a.score) - const tokenRank = new Map() - tokenSorted.forEach((r, i) => tokenRank.set(normalizePath(r.path), i + 1)) - - // Vector search (optional — gracefully degrades when embedding not configured) - let vectorRank = new Map() - let vectorCount = 0 - try { - const { useWikiStore } = await import("../shims/stores-node") - const embCfg = useWikiStore.getState().embeddingConfig - if (embCfg.enabled && embCfg.model) { - const { searchByEmbedding } = await import("../shims/embedding-stub") - const vectorResults = await searchByEmbedding(pp, query, embCfg, 10) - vectorCount = vectorResults.length - vectorResults.forEach((vr, i) => vectorRank.set(vr.id, i + 1)) - - const knownIds = new Set(results.map((r) => getFileStem(r.path))) - for (const vr of vectorResults) { - if (knownIds.has(vr.id)) continue - const dirs = ["entities", "concepts", "sources", "synthesis", "comparison", "queries"] - for (const dir of dirs) { - const tryPath = `${pp}/wiki/${dir}/${vr.id}.md` - try { - const content = await readFile(tryPath) - const title = extractTitle(content, `${vr.id}.md`) - results.push({ path: tryPath, title, snippet: buildSnippet(content, query), titleMatch: false, score: 0, images: extractImageRefs(content) }) - knownIds.add(vr.id); break - } catch { /* not in this dir */ } - } - } - } - } catch { /* vector search not available */ } - - // RRF fusion - for (const r of results) { - const tRank = tokenRank.get(normalizePath(r.path)) - const vRank = vectorRank.get(getFileStem(r.path)) - let rrf = 0 - if (tRank !== undefined) rrf += 1 / (RRF_K + tRank) - if (vRank !== undefined) rrf += 1 / (RRF_K + vRank) - r.score = rrf - } - - results.sort((a, b) => b.score !== a.score ? b.score - a.score : a.path.localeCompare(b.path)) - console.error(`[search] "${query}" | token:${tokenRank.size} vector:${vectorCount} → ${results.length} results`) - return results.slice(0, MAX_RESULTS) -} diff --git a/mcp-server/src/lib/wiki-graph.ts b/mcp-server/src/lib/wiki-graph.ts deleted file mode 100644 index 5160563a..00000000 --- a/mcp-server/src/lib/wiki-graph.ts +++ /dev/null @@ -1,211 +0,0 @@ -import { readFile, listDirectory } from "../shims/fs-node" -import type { FileNode } from "../types/wiki" -import { buildRetrievalGraph, calculateRelevance } from "./graph-relevance" -import { normalizePath } from "./path-utils" -import Graph from "graphology" -import louvain from "graphology-communities-louvain" - -export interface GraphNode { - id: string - label: string - type: string - path: string - linkCount: number - community: number -} - -export interface GraphEdge { - source: string - target: string - weight: number -} - -export interface CommunityInfo { - id: number - nodeCount: number - cohesion: number - topNodes: string[] -} - -function detectCommunities( - nodes: { id: string; label: string; linkCount: number }[], - edges: GraphEdge[], -): { assignments: Map; communities: CommunityInfo[] } { - if (nodes.length === 0) return { assignments: new Map(), communities: [] } - - const g = new Graph({ type: "undirected" }) - for (const node of nodes) g.addNode(node.id) - for (const edge of edges) { - if (g.hasNode(edge.source) && g.hasNode(edge.target)) { - const key = `${edge.source}->${edge.target}` - if (!g.hasEdge(key) && !g.hasEdge(`${edge.target}->${edge.source}`)) { - g.addEdgeWithKey(key, edge.source, edge.target, { weight: edge.weight }) - } - } - } - - const communityMap: Record = louvain(g, { resolution: 1 }) - const assignments = new Map(Object.entries(communityMap).map(([k, v]) => [k, v as number])) - - const groups = new Map() - for (const [nodeId, commId] of assignments) { - const list = groups.get(commId) ?? [] - list.push(nodeId) - groups.set(commId, list) - } - - const edgeSet = new Set() - for (const edge of edges) { - edgeSet.add(`${edge.source}:::${edge.target}`) - edgeSet.add(`${edge.target}:::${edge.source}`) - } - - const nodeInfo = new Map(nodes.map((n) => [n.id, { label: n.label, linkCount: n.linkCount }])) - const communities: CommunityInfo[] = [] - - for (const [commId, memberIds] of groups) { - const n = memberIds.length - let intraEdges = 0 - for (let i = 0; i < memberIds.length; i++) { - for (let j = i + 1; j < memberIds.length; j++) { - if (edgeSet.has(`${memberIds[i]}:::${memberIds[j]}`)) intraEdges++ - } - } - const possibleEdges = n > 1 ? (n * (n - 1)) / 2 : 1 - const cohesion = intraEdges / possibleEdges - const sorted = [...memberIds].sort( - (a, b) => (nodeInfo.get(b)?.linkCount ?? 0) - (nodeInfo.get(a)?.linkCount ?? 0) - ) - communities.push({ id: commId, nodeCount: n, cohesion, topNodes: sorted.slice(0, 5).map((id) => nodeInfo.get(id)?.label ?? id) }) - } - - communities.sort((a, b) => b.nodeCount - a.nodeCount) - const idRemap = new Map() - communities.forEach((c, idx) => { idRemap.set(c.id, idx); c.id = idx }) - for (const [nodeId, oldId] of assignments) assignments.set(nodeId, idRemap.get(oldId) ?? 0) - - return { assignments, communities } -} - -const WIKILINK_REGEX = /\[\[([^\]|]+?)(?:\|[^\]]+?)?\]\]/g - -function flattenMdFiles(nodes: FileNode[]): FileNode[] { - const files: FileNode[] = [] - for (const node of nodes) { - if (node.is_dir && node.children) files.push(...flattenMdFiles(node.children)) - else if (!node.is_dir && node.name.endsWith(".md")) files.push(node) - } - return files -} - -function extractTitle(content: string, fileName: string): string { - const fm = content.match(/^---\n[\s\S]*?^title:\s*["']?(.+?)["']?\s*$/m) - if (fm) return fm[1].trim() - const h = content.match(/^#\s+(.+)$/m) - if (h) return h[1].trim() - return fileName.replace(/\.md$/, "").replace(/-/g, " ") -} - -function extractType(content: string): string { - const m = content.match(/^---\n[\s\S]*?^type:\s*["']?(.+?)["']?\s*$/m) - return m ? m[1].trim().toLowerCase() : "other" -} - -function extractWikilinks(content: string): string[] { - const links: string[] = [] - const regex = new RegExp(WIKILINK_REGEX.source, "g") - let match: RegExpExecArray | null - while ((match = regex.exec(content)) !== null) links.push(match[1].trim()) - return links -} - -function resolveTarget(raw: string, nodeMap: Map): string | null { - if (nodeMap.has(raw)) return raw - const normalized = raw.toLowerCase().replace(/\s+/g, "-") - for (const id of nodeMap.keys()) { - if (id.toLowerCase() === normalized) return id - if (id.toLowerCase() === raw.toLowerCase()) return id - if (id.toLowerCase().replace(/\s+/g, "-") === normalized) return id - } - return null -} - -export async function buildWikiGraph( - projectPath: string, -): Promise<{ nodes: GraphNode[]; edges: GraphEdge[]; communities: CommunityInfo[] }> { - const wikiRoot = `${normalizePath(projectPath)}/wiki` - let tree: FileNode[] - try { tree = await listDirectory(wikiRoot) } catch { - return { nodes: [], edges: [], communities: [] } - } - - const mdFiles = flattenMdFiles(tree) - if (mdFiles.length === 0) return { nodes: [], edges: [], communities: [] } - - const nodeMap = new Map() - for (const file of mdFiles) { - const id = file.name.replace(/\.md$/, "") - let content = "" - try { content = await readFile(file.path) } catch { continue } - nodeMap.set(id, { id, label: extractTitle(content, file.name), type: extractType(content), path: file.path, links: extractWikilinks(content) }) - } - - const HIDDEN_TYPES = new Set(["query"]) - for (const [id, node] of nodeMap) { - if (HIDDEN_TYPES.has(node.type)) nodeMap.delete(id) - } - - const linkCounts = new Map() - for (const [id] of nodeMap) linkCounts.set(id, 0) - - const rawEdges: GraphEdge[] = [] - for (const [sourceId, nodeData] of nodeMap) { - for (const targetRaw of nodeData.links) { - const targetId = resolveTarget(targetRaw, nodeMap) - if (targetId === null || targetId === sourceId) continue - rawEdges.push({ source: sourceId, target: targetId, weight: 1 }) - linkCounts.set(sourceId, (linkCounts.get(sourceId) ?? 0) + 1) - linkCounts.set(targetId, (linkCounts.get(targetId) ?? 0) + 1) - } - } - - const seenEdges = new Set() - const dedupedEdges: { source: string; target: string }[] = [] - for (const edge of rawEdges) { - const key = `${edge.source}:::${edge.target}` - const reverseKey = `${edge.target}:::${edge.source}` - if (!seenEdges.has(key) && !seenEdges.has(reverseKey)) { - seenEdges.add(key) - dedupedEdges.push(edge) - } - } - - // Try to get retrieval graph for weighted edges (gracefully degrades) - let retrievalGraph: Awaited> | null = null - try { - const { useWikiStore } = await import("../shims/stores-node") - const dv = useWikiStore.getState().dataVersion - retrievalGraph = await buildRetrievalGraph(normalizePath(projectPath), dv) - } catch { /* ignore — weights default to 1 */ } - - const edges: GraphEdge[] = dedupedEdges.map((e) => { - let weight = 1 - if (retrievalGraph) { - const nodeA = retrievalGraph.nodes.get(e.source) - const nodeB = retrievalGraph.nodes.get(e.target) - if (nodeA && nodeB) weight = calculateRelevance(nodeA, nodeB, retrievalGraph) - } - return { source: e.source, target: e.target, weight } - }) - - const prelimNodes = Array.from(nodeMap.values()).map((n) => ({ id: n.id, label: n.label, linkCount: linkCounts.get(n.id) ?? 0 })) - const { assignments, communities } = detectCommunities(prelimNodes, edges) - - const nodes: GraphNode[] = Array.from(nodeMap.values()).map((n) => ({ - id: n.id, label: n.label, type: n.type, path: n.path, - linkCount: linkCounts.get(n.id) ?? 0, - community: assignments.get(n.id) ?? 0, - })) - - return { nodes, edges, communities } -} diff --git a/mcp-server/src/shims/embedding-stub.ts b/mcp-server/src/shims/embedding-stub.ts deleted file mode 100644 index 8a2117aa..00000000 --- a/mcp-server/src/shims/embedding-stub.ts +++ /dev/null @@ -1,32 +0,0 @@ -/** - * Stub for @/lib/embedding — vector search is not available in the skill/MCP layer - * without a running LanceDB instance. Returns empty results so BM25-only search works. - */ -export interface EmbeddingConfig { - enabled: boolean - model: string - apiBase?: string - apiKey?: string -} - -export interface VectorSearchResult { - id: string - score: number - path?: string -} - -export async function searchByEmbedding( - _projectPath: string, - _query: string, - _config: EmbeddingConfig, - _limit: number = 10, -): Promise { - return [] -} - -export async function createEmbedding( - _text: string, - _config: EmbeddingConfig, -): Promise { - return [] -} diff --git a/mcp-server/src/shims/fs-node.ts b/mcp-server/src/shims/fs-node.ts deleted file mode 100644 index de343bda..00000000 --- a/mcp-server/src/shims/fs-node.ts +++ /dev/null @@ -1,101 +0,0 @@ -/** - * Node.js drop-in replacement for @/commands/fs (nashsu Tauri IPC layer). - * Replaces all invoke("...") calls with standard Node.js fs operations. - */ -import * as fs from "fs" -import * as path from "path" -import type { FileNode } from "../types/wiki" - -export async function readFile(filePath: string): Promise { - return fs.readFileSync(filePath, "utf-8") -} - -export async function writeFile(filePath: string, content: string): Promise { - fs.mkdirSync(path.dirname(filePath), { recursive: true }) - fs.writeFileSync(filePath, content, "utf-8") -} - -export async function listDirectory(dirPath: string): Promise { - function walk(dir: string): FileNode[] { - const entries = fs.readdirSync(dir, { withFileTypes: true }) - return entries.map((entry) => { - const entryPath = path.join(dir, entry.name).replace(/\\/g, "/") - if (entry.isDirectory()) { - return { - name: entry.name, - path: entryPath, - is_dir: true, - children: walk(entryPath), - } - } - return { name: entry.name, path: entryPath, is_dir: false } - }) - } - return walk(dirPath) -} - -export async function copyFile(from: string, to: string): Promise { - fs.mkdirSync(path.dirname(to), { recursive: true }) - fs.copyFileSync(from, to) -} - -export async function deleteFile(filePath: string): Promise { - if (fs.existsSync(filePath)) fs.unlinkSync(filePath) -} - -export async function createDirectory(dirPath: string): Promise { - fs.mkdirSync(dirPath, { recursive: true }) -} - -export async function fileExists(filePath: string): Promise { - return fs.existsSync(filePath) -} - -export async function readFileAsBase64(filePath: string): Promise { - return fs.readFileSync(filePath).toString("base64") -} - -/** - * Text extraction for PDFs/DOCX/etc. - * In Node mode: returns raw file content if text-based, otherwise empty string. - * For real PDF extraction, users should pre-extract with markitdown or pdftotext. - */ -export async function preprocessFile(filePath: string): Promise { - const ext = path.extname(filePath).toLowerCase() - const textExts = [".md", ".txt", ".json", ".yaml", ".yml", ".csv", ".html", ".htm"] - if (textExts.includes(ext)) { - try { - return fs.readFileSync(filePath, "utf-8") - } catch { - return "" - } - } - // For binary files (PDF, DOCX, etc.) return empty — use pre-extracted markdown - console.warn(`[fs-node] preprocessFile: binary format not supported in Node mode: ${filePath}`) - return "" -} - -export async function findRelatedWikiPages( - sourceFile: string, - wikiRoot: string, -): Promise { - const stem = path.basename(sourceFile, path.extname(sourceFile)).toLowerCase() - const results: string[] = [] - function walk(dir: string): void { - if (!fs.existsSync(dir)) return - for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { - const full = path.join(dir, entry.name) - if (entry.isDirectory()) walk(full) - else if (entry.name.endsWith(".md")) { - try { - const content = fs.readFileSync(full, "utf-8") - if (content.toLowerCase().includes(stem)) { - results.push(full.replace(/\\/g, "/")) - } - } catch { /* skip */ } - } - } - } - walk(wikiRoot) - return results -} diff --git a/mcp-server/src/shims/stores-node.ts b/mcp-server/src/shims/stores-node.ts deleted file mode 100644 index 66fa3773..00000000 --- a/mcp-server/src/shims/stores-node.ts +++ /dev/null @@ -1,135 +0,0 @@ -/** - * Node.js drop-in for React zustand stores used by nashsu/llm_wiki lib files. - * Replaces all useXxxStore.getState() calls with module-level state. - */ - -export interface LlmConfig { - provider: string - apiKey: string - model: string - baseUrl?: string - temperature?: number - maxTokens?: number -} - -export interface EmbeddingConfig { - enabled: boolean - model: string - apiBase?: string - apiKey?: string -} - -interface WikiState { - projectPath: string - dataVersion: number - llmConfig: LlmConfig - embeddingConfig: EmbeddingConfig -} - -let wikiState: WikiState = { - projectPath: "", - dataVersion: 0, - llmConfig: { - provider: process.env.LLM_PROVIDER ?? "openai", - apiKey: process.env.OPENAI_API_KEY ?? process.env.LLM_API_KEY ?? "", - model: process.env.LLM_MODEL ?? "gpt-4o-mini", - baseUrl: process.env.LLM_BASE_URL, - }, - embeddingConfig: { - enabled: (process.env.EMBEDDING_ENABLED ?? "false") === "true", - model: process.env.EMBEDDING_MODEL ?? "", - apiBase: process.env.EMBEDDING_BASE_URL, - apiKey: process.env.EMBEDDING_API_KEY ?? process.env.OPENAI_API_KEY, - }, -} - -export const useWikiStore = { - getState: () => ({ ...wikiState }), - setState: (updater: Partial | ((s: WikiState) => Partial)) => { - if (typeof updater === "function") { - wikiState = { ...wikiState, ...updater(wikiState) } - } else { - wikiState = { ...wikiState, ...updater } - } - }, -} - -/** Configure the wiki store from environment variables or explicit config */ -export function configureWikiStore(config: Partial) { - wikiState = { ...wikiState, ...config } -} - -// ── Research store ─────────────────────────────────────────────────────────── -interface ResearchState { - activeProjectPath: string - isResearching: boolean -} - -let researchState: ResearchState = { - activeProjectPath: "", - isResearching: false, -} - -export const useResearchStore = { - getState: () => ({ ...researchState }), - setState: (updater: Partial) => { - researchState = { ...researchState, ...updater } - }, -} - -// ── Activity store (replaces Tauri event system) ───────────────────────────── -export interface ActivityItem { - id: string - type: string - title: string - status: "pending" | "running" | "done" | "error" - detail?: string - filesWritten?: string[] -} - -let activityItems: ActivityItem[] = [] -let activityIdCounter = 0 - -export const useActivityStore = { - getState: () => ({ - items: [...activityItems], - addItem: (item: Omit): string => { - const id = `activity-${++activityIdCounter}` - const newItem: ActivityItem = { id, ...item } - activityItems.push(newItem) - if (process.env.SKILL_VERBOSE === "1") { - console.error(`[activity:${item.type}] ${item.title} — ${item.status}`) - } - return id - }, - updateItem: (id: string, updates: Partial): void => { - const idx = activityItems.findIndex((i) => i.id === id) - if (idx >= 0) { - activityItems[idx] = { ...activityItems[idx], ...updates } - if (process.env.SKILL_VERBOSE === "1") { - const item = activityItems[idx] - console.error(`[activity:update] ${item.title} — ${item.status}: ${item.detail ?? ""}`) - } - } - }, - clearItems: () => { activityItems = [] }, - }), - addItem: (item: Omit): string => { - return useActivityStore.getState().addItem(item) - }, - updateItem: (id: string, updates: Partial): void => { - useActivityStore.getState().updateItem(id, updates) - }, -} - -// ── Chat store ─────────────────────────────────────────────────────────────── -export const useChatStore = { - getState: () => ({ messages: [] as unknown[] }), - setState: (_updater: unknown) => {}, -} - -// ── Review store ───────────────────────────────────────────────────────────── -export const useReviewStore = { - getState: () => ({ queue: [] as unknown[], isProcessing: false }), - setState: (_updater: unknown) => {}, -} diff --git a/mcp-server/src/types/wiki.ts b/mcp-server/src/types/wiki.ts deleted file mode 100644 index a51ace03..00000000 --- a/mcp-server/src/types/wiki.ts +++ /dev/null @@ -1,18 +0,0 @@ -export interface WikiProject { - id: string - name: string - path: string -} - -export interface FileNode { - name: string - path: string - is_dir: boolean - children?: FileNode[] -} - -export interface WikiPage { - path: string - content: string - frontmatter: Record -} diff --git a/mcp-server/tsconfig.json b/mcp-server/tsconfig.json deleted file mode 100644 index e37a4324..00000000 --- a/mcp-server/tsconfig.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "compilerOptions": { - "target": "ES2022", - "lib": ["ES2022"], - "module": "CommonJS", - "moduleResolution": "node", - "strict": true, - "esModuleInterop": true, - "allowSyntheticDefaultImports": true, - "skipLibCheck": true, - "outDir": "dist", - "rootDir": "src", - "declaration": true, - "sourceMap": true - }, - "include": ["src/**/*"], - "exclude": ["node_modules", "dist"] -} diff --git a/skill/package-lock.json b/skill/package-lock.json index ff78e035..12137924 100644 --- a/skill/package-lock.json +++ b/skill/package-lock.json @@ -9,8 +9,10 @@ "version": "0.4.6-skill.1", "dependencies": { "@modelcontextprotocol/sdk": "^1.0.0", + "@types/js-yaml": "^4.0.9", "graphology": "^0.25.4", - "graphology-communities-louvain": "^2.0.0" + "graphology-communities-louvain": "^2.0.0", + "js-yaml": "^4.1.1" }, "bin": { "llm-wiki": "dist/cli.js", @@ -73,6 +75,12 @@ } } }, + "node_modules/@types/js-yaml": { + "version": "4.0.9", + "resolved": "https://registry.npmjs.org/@types/js-yaml/-/js-yaml-4.0.9.tgz", + "integrity": "sha512-k4MGaQl5TGo/iipqb2UDG2UwjXziSWkh0uysQelTlJpX1qGlpUZYm8PnO4DxG1qBomtJUdYJ6qR6xdIah10JLg==", + "license": "MIT" + }, "node_modules/@types/node": { "version": "20.19.39", "resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.39.tgz", @@ -129,6 +137,12 @@ } } }, + "node_modules/argparse": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz", + "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==", + "license": "Python-2.0" + }, "node_modules/body-parser": { "version": "2.2.2", "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.2.tgz", @@ -743,6 +757,18 @@ "url": "https://github.com/sponsors/panva" } }, + "node_modules/js-yaml": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.1.tgz", + "integrity": "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==", + "license": "MIT", + "dependencies": { + "argparse": "^2.0.1" + }, + "bin": { + "js-yaml": "bin/js-yaml.js" + } + }, "node_modules/json-schema-traverse": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", diff --git a/skill/package.json b/skill/package.json index 8b8f8da7..dd07dde1 100644 --- a/skill/package.json +++ b/skill/package.json @@ -14,9 +14,11 @@ "mcp": "node dist/mcp-server.js" }, "dependencies": { + "@modelcontextprotocol/sdk": "^1.0.0", + "@types/js-yaml": "^4.0.9", "graphology": "^0.25.4", "graphology-communities-louvain": "^2.0.0", - "@modelcontextprotocol/sdk": "^1.0.0" + "js-yaml": "^4.1.1" }, "devDependencies": { "@types/node": "^20.0.0", diff --git a/skill/src/cli.ts b/skill/src/cli.ts index 991097ab..590e31c5 100644 --- a/skill/src/cli.ts +++ b/skill/src/cli.ts @@ -15,17 +15,21 @@ import * as fs from "fs" import { buildWikiGraph } from "./lib/wiki-graph" import { findSurprisingConnections, detectKnowledgeGaps } from "./lib/graph-insights" import { searchWiki } from "./lib/search" +import { autoIngest } from "./lib/ingest" +import { deepResearch } from "./lib/deep-research" async function main() { const [, , command, ...args] = process.argv if (!command || command === "help" || command === "--help") { usage(); return } switch (command) { - case "graph": return cmdGraph(args) - case "insights": return cmdInsights(args) - case "search": return cmdSearch(args) - case "status": return cmdStatus(args) - case "init": return cmdInit(args) - case "lint": return cmdLint(args) + case "graph": return cmdGraph(args) + case "insights": return cmdInsights(args) + case "search": return cmdSearch(args) + case "status": return cmdStatus(args) + case "init": return cmdInit(args) + case "lint": return cmdLint(args) + case "ingest": return cmdIngest(args) + case "deep-research": return cmdDeepResearch(args) default: console.error(`Unknown command: ${command}`) usage() @@ -41,21 +45,30 @@ USAGE: llm-wiki [options] COMMANDS: - graph Build knowledge graph (outputs JSON) - insights Show surprising connections + knowledge gaps - search Keyword search (BM25+RRF) - status Page count and type breakdown - init Initialize wiki directory structure - lint Check for broken links and orphan pages + graph Build knowledge graph (outputs JSON) + insights Show surprising connections + knowledge gaps + search Keyword search (BM25+RRF) + status Page count and type breakdown + init Initialize wiki directory structure + lint Check for broken links and orphan pages + ingest Two-stage LLM ingest of a markdown/text source + deep-research Web search → LLM synthesis → auto-ingest ENV VARS: SKILL_VERBOSE=1 Enable verbose activity logging WIKI_PATH Default project path for MCP server + OPENAI_API_KEY / LLM_API_KEY LLM credentials (required for ingest / deep-research) + LLM_BASE_URL Custom OpenAI-compatible endpoint + LLM_MODEL Model name (default: gpt-4o-mini) + TAVILY_API_KEY Tavily search API (required for deep-research) + WIKI_OUTPUT_LANGUAGE auto | English | Chinese | Japanese | ... EXAMPLES: llm-wiki graph ./my-project llm-wiki search ./my-project "attention mechanism" llm-wiki insights ./my-project + llm-wiki ingest ./my-project ./raw/paper.md + llm-wiki deep-research ./my-project "transformer architecture" `.trim()) } @@ -153,6 +166,70 @@ async function cmdLint(args: string[]) { console.log(`\n✓ ${nodes.length} pages checked — ${issues} issue(s)`) } +async function cmdIngest(args: string[]) { + const [wikiRoot, sourceFile, ...rest] = args + if (!wikiRoot || !sourceFile) { + console.error("Usage: ingest [--folder=context]") + process.exit(1) + } + const projectPath = path.resolve(wikiRoot) + const sourcePath = path.resolve(sourceFile) + if (!fs.existsSync(sourcePath)) { + console.error(`Source file not found: ${sourcePath}`) + process.exit(1) + } + const folderArg = rest.find((a) => a.startsWith("--folder=")) + const folderContext = folderArg ? folderArg.slice("--folder=".length) : undefined + + console.error(`Ingesting: ${sourcePath} → ${projectPath}`) + const result = await autoIngest(projectPath, sourcePath, undefined, undefined, folderContext) + if (result.cached) { + console.error(`✓ cache HIT — ${result.writtenPaths.length} files unchanged`) + } else { + console.error(`✓ ingested — ${result.writtenPaths.length} files written, ${result.reviewItems.length} review item(s), ${result.warnings.length} warning(s)`) + } + process.stdout.write(JSON.stringify({ + status: result.hardFailures.length === 0 ? "success" : "partial", + cached: result.cached, + pages: result.writtenPaths, + reviews_pending: result.reviewItems.length, + reviews: result.reviewItems.map((r) => ({ type: r.type, title: r.title, description: r.description })), + warnings: result.warnings, + hard_failures: result.hardFailures, + }, null, 2) + "\n") +} + +async function cmdDeepResearch(args: string[]) { + const [wikiRoot, ...rest] = args + const queriesArg = rest.find((a) => a.startsWith("--queries=")) + const noIngest = rest.includes("--no-ingest") + const topic = rest.filter((a) => !a.startsWith("--")).join(" ") + if (!wikiRoot || !topic) { + console.error("Usage: deep-research [--queries=q1|q2|q3] [--no-ingest]") + process.exit(1) + } + const projectPath = path.resolve(wikiRoot) + const searchQueries = queriesArg + ? queriesArg.slice("--queries=".length).split("|").map((s) => s.trim()).filter(Boolean) + : undefined + + console.error(`Researching: "${topic}" → ${projectPath}`) + const result = await deepResearch(projectPath, topic, { + searchQueries, + autoIngest: !noIngest, + }) + console.error(`✓ saved ${result.savedPath} (${result.webResultCount} sources, ${result.ingestedFiles.length} pages ingested)`) + process.stdout.write(JSON.stringify({ + status: "success", + topic: result.topic, + saved_path: result.savedPath, + web_result_count: result.webResultCount, + ingested: result.ingested, + ingested_files: result.ingestedFiles, + warnings: result.warnings, + }, null, 2) + "\n") +} + main().catch((err) => { console.error("Error:", err instanceof Error ? err.message : err) process.exit(1) diff --git a/skill/src/fs-node.ts b/skill/src/fs-node.ts deleted file mode 100644 index 0512ed89..00000000 --- a/skill/src/fs-node.ts +++ /dev/null @@ -1,156 +0,0 @@ -/** - * Node.js drop-in replacement for Tauri's @/commands/fs IPC layer. - * Maps all Tauri invoke() calls to standard Node.js fs operations. - * - * Original (Tauri): invoke("read_file", { path }) - * Replacement: fs.readFileSync(path, 'utf-8') - */ -import * as fs from "fs" -import * as path from "path" - -export interface FileNode { - name: string - path: string - is_dir: boolean - children?: FileNode[] -} - -export interface WikiProject { - id: string - name: string - path: string -} - -export interface FileBase64 { - base64: string - mimeType: string -} - -// --------------------------------------------------------------------------- -// Core file operations (replaces Tauri IPC) -// --------------------------------------------------------------------------- - -export async function readFile(filePath: string): Promise { - return fs.readFileSync(filePath, "utf-8") -} - -export async function writeFile(filePath: string, contents: string): Promise { - fs.mkdirSync(path.dirname(filePath), { recursive: true }) - fs.writeFileSync(filePath, contents, "utf-8") -} - -export async function listDirectory(dirPath: string): Promise { - if (!fs.existsSync(dirPath)) { - throw new Error(`Directory not found: ${dirPath}`) - } - return _listDirRecursive(dirPath) -} - -function _listDirRecursive(dirPath: string): FileNode[] { - const entries = fs.readdirSync(dirPath, { withFileTypes: true }) - return entries.map((entry) => { - const fullPath = path.join(dirPath, entry.name) - if (entry.isDirectory()) { - return { - name: entry.name, - path: fullPath, - is_dir: true, - children: _listDirRecursive(fullPath), - } - } - return { - name: entry.name, - path: fullPath, - is_dir: false, - } - }) -} - -export async function copyFile(source: string, destination: string): Promise { - fs.mkdirSync(path.dirname(destination), { recursive: true }) - fs.copyFileSync(source, destination) -} - -export async function deleteFile(filePath: string): Promise { - if (fs.existsSync(filePath)) { - fs.unlinkSync(filePath) - } -} - -export async function createDirectory(dirPath: string): Promise { - fs.mkdirSync(dirPath, { recursive: true }) -} - -export async function fileExists(filePath: string): Promise { - return fs.existsSync(filePath) -} - -export async function readFileAsBase64(filePath: string): Promise { - const buffer = fs.readFileSync(filePath) - const base64 = buffer.toString("base64") - const ext = path.extname(filePath).toLowerCase() - const mimeMap: Record = { - ".jpg": "image/jpeg", - ".jpeg": "image/jpeg", - ".png": "image/png", - ".gif": "image/gif", - ".webp": "image/webp", - ".pdf": "application/pdf", - } - return { base64, mimeType: mimeMap[ext] ?? "application/octet-stream" } -} - -export async function preprocessFile(filePath: string): Promise { - // For non-GUI mode: just read the file as text - // Real Tauri version uses Rust pdf-extract / docx-rs for binary formats - return fs.readFileSync(filePath, "utf-8") -} - -export async function findRelatedWikiPages( - projectPath: string, - sourceName: string, -): Promise { - const wikiDir = path.join(projectPath, "wiki") - if (!fs.existsSync(wikiDir)) return [] - - const results: string[] = [] - const searchTerm = path.basename(sourceName, path.extname(sourceName)).toLowerCase() - - function searchDir(dir: string) { - const entries = fs.readdirSync(dir, { withFileTypes: true }) - for (const entry of entries) { - const fullPath = path.join(dir, entry.name) - if (entry.isDirectory()) { - searchDir(fullPath) - } else if (entry.name.endsWith(".md")) { - try { - const content = fs.readFileSync(fullPath, "utf-8") - if (content.toLowerCase().includes(searchTerm)) { - results.push(fullPath) - } - } catch { - // skip unreadable files - } - } - } - } - - searchDir(wikiDir) - return results -} - -export async function createProject(name: string, projectPath: string): Promise { - fs.mkdirSync(projectPath, { recursive: true }) - return { id: path.basename(projectPath), name, path: projectPath } -} - -export async function openProject(projectPath: string): Promise { - if (!fs.existsSync(projectPath)) { - throw new Error(`Project not found: ${projectPath}`) - } - return { - id: path.basename(projectPath), - name: path.basename(projectPath), - path: projectPath, - } -} diff --git a/skill/src/lib/deep-research.ts b/skill/src/lib/deep-research.ts new file mode 100644 index 00000000..0657037b --- /dev/null +++ b/skill/src/lib/deep-research.ts @@ -0,0 +1,222 @@ +/** + * deep-research.ts — Node.js port of nashsu/llm_wiki's deep research flow. + * + * Pipeline: + * 1. Multi-query web search (Tavily) with URL deduplication. + * 2. LLM synthesis into a `wiki/queries/research--.md` page, + * with cross-references to existing wiki pages via [[wikilinks]]. + * 3. Optional auto-ingest of the synthesis page to extract entities/concepts. + * + * Compared to upstream: + * - No Zustand research-store / queue / panel UI — the function runs + * synchronously and returns the saved path. + * - Auto-ingest is opt-in via the `autoIngest` parameter (default true). + */ +import { readFile, writeFile } from "../shims/fs-node" +import { streamChat } from "./llm-client" +import { webSearch } from "./web-search" +import { autoIngest } from "./ingest" +import { normalizePath } from "./path-utils" +import { buildLanguageDirective } from "./output-language" +import { useWikiStore, useActivityStore } from "../shims/stores-node" +import type { LlmConfig } from "../shims/stores-node" + +export interface DeepResearchOptions { + /** Optional explicit search queries; defaults to [topic]. */ + searchQueries?: string[] + /** Max results per query (default 5). */ + maxResultsPerQuery?: number + /** Whether to auto-ingest the synthesis page (default true). */ + autoIngest?: boolean + /** Override LLM config (defaults to env-driven config). */ + llmConfig?: LlmConfig + /** Cancellation signal. */ + signal?: AbortSignal +} + +export interface DeepResearchResult { + topic: string + savedPath: string + fullPath: string + webResultCount: number + ingested: boolean + ingestedFiles: string[] + warnings: string[] +} + +function slugify(s: string): string { + return s.toLowerCase() + .replace(/[^a-z0-9\u4e00-\u9fff\s-]/g, "") + .trim() + .replace(/\s+/g, "-") + .slice(0, 50) || "topic" +} + +export async function deepResearch( + projectPath: string, + topic: string, + opts: DeepResearchOptions = {}, +): Promise { + const pp = normalizePath(projectPath) + const llmConfig = opts.llmConfig ?? useWikiStore.getState().llmConfig + const activity = useActivityStore + const activityId = activity.addItem({ + type: "deep-research", title: topic, status: "running", detail: "Starting...", filesWritten: [], + }) + + if (!llmConfig.apiKey && !llmConfig.baseUrl) { + const msg = "No LLM configured: set OPENAI_API_KEY (or LLM_API_KEY) and optionally LLM_BASE_URL / LLM_MODEL." + activity.updateItem(activityId, { status: "error", detail: msg }) + throw new Error(msg) + } + + const warnings: string[] = [] + const queries = opts.searchQueries && opts.searchQueries.length > 0 ? opts.searchQueries : [topic] + const maxResults = opts.maxResultsPerQuery ?? 5 + + // Step 1: web search (multi-query, dedup by URL) + activity.updateItem(activityId, { detail: `Searching web (${queries.length} queries)...` }) + const seenUrls = new Set() + const allResults: { title: string; url: string; content: string; score: number }[] = [] + for (const q of queries) { + try { + const resp = await webSearch(q, maxResults) + for (const r of resp.results) { + if (!r.url || seenUrls.has(r.url)) continue + seenUrls.add(r.url) + allResults.push(r) + } + } catch (err) { + const msg = `Web search failed for "${q}": ${err instanceof Error ? err.message : err}` + warnings.push(msg) + console.warn(`[deep-research] ${msg}`) + } + } + + if (allResults.length === 0) { + const msg = "No web search results — check TAVILY_API_KEY or query terms." + activity.updateItem(activityId, { status: "error", detail: msg }) + throw new Error(msg) + } + + // Step 2: LLM synthesis + activity.updateItem(activityId, { detail: `Synthesizing (${allResults.length} sources)...` }) + + let wikiIndex = "" + try { wikiIndex = await readFile(`${pp}/wiki/index.md`) } catch { /* no index yet */ } + + const searchContext = allResults + .map((r, i) => `[${i + 1}] **${r.title}** (${new URL(r.url).hostname})\n${r.content}`) + .join("\n\n") + + const systemPrompt = [ + "You are a research assistant. Synthesize the web search results into a comprehensive wiki page.", + "", + buildLanguageDirective(topic), + "", + "## Cross-referencing (IMPORTANT)", + "- The wiki already has existing pages listed in the Wiki Index below.", + "- When your synthesis mentions an entity or concept that exists in the wiki, ALWAYS use [[wikilink]] syntax.", + "", + "## Writing Rules", + "- Organize into clear sections with headings", + "- Cite web sources using [N] notation", + "- Note contradictions or gaps", + "- Neutral, encyclopedic tone", + "", + wikiIndex ? `## Existing Wiki Index\n${wikiIndex}` : "", + ].filter(Boolean).join("\n") + + let synthesis = "" + let streamError: Error | null = null + await streamChat( + llmConfig, + [ + { role: "system", content: systemPrompt }, + { role: "user", content: `Research topic: **${topic}**\n\n## Web Search Results\n\n${searchContext}\n\nSynthesize into a wiki page.` }, + ], + { + onToken: (t) => { synthesis += t }, + onDone: () => {}, + onError: (err) => { streamError = err }, + }, + opts.signal, + { temperature: 0.2 }, + ) + if (streamError) { + activity.updateItem(activityId, { status: "error", detail: `Synthesis failed: ${(streamError as Error).message}` }) + throw streamError + } + + // Step 3: write wiki/queries/research--.md + const date = new Date().toISOString().slice(0, 10) + const slug = slugify(topic) + const fileName = `research-${slug}-${date}.md` + const savedPath = `wiki/queries/${fileName}` + const fullPath = `${pp}/${savedPath}` + + const references = allResults.map((r, i) => { + let host = "" + try { host = new URL(r.url).hostname } catch { host = "" } + return `${i + 1}. [${r.title}](${r.url})${host ? ` — ${host}` : ""}` + }).join("\n") + + const cleanedSynthesis = synthesis + .replace(/\s*[\s\S]*?<\/think(?:ing)?>\s*/gi, "") + .replace(/\s*[\s\S]*$/gi, "") + .trimStart() + + const pageContent = [ + "---", + "type: query", + `title: "Research: ${topic.replace(/"/g, '\\"')}"`, + `created: ${date}`, + "origin: deep-research", + "tags: [research]", + `sources: ["${fileName}"]`, + "---", + "", + `# Research: ${topic}`, + "", + cleanedSynthesis, + "", + "## References", + "", + references, + "", + ].join("\n") + + await writeFile(fullPath, pageContent) + + // Step 4: optional auto-ingest + let ingested = false + let ingestedFiles: string[] = [] + if (opts.autoIngest !== false) { + activity.updateItem(activityId, { detail: "Auto-ingesting research result..." }) + try { + const result = await autoIngest(pp, fullPath, llmConfig, opts.signal) + ingested = true + ingestedFiles = result.writtenPaths + } catch (err) { + const msg = `Auto-ingest failed: ${err instanceof Error ? err.message : err}` + warnings.push(msg) + console.warn(`[deep-research] ${msg}`) + } + } + + activity.updateItem(activityId, { + status: "done", + detail: ingested ? `Done — saved ${savedPath}, ingested ${ingestedFiles.length} files` : `Done — saved ${savedPath}`, + filesWritten: [savedPath, ...ingestedFiles], + }) + + return { + topic, + savedPath, + fullPath, + webResultCount: allResults.length, + ingested, + ingestedFiles, + warnings, + } +} diff --git a/skill/src/lib/ingest.ts b/skill/src/lib/ingest.ts new file mode 100644 index 00000000..fdb0071e --- /dev/null +++ b/skill/src/lib/ingest.ts @@ -0,0 +1,616 @@ +/** + * ingest.ts — Node.js port of nashsu/llm_wiki's two-stage ingest pipeline. + * + * Compared to the upstream Tauri/React version (~1600 lines), this port + * intentionally drops: + * - Image extraction (PDF/PPTX/DOCX) — needs Rust pdfium + * - Vision-LLM caption pipeline — needs a multimodal endpoint + * - Embedding generation — optional, has its own shim + * - Review-store mutations — UI-side queue (we still PARSE + * REVIEW blocks and surface them + * in the return value) + * - Activity-store streaming — UI side; we use SKILL_VERBOSE + * + * The two-stage prompt structure, the FILE-block parser, the path-traversal + * guard, the language-mismatch guard, the SHA256 incremental cache, and + * the per-page LLM merge are all preserved verbatim from upstream. + */ +import { readFile, writeFile, fileExists } from "../shims/fs-node" +import { streamChat } from "./llm-client" +import type { LlmConfig } from "../shims/stores-node" +import { useWikiStore, useActivityStore } from "../shims/stores-node" +import { getFileName, normalizePath } from "./path-utils" +import { checkIngestCache, saveIngestCache } from "./ingest-cache" +import { sanitizeIngestedFileContent } from "./ingest-sanitize" +import { mergePageContent, type MergeFn } from "./page-merge" +import { withProjectLock } from "./project-mutex" +import { buildLanguageDirective } from "./output-language" +import { detectLanguage } from "./detect-language" + +// ── Types ──────────────────────────────────────────────────────────────────── + +export interface ParsedFileBlock { + path: string + content: string +} + +export interface ParseFileBlocksResult { + blocks: ParsedFileBlock[] + warnings: string[] +} + +export interface ReviewItem { + type: "contradiction" | "duplicate" | "missing-page" | "suggestion" | "confirm" + title: string + description: string + sourcePath: string + affectedPages?: string[] + searchQueries?: string[] + options: { label: string; action: string }[] +} + +export interface IngestResult { + writtenPaths: string[] + warnings: string[] + hardFailures: string[] + reviewItems: ReviewItem[] + cached: boolean +} + +// ── FILE-block parser (preserved from upstream) ────────────────────────────── + +const OPENER_LINE = /^---\s*FILE:\s*(.+?)\s*---\s*$/i +const CLOSER_LINE = /^---\s*END\s+FILE\s*---\s*$/i +const FENCE_LINE = /^\s{0,3}(```+|~~~+)/ + +/** + * Reject FILE block paths that try to escape the project's wiki/ directory. + * Identical to upstream isSafeIngestPath — see upstream comment for the + * threat model (LLM prompt-injection via ../../../ in source documents). + */ +export function isSafeIngestPath(p: string): boolean { + if (typeof p !== "string" || p.trim().length === 0) return false + if (/[\x00-\x1f]/.test(p)) return false + if (p.startsWith("/") || p.startsWith("\\")) return false + if (/^[a-zA-Z]:/.test(p)) return false + const normalized = p.replace(/\\/g, "/") + if (normalized.split("/").some((seg) => seg === "..")) return false + if (!normalized.startsWith("wiki/")) return false + return true +} + +export function parseFileBlocks(text: string): ParseFileBlocksResult { + const normalized = text.replace(/\r\n/g, "\n") + const lines = normalized.split("\n") + const blocks: ParsedFileBlock[] = [] + const warnings: string[] = [] + + let i = 0 + while (i < lines.length) { + const openerMatch = OPENER_LINE.exec(lines[i]) + if (!openerMatch) { i++; continue } + const path = openerMatch[1].trim() + i++ + + const contentLines: string[] = [] + let fenceMarker: string | null = null + let fenceLen = 0 + let closed = false + + while (i < lines.length) { + const line = lines[i] + const fenceMatch = FENCE_LINE.exec(line) + if (fenceMatch) { + const run = fenceMatch[1] + const ch = run[0] + const len = run.length + if (fenceMarker === null) { fenceMarker = ch; fenceLen = len } + else if (ch === fenceMarker && len >= fenceLen) { fenceMarker = null; fenceLen = 0 } + contentLines.push(line); i++; continue + } + if (fenceMarker === null && CLOSER_LINE.test(line)) { closed = true; i++; break } + contentLines.push(line); i++ + } + + if (!closed) { + const msg = `FILE block "${path || "(unnamed)"}" was not closed before end of stream — likely truncation. Block dropped.` + console.warn(`[ingest] ${msg}`) + warnings.push(msg) + continue + } + if (!path) { + const msg = "FILE block with empty path skipped." + console.warn(`[ingest] ${msg}`) + warnings.push(msg) + continue + } + if (!isSafeIngestPath(path)) { + const msg = `FILE block with unsafe path "${path}" rejected (must be under wiki/, no .., no absolute paths).` + console.warn(`[ingest] ${msg}`) + warnings.push(msg) + continue + } + blocks.push({ path, content: contentLines.join("\n") }) + } + + return { blocks, warnings } +} + +// ── REVIEW-block parser (preserved from upstream, returned as data) ────────── + +const REVIEW_BLOCK_REGEX = /---REVIEW:\s*(\w[\w-]*)\s*\|\s*(.+?)\s*---\n([\s\S]*?)---END REVIEW---/g + +export function parseReviewBlocks(text: string, sourcePath: string): ReviewItem[] { + const items: ReviewItem[] = [] + const matches = text.matchAll(REVIEW_BLOCK_REGEX) + for (const m of matches) { + const rawType = m[1].trim().toLowerCase() + const title = m[2].trim() + const body = m[3].trim() + const type = ( + ["contradiction", "duplicate", "missing-page", "suggestion"].includes(rawType) + ? rawType + : "confirm" + ) as ReviewItem["type"] + + const optionsMatch = body.match(/^OPTIONS:\s*(.+)$/m) + const options = optionsMatch + ? optionsMatch[1].split("|").map((o) => { const label = o.trim(); return { label, action: label } }) + : [{ label: "Approve", action: "Approve" }, { label: "Skip", action: "Skip" }] + + const pagesMatch = body.match(/^PAGES:\s*(.+)$/m) + const affectedPages = pagesMatch ? pagesMatch[1].split(",").map((p) => p.trim()) : undefined + + const searchMatch = body.match(/^SEARCH:\s*(.+)$/m) + const searchQueries = searchMatch + ? searchMatch[1].split("|").map((q) => q.trim()).filter((q) => q.length > 0) + : undefined + + const description = body + .replace(/^OPTIONS:.*$/m, "") + .replace(/^PAGES:.*$/m, "") + .replace(/^SEARCH:.*$/m, "") + .trim() + + items.push({ type, title, description, sourcePath, affectedPages, searchQueries, options }) + } + return items +} + +// ── Prompts (preserved from upstream) ──────────────────────────────────────── + +export function languageRule(sourceContent: string = ""): string { + return buildLanguageDirective(sourceContent) +} + +export function buildAnalysisPrompt(purpose: string, index: string, sourceContent: string = ""): string { + return [ + "You are an expert research analyst. Read the source document and produce a structured analysis.", + "", + languageRule(sourceContent), + "", + "## Key Entities", + "List people, organizations, products, datasets, tools mentioned. For each:", + "- Name and type", + "- Role in the source (central vs. peripheral)", + "- Whether it likely already exists in the wiki (check the index)", + "", + "## Key Concepts", + "List theories, methods, techniques, phenomena. For each:", + "- Name and brief definition", + "- Why it matters in this source", + "", + "## Main Arguments & Findings", + "- What are the core claims or results?", + "- What evidence supports them?", + "", + "## Connections to Existing Wiki", + "- What existing pages does this source relate to?", + "", + "## Recommendations", + "- What wiki pages should be created or updated?", + "", + purpose ? `## Wiki Purpose\n${purpose}` : "", + index ? `## Current Wiki Index\n${index}` : "", + ].filter(Boolean).join("\n") +} + +export function buildGenerationPrompt( + schema: string, + purpose: string, + index: string, + sourceFileName: string, + overview?: string, + sourceContent: string = "", +): string { + const sourceBaseName = sourceFileName.replace(/\.[^.]+$/, "") + return [ + "You are a wiki maintainer. Based on the analysis provided, generate wiki files.", + "", + languageRule(sourceContent), + "", + `## IMPORTANT: Source File`, + `The original source file is: **${sourceFileName}**`, + `All wiki pages generated from this source MUST include this filename in their frontmatter \`sources\` field.`, + "", + "## What to generate", + `1. A source summary page at **wiki/sources/${sourceBaseName}.md** (MUST use this exact path)`, + "2. Entity pages in wiki/entities/ for key entities identified in the analysis", + "3. Concept pages in wiki/concepts/ for key concepts identified in the analysis", + "4. An updated wiki/index.md — add new entries to existing categories", + "5. A log entry for wiki/log.md (just the new entry to append, format: ## [YYYY-MM-DD] ingest | Title)", + "6. An updated wiki/overview.md — high-level summary of ALL topics", + "", + "## Frontmatter Rules (CRITICAL — parser is strict)", + "1. The VERY FIRST line MUST be exactly `---`. Do NOT wrap in ```yaml fences.", + "2. Each line is `key: value`.", + "3. Frontmatter ends with another `---`.", + "4. Arrays use inline form: `tags: [a, b, c]`. Wikilinks belong in the BODY only.", + "", + "Required fields:", + " • type — source | entity | concept | comparison | query | synthesis", + " • title — string (quote if it contains a colon)", + " • created — YYYY-MM-DD", + " • updated — YYYY-MM-DD", + " • tags — array of bare strings", + " • related — array of bare wiki page slugs (no `wiki/`, no `.md`, no `[[…]]`)", + ` • sources — array of source filenames; MUST include "${sourceFileName}".`, + "", + "Use [[wikilink]] syntax in the BODY for cross-references. Use kebab-case filenames.", + "", + purpose ? `## Wiki Purpose\n${purpose}` : "", + schema ? `## Wiki Schema\n${schema}` : "", + index ? `## Current Wiki Index\n${index}` : "", + overview ? `## Current Overview (update this)\n${overview}` : "", + "", + "## Output Format (MUST FOLLOW EXACTLY)", + "Your ENTIRE response is FILE blocks followed by optional REVIEW blocks. Nothing else.", + "", + "FILE block template:", + "---FILE: wiki/path/to/page.md---", + "(complete file content with YAML frontmatter)", + "---END FILE---", + "", + "REVIEW block template (optional):", + "---REVIEW: type | Title---", + "Description.", + "OPTIONS: Create Page | Skip", + "---END REVIEW---", + "", + "## Output Requirements (STRICT)", + "1. The FIRST character of your response MUST be `-` (the opening of `---FILE:`).", + "2. NO preamble, NO trailing prose, NO restating the analysis.", + "3. Between blocks, only blank lines.", + "", + "---", + "", + languageRule(sourceContent), + ].filter(Boolean).join("\n") +} + +// ── Language guard (preserved) ─────────────────────────────────────────────── + +function contentMatchesTargetLanguage(content: string, target: string): boolean { + const fmEnd = content.indexOf("\n---\n", 3) + let body = fmEnd > 0 ? content.slice(fmEnd + 5) : content + body = body + .replace(/```[\s\S]*?```/g, "") + .replace(/\$\$[\s\S]*?\$\$/g, "") + .replace(/\$[^$\n]*\$/g, "") + const sample = body.slice(0, 1500) + if (sample.trim().length < 20) return true + const detected = detectLanguage(sample) + const cjk = new Set(["Chinese", "Traditional Chinese", "Japanese", "Korean"]) + const targetIsCjk = cjk.has(target) + const detectedIsCjk = cjk.has(detected) + if (targetIsCjk) return detectedIsCjk + return !detectedIsCjk && !["Arabic", "Hindi", "Thai", "Hebrew"].includes(detected) +} + +// ── Helpers ────────────────────────────────────────────────────────────────── + +async function tryReadFile(path: string): Promise { + try { return await readFile(path) } catch { return "" } +} + +async function backupExistingPage(projectPath: string, relativePath: string, existingContent: string): Promise { + const stamp = new Date().toISOString().replace(/[:.]/g, "-") + const sanitized = relativePath.replace(/[/\\]/g, "_") + await writeFile(`${projectPath}/.llm-wiki/page-history/${sanitized}-${stamp}`, existingContent) +} + +function buildPageMerger(llmConfig: LlmConfig): MergeFn { + return async (existingContent, incomingContent, sourceFileName, signal) => { + const systemPrompt = [ + "You are merging two versions of the same wiki page into one coherent document.", + "Output ONE merged version that:", + "- Preserves every factual claim from both versions", + "- Eliminates redundancy", + "- Uses consistent markdown structure", + "- Keeps `[[wikilink]]` references intact", + "Output requirements:", + "- The FIRST character MUST be `-` (the opening of `---`)", + "- Output the COMPLETE file: YAML frontmatter + body", + "- No preamble, no analysis prose", + ].join("\n") + const userMessage = [ + "## Existing version on disk", "", existingContent, "", + "---", "", + `## Newly generated version (from ${sourceFileName})`, "", incomingContent, "", + "---", "", + "Now output the merged file. Start with `---` on the first line.", + ].join("\n") + let result = "" + let streamError: Error | null = null + await new Promise((resolve) => { + streamChat(llmConfig, [ + { role: "system", content: systemPrompt }, + { role: "user", content: userMessage }, + ], { + onToken: (t) => { result += t }, + onDone: () => resolve(), + onError: (err) => { streamError = err; resolve() }, + }, signal, { temperature: 0.1 }).catch((err) => { + streamError = err instanceof Error ? err : new Error(String(err)) + resolve() + }) + }) + if (streamError) throw streamError + return result + } +} + +// ── Write FILE blocks (preserved) ──────────────────────────────────────────── + +async function writeFileBlocks( + projectPath: string, + text: string, + llmConfig: LlmConfig, + sourceFileName: string, + signal?: AbortSignal, +): Promise<{ writtenPaths: string[]; warnings: string[]; hardFailures: string[] }> { + const { blocks, warnings: parseWarnings } = parseFileBlocks(text) + const warnings = [...parseWarnings] + const writtenPaths: string[] = [] + const hardFailures: string[] = [] + + const targetLang = process.env.WIKI_OUTPUT_LANGUAGE && process.env.WIKI_OUTPUT_LANGUAGE !== "auto" + ? process.env.WIKI_OUTPUT_LANGUAGE + : "" + + for (const { path: relativePath, content: rawContent } of blocks) { + const content = sanitizeIngestedFileContent(rawContent) + + const isLog = relativePath.endsWith("/log.md") || relativePath === "wiki/log.md" + const isEntityOrSource = + relativePath.startsWith("wiki/entities/") || + relativePath.includes("/entities/") || + relativePath.startsWith("wiki/sources/") || + relativePath.includes("/sources/") + if (targetLang && !isLog && !isEntityOrSource && !contentMatchesTargetLanguage(content, targetLang)) { + const msg = `Dropped "${relativePath}" — body language doesn't match target ${targetLang}.` + console.warn(`[ingest] ${msg}`) + warnings.push(msg) + continue + } + + const fullPath = `${projectPath}/${relativePath}` + try { + if (isLog) { + const existing = await tryReadFile(fullPath) + const appended = existing ? `${existing}\n\n${content.trim()}` : content.trim() + await writeFile(fullPath, appended) + } else if ( + relativePath === "wiki/index.md" || relativePath.endsWith("/index.md") || + relativePath === "wiki/overview.md" || relativePath.endsWith("/overview.md") + ) { + await writeFile(fullPath, content) + } else { + const existing = await tryReadFile(fullPath) + const toWrite = await mergePageContent( + content, + existing || null, + buildPageMerger(llmConfig), + { + sourceFileName, + pagePath: relativePath, + signal, + backup: (oldContent) => backupExistingPage(projectPath, relativePath, oldContent), + }, + ) + await writeFile(fullPath, toWrite) + } + writtenPaths.push(relativePath) + } catch (err) { + const msg = `Failed to write "${relativePath}": ${err instanceof Error ? err.message : String(err)}` + console.error(`[ingest] ${msg}`) + warnings.push(msg) + hardFailures.push(relativePath) + } + } + + return { writtenPaths, warnings, hardFailures } +} + +// ── Main entry: autoIngest ─────────────────────────────────────────────────── + +export async function autoIngest( + projectPath: string, + sourcePath: string, + llmConfig?: LlmConfig, + signal?: AbortSignal, + folderContext?: string, +): Promise { + const cfg = llmConfig ?? useWikiStore.getState().llmConfig + return withProjectLock(normalizePath(projectPath), () => + autoIngestImpl(projectPath, sourcePath, cfg, signal, folderContext), + ) +} + +async function autoIngestImpl( + projectPath: string, + sourcePath: string, + llmConfig: LlmConfig, + signal?: AbortSignal, + folderContext?: string, +): Promise { + const pp = normalizePath(projectPath) + const sp = normalizePath(sourcePath) + const fileName = getFileName(sp) + const activity = useActivityStore + const activityId = activity.addItem({ + type: "ingest", title: fileName, status: "running", detail: "Reading source...", filesWritten: [], + }) + + if (!llmConfig.apiKey && !llmConfig.baseUrl) { + const msg = "No LLM configured: set OPENAI_API_KEY (or LLM_API_KEY) and optionally LLM_BASE_URL / LLM_MODEL." + activity.updateItem(activityId, { status: "error", detail: msg }) + throw new Error(msg) + } + + const [sourceContent, schema, purpose, index, overview] = await Promise.all([ + tryReadFile(sp), + tryReadFile(`${pp}/schema.md`), + tryReadFile(`${pp}/purpose.md`), + tryReadFile(`${pp}/wiki/index.md`), + tryReadFile(`${pp}/wiki/overview.md`), + ]) + + if (!sourceContent.trim()) { + const msg = `Source file "${fileName}" is empty or unreadable.` + activity.updateItem(activityId, { status: "error", detail: msg }) + throw new Error(msg) + } + + // Cache check + const cachedFiles = await checkIngestCache(pp, fileName, sourceContent) + if (cachedFiles !== null) { + activity.updateItem(activityId, { + status: "done", + detail: `Skipped (unchanged) — ${cachedFiles.length} files from previous ingest`, + filesWritten: cachedFiles, + }) + return { writtenPaths: cachedFiles, warnings: [], hardFailures: [], reviewItems: [], cached: true } + } + + const truncatedContent = sourceContent.length > 50000 + ? sourceContent.slice(0, 50000) + "\n\n[...truncated...]" + : sourceContent + + // Stage 1: analysis + activity.updateItem(activityId, { detail: "Step 1/2: Analyzing source..." }) + let analysis = "" + let stage1Error: Error | null = null + await streamChat( + llmConfig, + [ + { role: "system", content: buildAnalysisPrompt(purpose, index, truncatedContent) }, + { role: "user", content: `Analyze this source document:\n\n**File:** ${fileName}${folderContext ? `\n**Folder context:** ${folderContext}` : ""}\n\n---\n\n${truncatedContent}` }, + ], + { + onToken: (t) => { analysis += t }, + onDone: () => {}, + onError: (err) => { stage1Error = err }, + }, + signal, + { temperature: 0.1 }, + ) + if (stage1Error) { + activity.updateItem(activityId, { status: "error", detail: `Analysis failed: ${(stage1Error as Error).message}` }) + throw stage1Error + } + + // Stage 2: generation + activity.updateItem(activityId, { detail: "Step 2/2: Generating wiki pages..." }) + let generation = "" + let stage2Error: Error | null = null + await streamChat( + llmConfig, + [ + { role: "system", content: buildGenerationPrompt(schema, purpose, index, fileName, overview, truncatedContent) }, + { + role: "user", + content: [ + `Source document to process: **${fileName}**`, + "", + "The Stage 1 analysis below is CONTEXT. Do NOT echo it. Output FILE/REVIEW blocks only.", + "", + "## Stage 1 Analysis (context only)", + "", analysis, "", + "## Original Source Content", + "", truncatedContent, "", + "---", + "", + `Now emit the FILE blocks for the wiki files derived from **${fileName}**.`, + "Your response MUST begin with `---FILE:` as the very first characters.", + ].join("\n"), + }, + ], + { + onToken: (t) => { generation += t }, + onDone: () => {}, + onError: (err) => { stage2Error = err }, + }, + signal, + { temperature: 0.1 }, + ) + if (stage2Error) { + activity.updateItem(activityId, { status: "error", detail: `Generation failed: ${(stage2Error as Error).message}` }) + throw stage2Error + } + + // Stage 3: write + activity.updateItem(activityId, { detail: "Writing files..." }) + const { writtenPaths, warnings, hardFailures } = await writeFileBlocks(pp, generation, llmConfig, fileName, signal) + + // Fallback: ensure at least a source-summary page exists + const sourceBaseName = fileName.replace(/\.[^.]+$/, "") + const sourceSummaryPath = `wiki/sources/${sourceBaseName}.md` + const hasSourceSummary = writtenPaths.some((p) => p.startsWith("wiki/sources/")) + if (!hasSourceSummary && !signal?.aborted) { + const date = new Date().toISOString().slice(0, 10) + const fallbackContent = [ + "---", + "type: source", + `title: "Source: ${fileName}"`, + `created: ${date}`, + `updated: ${date}`, + `sources: ["${fileName}"]`, + "tags: []", + "related: []", + "---", + "", + `# Source: ${fileName}`, + "", + analysis ? analysis.slice(0, 3000) : "(Analysis not available)", + "", + ].join("\n") + try { + await writeFile(`${pp}/${sourceSummaryPath}`, fallbackContent) + writtenPaths.push(sourceSummaryPath) + } catch { /* non-critical */ } + } + + const reviewItems = parseReviewBlocks(generation, sp) + + // Cache only on full success + if (writtenPaths.length > 0 && hardFailures.length === 0) { + await saveIngestCache(pp, fileName, sourceContent, writtenPaths) + } + + // Best-effort: bump dataVersion so callers can invalidate caches + try { useWikiStore.setState((s) => ({ dataVersion: s.dataVersion + 1 })) } catch { /* ignore */ } + + const detail = writtenPaths.length > 0 + ? `${writtenPaths.length} files written${reviewItems.length > 0 ? `, ${reviewItems.length} review item(s)` : ""}` + : "No files generated" + activity.updateItem(activityId, { + status: writtenPaths.length > 0 ? "done" : "error", + detail, + filesWritten: writtenPaths, + }) + + // fileExists is imported but only used by the cache; keep referenced for tree-shake clarity + void fileExists + return { writtenPaths, warnings, hardFailures, reviewItems, cached: false } +} diff --git a/skill/src/lib/web-search.ts b/skill/src/lib/web-search.ts index 4eb8f408..615e81fa 100644 --- a/skill/src/lib/web-search.ts +++ b/skill/src/lib/web-search.ts @@ -23,7 +23,8 @@ export async function webSearch(query: string, maxResults: number = 5): Promise< } try { - const response = await fetch("https://api.tavily.com/search", { + const baseUrl = (process.env.TAVILY_BASE_URL ?? "https://api.tavily.com").replace(/\/+$/, "") + const response = await fetch(`${baseUrl}/search`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ diff --git a/skill/src/mcp-server.ts b/skill/src/mcp-server.ts index 769e6ec0..b9a32a33 100644 --- a/skill/src/mcp-server.ts +++ b/skill/src/mcp-server.ts @@ -29,6 +29,8 @@ import * as path from "path" import { buildWikiGraph } from "./lib/wiki-graph" import { findSurprisingConnections, detectKnowledgeGaps } from "./lib/graph-insights" import { searchWiki } from "./lib/search" +import { autoIngest } from "./lib/ingest" +import { deepResearch } from "./lib/deep-research" const DEFAULT_WIKI_PATH = process.env.WIKI_PATH ?? process.cwd() const PKG_VERSION = "0.4.6-mcp" @@ -108,6 +110,33 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({ required: [], }, }, + { + name: "wiki_ingest", + description: "Two-stage LLM ingest of a source markdown/text file into the wiki. Stage 1 analyzes (entities, concepts, connections); stage 2 emits FILE blocks that are written under wiki/. Requires OPENAI_API_KEY (or LLM_API_KEY) configured. SHA256 incremental cache skips unchanged sources.", + inputSchema: { + type: "object", + properties: { + source_file: { type: "string", description: "Absolute path to the source markdown/text file to ingest" }, + project_path: { type: "string", description: "Path to wiki project (defaults to WIKI_PATH env var)" }, + folder_context: { type: "string", description: "Optional folder hint for LLM categorization (e.g. 'papers/energy')" }, + }, + required: ["source_file"], + }, + }, + { + name: "wiki_deep_research", + description: "Multi-query web search → LLM synthesis → optional auto-ingest. Saves a research page to wiki/queries/ then (by default) ingests it to extract entities/concepts. Requires OPENAI_API_KEY and TAVILY_API_KEY.", + inputSchema: { + type: "object", + properties: { + topic: { type: "string", description: "Research topic (free-form)" }, + project_path: { type: "string", description: "Path to wiki project" }, + search_queries: { type: "array", items: { type: "string" }, description: "Optional explicit search queries (defaults to [topic])" }, + auto_ingest: { type: "boolean", description: "Whether to auto-ingest the synthesis page (default true)" }, + }, + required: ["topic"], + }, + }, ], })) @@ -225,6 +254,61 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => { return { content: [{ type: "text", text: text }] } } + case "wiki_ingest": { + if (!args.source_file) throw new McpError(ErrorCode.InvalidParams, "source_file is required") + const sourcePath = path.resolve(args.source_file as string) + const folderContext = args.folder_context as string | undefined + const result = await autoIngest(projectPath, sourcePath, undefined, undefined, folderContext) + const lines = [ + result.cached + ? `✓ Cache HIT — ${result.writtenPaths.length} files unchanged for "${path.basename(sourcePath)}"` + : `✓ Ingested "${path.basename(sourcePath)}" — ${result.writtenPaths.length} files written`, + ...result.writtenPaths.map((p) => ` - ${p}`), + ] + if (result.reviewItems.length > 0) { + lines.push("", `Review items (${result.reviewItems.length}):`) + for (const r of result.reviewItems) lines.push(` - [${r.type}] ${r.title}`) + } + if (result.warnings.length > 0) { + lines.push("", `Warnings (${result.warnings.length}):`) + for (const w of result.warnings) lines.push(` - ${w}`) + } + if (result.hardFailures.length > 0) { + lines.push("", `Hard failures (${result.hardFailures.length}):`) + for (const f of result.hardFailures) lines.push(` - ${f}`) + } + return { content: [{ type: "text", text: lines.join("\n") }] } + } + + case "wiki_deep_research": { + if (!args.topic) throw new McpError(ErrorCode.InvalidParams, "topic is required") + const topic = args.topic as string + const searchQueries = Array.isArray(args.search_queries) + ? (args.search_queries as string[]).filter((s) => typeof s === "string" && s.trim()) + : undefined + const autoIngestFlag = typeof args.auto_ingest === "boolean" ? args.auto_ingest : true + const result = await deepResearch(projectPath, topic, { + searchQueries, + autoIngest: autoIngestFlag, + }) + const lines = [ + `✓ Deep research on "${topic}" complete`, + ` Saved: ${result.savedPath}`, + ` Web results: ${result.webResultCount}`, + result.ingested + ? ` Auto-ingested ${result.ingestedFiles.length} wiki page(s)` + : ` Auto-ingest skipped`, + ] + if (result.ingestedFiles.length > 0) { + for (const f of result.ingestedFiles) lines.push(` - ${f}`) + } + if (result.warnings.length > 0) { + lines.push("", `Warnings (${result.warnings.length}):`) + for (const w of result.warnings) lines.push(` - ${w}`) + } + return { content: [{ type: "text", text: lines.join("\n") }] } + } + default: throw new McpError(ErrorCode.MethodNotFound, `Unknown tool: ${name}`) } diff --git a/skill/src/stores-node.ts b/skill/src/stores-node.ts deleted file mode 100644 index b18d2a62..00000000 --- a/skill/src/stores-node.ts +++ /dev/null @@ -1,160 +0,0 @@ -/** - * Node.js state management replacement for React stores. - * Replaces zustand-based stores with simple module-level state. - * - * Affected stores: - * @/stores/wiki-store → wikiStore - * @/stores/research-store → researchStore - * @/stores/chat-store → chatStore - * @/stores/activity-store → activityStore - * @/stores/review-store → reviewStore - */ - -// --------------------------------------------------------------------------- -// Types -// --------------------------------------------------------------------------- - -export interface LlmConfig { - provider: "openai" | "anthropic" | "google" | "ollama" | "custom" - apiKey: string - model: string - baseUrl?: string -} - -export interface EmbeddingConfig { - enabled: boolean - model: string - apiBase?: string -} - -export interface SearchApiConfig { - provider: "tavily" | "serper" | "none" - apiKey?: string -} - -export interface ReviewItem { - id: string - filePath: string - content: string - reason: string -} - -// --------------------------------------------------------------------------- -// Wiki Store (replaces useWikiStore) -// --------------------------------------------------------------------------- - -const _wikiState = { - projectPath: "", - dataVersion: 0, - embeddingConfig: { - enabled: false, - model: "", - apiBase: undefined as string | undefined, - } as EmbeddingConfig, - llmConfig: { - provider: "openai" as const, - apiKey: process.env.OPENAI_API_KEY ?? "", - model: process.env.LLM_MODEL ?? "gpt-4o", - baseUrl: process.env.OPENAI_API_BASE, - } as LlmConfig, - fileTree: [] as unknown[], -} - -export const useWikiStore = { - getState: () => ({ - ..._wikiState, - setFileTree: (tree: unknown[]) => { _wikiState.fileTree = tree }, - bumpDataVersion: () => { _wikiState.dataVersion++ }, - }), -} - -export function configureWikiStore(opts: { - projectPath: string - llmConfig?: Partial - embeddingConfig?: Partial -}) { - _wikiState.projectPath = opts.projectPath - if (opts.llmConfig) Object.assign(_wikiState.llmConfig, opts.llmConfig) - if (opts.embeddingConfig) Object.assign(_wikiState.embeddingConfig, opts.embeddingConfig) -} - -// --------------------------------------------------------------------------- -// Research Store (replaces useResearchStore) -// --------------------------------------------------------------------------- - -interface ResearchTask { - id: string - topic: string - status: "queued" | "searching" | "synthesizing" | "saving" | "done" | "error" - searchQueries?: string[] - webResults?: unknown[] - synthesis?: string - savedPath?: string - error?: string -} - -const _researchState = { - tasks: [] as ResearchTask[], - maxConcurrent: 3, - panelOpen: false, -} - -export const useResearchStore = { - getState: () => ({ - ..._researchState, - addTask: (topic: string) => { - const id = `task-${Date.now()}-${Math.random().toString(36).slice(2)}` - _researchState.tasks.push({ id, topic, status: "queued" }) - return id - }, - updateTask: (id: string, updates: Partial) => { - const task = _researchState.tasks.find((t) => t.id === id) - if (task) Object.assign(task, updates) - }, - getNextQueued: () => _researchState.tasks.find((t) => t.status === "queued") ?? null, - getRunningCount: () => _researchState.tasks.filter( - (t) => t.status === "searching" || t.status === "synthesizing" || t.status === "saving" - ).length, - setPanelOpen: (open: boolean) => { _researchState.panelOpen = open }, - }), -} - -// --------------------------------------------------------------------------- -// Activity Store (replaces useActivityStore) -// --------------------------------------------------------------------------- - -export const useActivityStore = { - getState: () => ({ - addActivity: (msg: string) => { - console.log(`[Activity] ${msg}`) - }, - }), -} - -// --------------------------------------------------------------------------- -// Chat Store (replaces useChatStore) -// --------------------------------------------------------------------------- - -export const useChatStore = { - getState: () => ({ - addMessage: () => {}, - }), -} - -// --------------------------------------------------------------------------- -// Review Store (replaces useReviewStore) -// --------------------------------------------------------------------------- - -const _reviewState = { - items: [] as ReviewItem[], -} - -export const useReviewStore = { - getState: () => ({ - items: _reviewState.items, - addItem: (item: ReviewItem) => { _reviewState.items.push(item) }, - removeItem: (id: string) => { - _reviewState.items = _reviewState.items.filter((i) => i.id !== id) - }, - }), -} diff --git a/skill/src/test-server/e2e.ts b/skill/src/test-server/e2e.ts new file mode 100644 index 00000000..11361884 --- /dev/null +++ b/skill/src/test-server/e2e.ts @@ -0,0 +1,641 @@ +/* eslint-disable @typescript-eslint/no-var-requires */ +/** + * End-to-end test harness — runs every CLI command and every MCP tool against + * a real on-disk wiki fixture and a real local OpenAI/Tavily-compatible HTTP + * server. No code-level mocks. All paths exercise real fetch / SSE parsing / + * file I/O. + * + * Output: a markdown report at skill/docs/test-report.md plus stdout. + */ +import * as fs from "fs" +import * as path from "path" +import * as os from "os" +import { spawn, type ChildProcessWithoutNullStreams } from "child_process" +import { startFakeServer, type FakeServerHandle } from "../test-server/fake-llm-server" + +const SKILL_ROOT = path.resolve(__dirname, "../..") +const CLI = path.join(SKILL_ROOT, "dist", "cli.js") +const MCP = path.join(SKILL_ROOT, "dist", "mcp-server.js") + +interface CaseResult { + name: string + command: string + exitCode: number | null + stdout: string + stderr: string + ok: boolean + notes?: string +} + +const results: CaseResult[] = [] + +function runCli(name: string, args: string[], env: NodeJS.ProcessEnv = {}): Promise { + process.stderr.write(`[e2e] running: ${name}\n`) + return new Promise((resolve) => { + const proc = spawn(process.execPath, [CLI, ...args], { + env: { ...process.env, ...env }, + }) + let stdout = "" + let stderr = "" + proc.stdout.on("data", (d) => { stdout += d.toString() }) + proc.stderr.on("data", (d) => { stderr += d.toString() }) + const timer = setTimeout(() => { proc.kill("SIGKILL") }, 60_000) + proc.on("close", (code) => { + clearTimeout(timer) + const r: CaseResult = { + name, + command: `node dist/cli.js ${args.join(" ")}`, + exitCode: code, + stdout, + stderr, + ok: code === 0, + } + process.stderr.write(`[e2e] exit=${code} stdout=${stdout.length}B stderr=${stderr.length}B\n`) + results.push(r) + resolve(r) + }) + }) +} + +// ── Build the wiki fixture ────────────────────────────────────────────────── +const TMP = fs.mkdtempSync(path.join(os.tmpdir(), "llm-wiki-e2e-")) +const PROJECT = path.join(TMP, "project") +const RAW = path.join(TMP, "raw") +fs.mkdirSync(PROJECT, { recursive: true }) +fs.mkdirSync(RAW, { recursive: true }) + +console.log(`[e2e] fixture root: ${TMP}`) + +// Use init to scaffold (call moved into main() so we can await) +async function scaffoldFixture(): Promise { + await runCli("init", ["init", PROJECT]) +} + +// Drop a small but realistic seed wiki: a few interlinked pages +const seed: Record = { + "wiki/concepts/transformer.md": [ + "---", + "type: concept", + "title: Transformer", + "created: 2026-04-01", + "updated: 2026-04-01", + "tags: [ml, architecture]", + "related: [attention-mechanism, bert]", + 'sources: ["intro.md"]', + "---", + "", + "# Transformer", + "", + "The Transformer is a neural network architecture based on the [[attention-mechanism]].", + "It powers models like [[bert]] and modern large language models.", + "", + ].join("\n"), + "wiki/concepts/attention-mechanism.md": [ + "---", + "type: concept", + "title: Attention Mechanism", + "created: 2026-04-01", + "updated: 2026-04-01", + "tags: [ml]", + "related: [transformer]", + 'sources: ["intro.md"]', + "---", + "", + "# Attention Mechanism", + "", + "Attention lets a model focus on relevant parts of its input.", + "It is the core innovation behind the [[transformer]].", + "", + ].join("\n"), + "wiki/entities/bert.md": [ + "---", + "type: entity", + "title: BERT", + "created: 2026-04-02", + "updated: 2026-04-02", + "tags: [ml, model]", + "related: [transformer]", + 'sources: ["bert-paper.md"]', + "---", + "", + "# BERT", + "", + "BERT is a [[transformer]]-based language model from Google (2018).", + "", + ].join("\n"), + "wiki/entities/orphan-thing.md": [ + "---", + "type: entity", + "title: Orphan Thing", + "created: 2026-04-03", + "updated: 2026-04-03", + "tags: []", + "related: []", + 'sources: ["misc.md"]', + "---", + "", + "# Orphan Thing", + "", + "An entity with no inbound or outbound links — should trip the lint check.", + "", + ].join("\n"), + "wiki/sources/intro.md": [ + "---", + "type: source", + 'title: "Source: intro.md"', + "created: 2026-04-01", + "updated: 2026-04-01", + 'sources: ["intro.md"]', + "tags: []", + "related: [transformer, attention-mechanism]", + "---", + "", + "# Source: intro.md", + "", + "Introduces [[transformer]] and [[attention-mechanism]].", + "", + ].join("\n"), + "wiki/index.md": [ + "---", + "title: Index", + "type: overview", + "---", + "", + "# Knowledge Base", + "", + "## Concepts", + "- [[transformer]]", + "- [[attention-mechanism]]", + "", + "## Entities", + "- [[bert]]", + "", + ].join("\n"), +} +for (const [rel, content] of Object.entries(seed)) { + const p = path.join(PROJECT, rel) + fs.mkdirSync(path.dirname(p), { recursive: true }) + fs.writeFileSync(p, content) +} + +// Drop a raw source for ingest +const RAW_SOURCE = path.join(RAW, "rnn-vs-transformer.md") +fs.writeFileSync(RAW_SOURCE, [ + "# RNNs vs Transformers", + "", + "Recurrent Neural Networks (RNNs) process sequences token-by-token, while the", + "Transformer architecture relies on self-attention and processes the whole", + "sequence in parallel. Models like BERT and GPT are based on Transformers.", + "Mamba is a recent state-space model that revisits some RNN ideas.", + "", +].join("\n")) + +// ── Phase 1: non-LLM commands ─────────────────────────────────────────────── +async function runNonLLMPhase(): Promise { + await runCli("status", ["status", PROJECT]) + await runCli("search", ["search", PROJECT, "attention"]) + await runCli("graph", ["graph", PROJECT]) + await runCli("insights", ["insights", PROJECT]) + await runCli("lint", ["lint", PROJECT]) +} + +// ── Phase 2: ingest with the local fake LLM server ────────────────────────── +async function withFakeServer(fn: (h: FakeServerHandle) => Promise): Promise { + const handle = await startFakeServer(0) + try { return await fn(handle) } finally { await handle.close() } +} + +function ingestStage1Reply(): string[] { + return [ + "## Key Entities\n", + "- **Mamba** (model)\n", + "- **GPT** (model)\n\n", + "## Key Concepts\n", + "- **Recurrent Neural Network (RNN)**: sequential processing.\n", + "- **Self-attention**: parallel sequence modeling.\n\n", + "## Recommendations\n", + "- Create concept page for RNN.\n", + "- Create entity page for Mamba.\n", + ] +} + +function ingestStage2Reply(sourceFile: string): string[] { + // Build one big string then chunk it; easier than threading chunks + const today = new Date().toISOString().slice(0, 10) + const text = [ + `---FILE: wiki/sources/${sourceFile.replace(/\.[^.]+$/, "")}.md---`, + "---", + "type: source", + `title: "Source: ${sourceFile}"`, + `created: ${today}`, + `updated: ${today}`, + `sources: ["${sourceFile}"]`, + "tags: [ml]", + "related: [transformer, recurrent-neural-network, mamba]", + "---", + "", + `# Source: ${sourceFile}`, + "", + "Compares [[recurrent-neural-network]] and [[transformer]] approaches.", + "Notes [[mamba]] as a hybrid state-space model.", + "---END FILE---", + "", + "---FILE: wiki/concepts/recurrent-neural-network.md---", + "---", + "type: concept", + "title: Recurrent Neural Network", + `created: ${today}`, + `updated: ${today}`, + "tags: [ml, architecture]", + "related: [transformer]", + `sources: ["${sourceFile}"]`, + "---", + "", + "# Recurrent Neural Network", + "", + "RNNs process sequences token-by-token. Largely superseded by the [[transformer]].", + "---END FILE---", + "", + "---FILE: wiki/entities/mamba.md---", + "---", + "type: entity", + "title: Mamba", + `created: ${today}`, + `updated: ${today}`, + "tags: [ml, model]", + "related: [recurrent-neural-network, transformer]", + `sources: ["${sourceFile}"]`, + "---", + "", + "# Mamba", + "", + "Mamba is a state-space model that revisits some [[recurrent-neural-network]] ideas.", + "---END FILE---", + "", + "---FILE: wiki/log.md---", + `## [${today}] ingest | RNN vs Transformer`, + "Added recurrent-neural-network and mamba pages.", + "---END FILE---", + "", + "---REVIEW: suggestion | Add Linear Attention page---", + "Linear attention deserves its own page.", + "OPTIONS: Create Page | Skip", + "PAGES: wiki/concepts/transformer.md", + "SEARCH: linear attention transformer | efficient attention mechanism", + "---END REVIEW---", + ].join("\n") + // Stream in ~200-char chunks to exercise SSE buffering + const chunks: string[] = [] + for (let i = 0; i < text.length; i += 200) chunks.push(text.slice(i, i + 200)) + return chunks +} + +async function runLLMPhase(): Promise { + await withFakeServer(async (handle) => { + const env: NodeJS.ProcessEnv = { + LLM_BASE_URL: handle.baseUrl, + LLM_API_KEY: "test-key", + OPENAI_API_KEY: "test-key", + LLM_MODEL: "fake-model", + TAVILY_API_KEY: "test-key", + TAVILY_BASE_URL: handle.baseUrl, + WIKI_OUTPUT_LANGUAGE: "English", + } + + // ── ingest ──────────────────────────────────────────────── + handle.pushChat({ match: "expert research analyst", chunks: ingestStage1Reply() }) + handle.pushChat({ match: "wiki maintainer", chunks: ingestStage2Reply("rnn-vs-transformer.md") }) + await runCli("ingest", ["ingest", PROJECT, RAW_SOURCE], env) + + // Verify files actually landed on disk + const expected = [ + "wiki/sources/rnn-vs-transformer.md", + "wiki/concepts/recurrent-neural-network.md", + "wiki/entities/mamba.md", + ] + const missing = expected.filter((p) => !fs.existsSync(path.join(PROJECT, p))) + results[results.length - 1].notes = missing.length === 0 + ? `All expected files written: ${expected.join(", ")}` + : `MISSING: ${missing.join(", ")}` + if (missing.length > 0) results[results.length - 1].ok = false + + // ── ingest cache hit (re-run) ──────────────────────────── + await runCli("ingest (cache hit)", ["ingest", PROJECT, RAW_SOURCE], env) + const cacheCalls = handle.callCount().chat + results[results.length - 1].notes = `Total LLM chat calls so far: ${cacheCalls} (cache hit should not increment beyond 2)` + if (cacheCalls > 2) results[results.length - 1].ok = false + + // ── deep-research ──────────────────────────────────────── + handle.pushSearch([ + { title: "Mixture of Experts overview", url: "https://example.com/moe", content: "MoE routes tokens to experts." }, + { title: "Switch Transformer", url: "https://example.com/switch", content: "Sparse expert routing." }, + ]) + const synthesis = [ + "# Mixture of Experts (MoE)", + "", + "MoE architectures route tokens to specialized expert sub-networks [1].", + "[[transformer]] models like Switch Transformer demonstrate sparse expert routing [2].", + "", + ].join("\n") + handle.pushChat({ match: "research assistant", chunks: synthesis.match(/.{1,80}/gs) ?? [synthesis] }) + // The auto-ingest stage runs analysis + generation again on the saved page + handle.pushChat({ match: "expert research analyst", chunks: ["## Key Concepts\n- MoE\n"] }) + const today = new Date().toISOString().slice(0, 10) + const moeBlock = [ + `---FILE: wiki/sources/research-mixture-of-experts-${today}.md---`, + "---", + "type: source", + 'title: "Source: MoE research"', + `created: ${today}`, + `updated: ${today}`, + `sources: ["research-mixture-of-experts-${today}.md"]`, + "tags: [ml, research]", + "related: [mixture-of-experts, transformer]", + "---", + "# Source: MoE research", + "Summary of MoE research.", + "---END FILE---", + "", + "---FILE: wiki/concepts/mixture-of-experts.md---", + "---", + "type: concept", + "title: Mixture of Experts", + `created: ${today}`, + `updated: ${today}`, + "tags: [ml]", + "related: [transformer]", + `sources: ["research-mixture-of-experts-${today}.md"]`, + "---", + "# Mixture of Experts", + "Sparse routing across experts. See [[transformer]].", + "---END FILE---", + ].join("\n") + handle.pushChat({ match: "wiki maintainer", chunks: moeBlock.match(/.{1,200}/gs) ?? [moeBlock] }) + + await runCli("deep-research", ["deep-research", PROJECT, "Mixture of Experts"], env) + const moePath = path.join(PROJECT, "wiki/concepts/mixture-of-experts.md") + const queriesPath = path.join(PROJECT, "wiki/queries") + const queryFiles = fs.existsSync(queriesPath) ? fs.readdirSync(queriesPath) : [] + results[results.length - 1].notes = [ + `query files: ${queryFiles.join(", ") || "(none)"}`, + `mixture-of-experts page exists: ${fs.existsSync(moePath)}`, + `final calls: chat=${handle.callCount().chat}, search=${handle.callCount().search}`, + ].join(" | ") + if (!fs.existsSync(moePath) || queryFiles.length === 0) results[results.length - 1].ok = false + }) +} + +// ── Phase 3: real MCP JSON-RPC handshake + tool calls ─────────────────────── +interface RpcResult { ok: boolean; raw: string; tools?: any[]; result?: any } + +async function runMcpPhase(): Promise { + await withFakeServer(async (handle) => { + const env: NodeJS.ProcessEnv = { + ...process.env, + WIKI_PATH: PROJECT, + LLM_BASE_URL: handle.baseUrl, + LLM_API_KEY: "test-key", + OPENAI_API_KEY: "test-key", + LLM_MODEL: "fake-model", + TAVILY_API_KEY: "test-key", + TAVILY_BASE_URL: handle.baseUrl, + WIKI_OUTPUT_LANGUAGE: "English", + } + const child = spawn(process.execPath, [MCP], { env, stdio: ["pipe", "pipe", "pipe"] }) + let stdoutBuf = "" + let stderrBuf = "" + child.stdout.on("data", (d) => { stdoutBuf += d.toString() }) + child.stderr.on("data", (d) => { stderrBuf += d.toString() }) + + const sendAndWait = async (req: any): Promise => { + const before = stdoutBuf.length + child.stdin.write(JSON.stringify(req) + "\n") + const deadline = Date.now() + 30_000 + while (Date.now() < deadline) { + await new Promise((r) => setTimeout(r, 50)) + if (stdoutBuf.length > before) { + const slice = stdoutBuf.slice(before) + // Find a complete JSON line + for (const line of slice.split("\n")) { + if (!line.trim()) continue + try { + const parsed = JSON.parse(line) + if (parsed.id === req.id) { + return { ok: !parsed.error, raw: line, tools: parsed.result?.tools, result: parsed.result } + } + } catch { /* keep waiting */ } + } + } + } + return { ok: false, raw: stdoutBuf.slice(before) } + } + + const record = (name: string, req: any, resp: RpcResult, expectOk = true) => { + results.push({ + name: `mcp:${name}`, + command: `MCP ${req.method} ${req.params?.name ?? ""}`.trim(), + exitCode: 0, + stdout: resp.raw.slice(0, 2000), + stderr: "", + ok: expectOk ? resp.ok : !resp.ok, + }) + } + + // Initialize + const init = await sendAndWait({ + jsonrpc: "2.0", id: 1, method: "initialize", + params: { protocolVersion: "2024-11-05", capabilities: {}, clientInfo: { name: "e2e", version: "1.0" } }, + }) + record("initialize", { id: 1, method: "initialize" }, init) + + // List tools + const list = await sendAndWait({ jsonrpc: "2.0", id: 2, method: "tools/list" }) + record("tools/list", { id: 2, method: "tools/list" }, list) + const toolNames = (list.tools ?? []).map((t: any) => t.name) + results[results.length - 1].notes = `tools: ${toolNames.join(", ")}` + const expectedTools = ["wiki_status", "wiki_search", "wiki_graph", "wiki_insights", "wiki_lint", "wiki_ingest", "wiki_deep_research"] + const missingTools = expectedTools.filter((t) => !toolNames.includes(t)) + if (missingTools.length > 0) { + results[results.length - 1].ok = false + results[results.length - 1].notes += ` | MISSING: ${missingTools.join(", ")}` + } + + // Call non-LLM tools + for (const [callId, name, args] of [ + [3, "wiki_status", {}], + [4, "wiki_search", { query: "transformer" }], + [5, "wiki_graph", { format: "summary" }], + [6, "wiki_insights", {}], + [7, "wiki_lint", {}], + ] as Array<[number, string, any]>) { + const r = await sendAndWait({ + jsonrpc: "2.0", id: callId, method: "tools/call", + params: { name, arguments: args }, + }) + record(name, { id: callId, method: "tools/call", params: { name } }, r) + } + + // Call wiki_ingest with a fresh raw file + const RAW2 = path.join(RAW, "moe-deep-dive.md") + fs.writeFileSync(RAW2, "# MoE deep dive\n\nMixture of experts and routing.\n") + handle.pushChat({ match: "expert research analyst", chunks: ["## Key Concepts\n- routing\n"] }) + const today = new Date().toISOString().slice(0, 10) + const block = [ + `---FILE: wiki/sources/moe-deep-dive.md---`, + "---", + "type: source", + 'title: "Source: MoE deep dive"', + `created: ${today}`, + `updated: ${today}`, + 'sources: ["moe-deep-dive.md"]', + "tags: [ml]", + "related: [mixture-of-experts]", + "---", + "# Source: MoE deep dive", + "Re-iterates [[mixture-of-experts]] with deeper analysis.", + "---END FILE---", + ].join("\n") + handle.pushChat({ match: "wiki maintainer", chunks: block.match(/.{1,200}/gs) ?? [block] }) + + const ingestRpc = await sendAndWait({ + jsonrpc: "2.0", id: 8, method: "tools/call", + params: { name: "wiki_ingest", arguments: { source_file: RAW2 } }, + }) + record("wiki_ingest", { id: 8, method: "tools/call", params: { name: "wiki_ingest" } }, ingestRpc) + + // Call wiki_deep_research + handle.pushSearch([ + { title: "RLHF survey", url: "https://example.com/rlhf", content: "Reinforcement learning from human feedback." }, + ]) + const synthesis = "# RLHF\n\nRLHF aligns LLMs using human preference data [1]." + handle.pushChat({ match: "research assistant", chunks: synthesis.match(/.{1,40}/gs) ?? [synthesis] }) + handle.pushChat({ match: "expert research analyst", chunks: ["## Key Concepts\n- RLHF\n"] }) + const dr = [ + `---FILE: wiki/sources/research-rlhf-${today}.md---`, + "---", + "type: source", + 'title: "Source: RLHF research"', + `created: ${today}`, + `updated: ${today}`, + `sources: ["research-rlhf-${today}.md"]`, + "tags: [ml]", + "related: [rlhf]", + "---", + "# Source: RLHF research", + "RLHF research notes.", + "---END FILE---", + "", + "---FILE: wiki/concepts/rlhf.md---", + "---", + "type: concept", + "title: RLHF", + `created: ${today}`, + `updated: ${today}`, + "tags: [ml, alignment]", + "related: [transformer]", + `sources: ["research-rlhf-${today}.md"]`, + "---", + "# RLHF", + "Reinforcement learning from human feedback. See [[transformer]].", + "---END FILE---", + ].join("\n") + handle.pushChat({ match: "wiki maintainer", chunks: dr.match(/.{1,200}/gs) ?? [dr] }) + + const drRpc = await sendAndWait({ + jsonrpc: "2.0", id: 9, method: "tools/call", + params: { name: "wiki_deep_research", arguments: { topic: "RLHF" } }, + }) + record("wiki_deep_research", { id: 9, method: "tools/call", params: { name: "wiki_deep_research" } }, drRpc) + + child.kill("SIGTERM") + await new Promise((r) => setTimeout(r, 200)) + void stderrBuf + }) +} + +// ── Run everything and emit report ────────────────────────────────────────── +async function main() { + await scaffoldFixture() + await runNonLLMPhase() + // Phase 2 + 3 + await runLLMPhase() + await runMcpPhase() + + // Re-run lint after ingest to confirm orphans changed + await runCli("lint (post-ingest)", ["lint", PROJECT]) + + // Build the report + const lines: string[] = [] + lines.push("# llm-wiki Skill + MCP — End-to-End Test Report") + lines.push("") + lines.push(`> Generated: ${new Date().toISOString()}`) + lines.push(`> Fixture: \`${TMP}\``) + lines.push(`> Node: ${process.version}`) + lines.push("") + lines.push("## Methodology") + lines.push("") + lines.push("All cases run **the real built CLI / MCP server** (`dist/cli.js`, `dist/mcp-server.js`)") + lines.push("against a real on-disk wiki fixture. LLM and Tavily traffic is served by") + lines.push("`dist/test-server/fake-llm-server.js`, a real local HTTP server that speaks the") + lines.push("OpenAI-compatible Chat Completions SSE protocol and the Tavily search REST") + lines.push("protocol — no code-level mocks. The skill code (`llm-client.ts`, `web-search.ts`)") + lines.push("runs unmodified and exercises real `fetch` / SSE parsing / JSON decoding.") + lines.push("") + const passed = results.filter((r) => r.ok).length + const failed = results.length - passed + lines.push(`## Summary: ${passed}/${results.length} passed${failed ? `, ${failed} failed` : ""}`) + lines.push("") + lines.push("| # | Case | Exit | Status | Notes |") + lines.push("|---|------|------|--------|-------|") + results.forEach((r, i) => { + lines.push(`| ${i + 1} | ${r.name} | ${r.exitCode} | ${r.ok ? "✅ pass" : "❌ fail"} | ${r.notes ?? ""} |`) + }) + lines.push("") + lines.push("## Per-case detail") + lines.push("") + for (const r of results) { + lines.push(`### ${r.name}`) + lines.push("") + lines.push("```") + lines.push(`$ ${r.command}`) + lines.push("```") + if (r.notes) { lines.push(""); lines.push(`**Notes**: ${r.notes}`); lines.push("") } + lines.push("**stdout (first 60 lines):**") + lines.push("```") + lines.push(r.stdout.split("\n").slice(0, 60).join("\n")) + lines.push("```") + if (r.stderr.trim()) { + lines.push("**stderr (first 30 lines):**") + lines.push("```") + lines.push(r.stderr.split("\n").slice(0, 30).join("\n")) + lines.push("```") + } + lines.push("") + } + // Final on-disk wiki snapshot + lines.push("## Final wiki snapshot (file tree)") + lines.push("") + lines.push("```") + function walk(dir: string, prefix = ""): string[] { + const out: string[] = [] + for (const e of fs.readdirSync(dir, { withFileTypes: true })) { + if (e.name.startsWith(".")) continue + const full = path.join(dir, e.name) + out.push(`${prefix}${e.name}${e.isDirectory() ? "/" : ""}`) + if (e.isDirectory()) out.push(...walk(full, prefix + " ")) + } + return out + } + lines.push(walk(PROJECT).join("\n")) + lines.push("```") + lines.push("") + + const reportPath = path.join(SKILL_ROOT, "docs", "test-report.md") + fs.mkdirSync(path.dirname(reportPath), { recursive: true }) + fs.writeFileSync(reportPath, lines.join("\n")) + console.log(`\n[e2e] report written: ${reportPath}`) + console.log(`[e2e] ${passed}/${results.length} passed`) + if (failed > 0) process.exitCode = 1 +} + +main().catch((err) => { console.error(err); process.exit(1) }) diff --git a/skill/src/test-server/fake-llm-server.ts b/skill/src/test-server/fake-llm-server.ts new file mode 100644 index 00000000..7fb50ffe --- /dev/null +++ b/skill/src/test-server/fake-llm-server.ts @@ -0,0 +1,167 @@ +/** + * fake-llm-server.ts — A real HTTP server that speaks the OpenAI-compatible + * Chat Completions SSE protocol AND the Tavily search REST protocol. + * + * This is NOT a code-level mock. The skill code (llm-client.ts, web-search.ts) + * runs unmodified and goes through real fetch / real SSE parsing / real JSON + * decoding. Only the upstream provider is replaced with a deterministic local + * server so tests are reproducible and don't burn API credits. + * + * Routes: + * POST /v1/chat/completions → OpenAI-style streaming SSE + * POST /search → Tavily search results + * + * Programming model: the server picks its response based on a sequential + * "script" you push via PUSH_SCRIPT, or falls back to a default echo. This + * lets a test pre-arm Stage 1 (analysis) and Stage 2 (generation) responses + * for the two-stage ingest pipeline. + * + * Run standalone: + * node dist/test-server/fake-llm-server.js [port] + */ +import * as http from "http" + +export interface ScriptedResponse { + /** Match against the user/system message text (substring). Empty = match any. */ + match?: string + /** SSE chunks to stream (each becomes one delta token). */ + chunks: string[] +} + +export interface FakeServerHandle { + port: number + baseUrl: string + pushChat: (resp: ScriptedResponse) => void + pushSearch: (results: { title: string; url: string; content: string; score?: number }[]) => void + callCount: () => { chat: number; search: number } + close: () => Promise +} + +interface InternalState { + chatScript: ScriptedResponse[] + searchScript: { title: string; url: string; content: string; score?: number }[][] + chatCalls: number + searchCalls: number +} + +function sseEncode(content: string): string { + const payload = { + choices: [{ delta: { content }, index: 0, finish_reason: null }], + } + return `data: ${JSON.stringify(payload)}\n\n` +} + +function sseDone(): string { + const payload = { + choices: [{ delta: {}, index: 0, finish_reason: "stop" }], + } + return `data: ${JSON.stringify(payload)}\n\ndata: [DONE]\n\n` +} + +async function readBody(req: http.IncomingMessage): Promise { + return new Promise((resolve, reject) => { + const chunks: Buffer[] = [] + req.on("data", (c) => chunks.push(c)) + req.on("end", () => resolve(Buffer.concat(chunks).toString("utf-8"))) + req.on("error", reject) + }) +} + +export function startFakeServer(port = 0): Promise { + const state: InternalState = { + chatScript: [], + searchScript: [], + chatCalls: 0, + searchCalls: 0, + } + + const server = http.createServer(async (req, res) => { + try { + const url = req.url ?? "" + + if (req.method === "POST" && url.startsWith("/v1/chat/completions")) { + state.chatCalls++ + const bodyText = await readBody(req) + let parsed: any = {} + try { parsed = JSON.parse(bodyText) } catch { /* ignore */ } + const lastMsg = (parsed.messages ?? []).map((m: any) => m.content ?? "").join("\n") + + // Pick the first script entry that matches; fall back to the first + // unmatched ("match" undefined) entry; else echo. + let chosen: ScriptedResponse | undefined + for (let i = 0; i < state.chatScript.length; i++) { + const s = state.chatScript[i] + if (!s.match || lastMsg.includes(s.match)) { + chosen = s + state.chatScript.splice(i, 1) + break + } + } + if (!chosen) { + chosen = { chunks: [`echo: ${lastMsg.slice(0, 80)}`] } + } + + res.writeHead(200, { + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + Connection: "keep-alive", + }) + for (const chunk of chosen.chunks) { + res.write(sseEncode(chunk)) + // tiny delay so SSE parser exercises buffering + await new Promise((r) => setTimeout(r, 5)) + } + res.write(sseDone()) + res.end() + return + } + + if (req.method === "POST" && url.startsWith("/search")) { + state.searchCalls++ + await readBody(req) + const results = state.searchScript.shift() ?? [] + res.writeHead(200, { "Content-Type": "application/json" }) + res.end(JSON.stringify({ results: results.map((r) => ({ score: 0.5, ...r })) })) + return + } + + res.writeHead(404, { "Content-Type": "text/plain" }) + res.end("not found") + } catch (err) { + res.writeHead(500, { "Content-Type": "text/plain" }) + res.end(String(err)) + } + }) + + return new Promise((resolve, reject) => { + server.once("error", reject) + server.listen(port, "127.0.0.1", () => { + const addr = server.address() + const actualPort = typeof addr === "object" && addr ? addr.port : port + resolve({ + port: actualPort, + baseUrl: `http://127.0.0.1:${actualPort}`, + pushChat: (resp) => state.chatScript.push(resp), + pushSearch: (results) => state.searchScript.push(results), + callCount: () => ({ chat: state.chatCalls, search: state.searchCalls }), + close: () => new Promise((r) => server.close(() => r())), + }) + }) + }) +} + +// CLI entry: run standalone for ad-hoc testing. +if (require.main === module) { + const port = parseInt(process.argv[2] ?? "0", 10) || 0 + startFakeServer(port).then((h) => { + // Pre-arm a default chat response so curl works + h.pushChat({ chunks: ["Hello", " from", " fake", " LLM"] }) + console.log(`Fake LLM server listening on ${h.baseUrl}`) + console.log(`Endpoints:`) + console.log(` POST ${h.baseUrl}/v1/chat/completions`) + console.log(` POST ${h.baseUrl}/search`) + }).catch((err) => { + console.error("Failed to start fake server:", err) + process.exit(1) + }) +} From 2c334d36d1555e96205d955af72f382533c4c803 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 2 May 2026 10:39:00 +0000 Subject: [PATCH 5/6] Docs (architecture, per-host usage, SKILL.md, HERMES.md) + CodeQL fixes Agent-Logs-Url: https://github.com/toughhou/llm_wiki/sessions/63c92ae1-4eba-4083-9087-deb3a07b750c Co-authored-by: toughhou <7472236+toughhou@users.noreply.github.com> --- HERMES.md | 90 ++++-- SKILL.md | 378 +++++++---------------- skill/docs/skill-mcp-progress.md | 307 ++++++------------ skill/package.json | 3 +- skill/src/lib/deep-research.ts | 2 +- skill/src/test-server/e2e.ts | 2 +- skill/src/test-server/fake-llm-server.ts | 4 +- 7 files changed, 268 insertions(+), 518 deletions(-) diff --git a/HERMES.md b/HERMES.md index 714d30f8..050a3aa1 100644 --- a/HERMES.md +++ b/HERMES.md @@ -1,40 +1,78 @@ -# llm-wiki-nashsu — Hermes Skill 入口 +# llm-wiki — Hermes Skill Entry -> **适配状态**：⚠️ 部分适配（需完成 GUI→CLI 工程改造后方可完整使用） -> **当前可用**：graph、insights、search 命令（无需 LLM） -> **待完成**：ingest、deep-research 命令（需替换 Tauri IPC → Node.js fs） +> **Status: ✅ Complete** — all CLI commands and MCP tools are +> implemented and validated end-to-end (see +> [`skill/docs/test-report.md`](skill/docs/test-report.md)). -## 触发条件 +## What this skill does -加载本技能当用户明确提到： -- "图谱分析"、"知识图谱"、"图谱洞察" -- "深度研究" -- "知识缺口"、"惊人连接" +Build and maintain a structured knowledge base ("wiki") from raw +documents. Every command operates on a single project root that +contains a `wiki/` subdirectory. See [`SKILL.md`](SKILL.md) for the +full command reference. -## 与 llm-wiki-skill 的关系 +## Trigger conditions -本技能**补充** llm-wiki-skill，提供更高级的图谱分析能力： -- llm-wiki-skill：负责日常 ingest、Hermes 调度、中文内容源 -- llm-wiki-nashsu：负责高级图谱分析、深度研究 +Load this skill when the user mentions: -## 主要工作流 +- "知识库" / "wiki" +- "知识图谱" / "graph analysis" / "knowledge graph" +- "深度研究" / "deep research" +- "知识缺口" / "knowledge gap" +- "惊人连接" / "surprising connection" +- ingest / search / lint operations against an existing wiki -详见 `SKILL.md`。 +## Two integration modes -## 安装路径 +### Mode A — CLI shell-out (the original Hermes path) ```bash -# Hermes 安装 -bash install.sh --platform hermes +node skill/dist/cli.js [args] +``` + +All eight commands (`status`, `search`, `graph`, `insights`, `lint`, +`init`, `ingest`, `deep-research`) follow this pattern. JSON output is +produced where appropriate so downstream agents can parse it. + +### Mode B — Hermes MCP client + +Recent Hermes versions support MCP. Register the server in your +Hermes MCP config: + +```yaml +servers: + llm-wiki: + command: node + args: ["/abs/path/to/llm_wiki/skill/dist/mcp-server.js"] + env: + WIKI_PATH: /Users/me/wiki + OPENAI_API_KEY: sk-... + LLM_MODEL: gpt-4o-mini + TAVILY_API_KEY: tvly-... +``` + +The same seven `wiki_*` tools appear as native Hermes tool calls. -# 直接使用 -node skill/cli.js graph -node skill/cli.js insights -node skill/cli.js search +Detailed walk-through: [`skill/docs/usage-hermes.md`](skill/docs/usage-hermes.md). + +## Install + +```bash +cd skill +npm install +npm run build +``` + +Or use the existing repo installer: + +```bash +bash install.sh --platform hermes ``` -## 注意事项 +## Requirements -- Node.js >= 20 运行时必须可用 -- 中文素材源（微信/知乎/小红书）请使用 llm-wiki-skill -- ingest 功能目前需要完成 Tauri IPC 替换工程（约 10-13 人日） +- Node.js ≥ 20 +- `OPENAI_API_KEY` (or `LLM_API_KEY` + `LLM_BASE_URL`) for `ingest` + and `deep-research` +- `TAVILY_API_KEY` for `deep-research` +- All other commands work without any LLM credentials diff --git a/SKILL.md b/SKILL.md index 267bd302..5805e3f6 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,298 +1,136 @@ --- -name: llm-wiki-nashsu -version: 0.4.6-skill -author: nashsu (GUI→Skill 适配: bid-sys team) +name: llm-wiki +version: 0.4.6-skill.1 license: MIT description: | - 基于 nashsu/llm_wiki 后端逻辑提取的知识库技能（无 GUI）。 - 核心算法包括：4 信号图谱相关度模型、Louvain 社区检测、图谱洞察（惊人连接+知识缺口）、 - RRF 混合搜索（BM25+向量）、深度研究（网络搜索+自动消化）、异步审核队列。 - 触发条件：用户明确提到知识库、wiki、图谱分析、深度研究，或要求对已初始化的知识库执行 - 消化、搜索、健康检查等操作。 + Backend port of nashsu/llm_wiki (knowledge-base builder + maintainer) + delivered as a single Node.js library exposed through three entry + points: a CLI, an MCP stdio server, and this skill manifest. + + Trigger conditions: the user mentions a "knowledge base", "wiki", + "knowledge graph", "graph analysis", "deep research", "ingest a + source into the wiki", "知识库", "知识图谱", "深度研究", or asks + to operate on an already-initialized wiki directory (search, + health check, insights, etc.). metadata: - hermes: - tags: - - knowledge-base - - wiki - - graph-analysis - - deep-research - - semantic-search origin: nashsu/llm_wiki (GUI stripped, backend extracted) runtime: node >= 20 - adapted_from: https://github.com/nashsu/llm_wiki ---- - -# llm-wiki-nashsu — 高级知识库后端技能 - -> 从 nashsu/llm_wiki 提取的后端逻辑，去除 Tauri GUI 后适配为 Hermes Skill。 -> 与 llm-wiki-skill 相比，本技能具有**显著更强的图谱分析能力**，但需要 Node.js 运行时。 - -## 核心差异化能力 - -| 能力 | 本技能 | llm-wiki-skill | -|------|-------|----------------| -| **图谱相关度** | 4 信号模型（直接链接×3 + 来源重叠×4 + Adamic-Adar×1.5 + 类型亲和×1）| 3 信号模型（共引强度 + 来源重叠 + 类型亲和度）| -| **社区检测** | Louvain 算法 + 凝聚度评分 | Louvain 算法（graph-analysis.js）| -| **图谱洞察** | 惊人连接 + 知识缺口 + 桥节点检测 | 惊人连接 + 桥节点 + 孤立节点 + 稀疏社区（大图自动降级）| -| **搜索** | RRF 混合（BM25 + 向量） | Grep + 别名展开 + 段落上限 | -| **深度研究** | 网络搜索→LLM 综合→自动消化 | 无 | -| **审核队列** | 异步异步 sweep-reviews 系统 | 无 | -| **图像处理** | 视觉 API 图像标注管线 | 无 | -| **数字山水可视化** | 无（sigma.js 通用图谱）| ✅ 东方编辑部 × 数字山水风交互式 HTML | -| **置信度标注** | 无 | ✅ EXTRACTED / INFERRED / AMBIGUOUS / UNVERIFIED | -| **SessionStart hook** | 无 | ✅ 会话自动注入 wiki 上下文 | - ---- - -## Script Directory - -Scripts located in `skill/` subdirectory relative to this SKILL.md. - -**Path Resolution**: -1. `SKILL_DIR` = this SKILL.md's directory -2. Script path = `${SKILL_DIR}/skill/` - ---- - -## 依赖要求 - -``` -node >= 20 -npm >= 9 -``` - -**可选依赖（启用向量搜索）**： -- 配置 `EMBEDDING_API_BASE` 和 `EMBEDDING_MODEL` 环境变量（OpenAI 兼容端点） - ---- - -## 工作流命令 - -### 1. init — 初始化知识库 - -```bash -node ${SKILL_DIR}/skill/cli.js init [topic] [lang] -``` - -**参数**： -- `wiki_root`：wiki 工作目录（绝对路径） -- `topic`：知识库主题（可选，默认 "My Knowledge Base"） -- `lang`：语言（可选，`zh`|`en`，默认 `en`） - -**产物**： -``` -/ -├── wiki/ -│ ├── entities/ -│ ├── concepts/ -│ ├── sources/ -│ ├── queries/ -│ └── index.md -├── raw/ -└── .wiki-config.json -``` - ---- - -### 2. ingest — 消化素材 - -```bash -node ${SKILL_DIR}/skill/cli.js ingest [--llm-api-key=KEY] -``` - -**参数**： -- `wiki_root`：wiki 工作目录 -- `file_path`：待消化的文件路径（支持 .md / .txt；PDF/DOCX 需先转为文本） -- `--llm-api-key`：LLM API Key（也可通过 `OPENAI_API_KEY` 环境变量传入） - -**处理流程**（源自 `ingest.ts`）： -1. **Step 1**：LLM 分析素材 → 生成结构化 JSON（实体、概念、关系） -2. **Step 2**：基于 JSON 生成 wiki 页面： - - `wiki/sources/{slug}.md` — 素材摘要页（含 `sources: []` frontmatter） - - `wiki/entities/{name}.md` — 实体页（仅限新实体） - - `wiki/concepts/{name}.md` — 概念页（仅限新概念） -3. **自动消化**：生成的页面自动进入 wiki 图谱（下次 graph 命令时生效） -4. **审核标记**：LLM 自动标记需人工判断的条目（`review: true` frontmatter） - -**产物示例**： -```json -{ - "status": "success", - "pages": [ - "wiki/sources/2026-04-30-企业资质证书.md", - "wiki/entities/市政公用工程施工总承包壹级.md" - ], - "reviews_pending": 1 -} -``` - ---- - -### 3. batch-ingest — 批量消化 - -```bash -node ${SKILL_DIR}/skill/cli.js batch-ingest -``` - -按目录递归处理所有 `.md`/`.txt` 文件，保留目录结构作为分类上下文。 -失败不阻塞后续文件（标记失败项，继续）。 - ---- - -### 4. search — 智能搜索 - -```bash -node ${SKILL_DIR}/skill/cli.js search [--limit=20] -``` - -**算法**（源自 `search.ts`，18KB）： -1. **BM25 词法搜索**：中文 CJK bigram 分词 + 停用词过滤 + 精确词组匹配加权 -2. **向量语义搜索**（可选）：LanceDB ANN 检索（需配置嵌入端点） -3. **RRF 融合**：倒数秩融合（K=60），避免量纲不一致 - -**输出**：JSON 格式检索结果（path, title, snippet, score, images） - ---- - -### 5. graph — 构建知识图谱 - -```bash -node ${SKILL_DIR}/skill/cli.js graph [--output=graph-data.json] -``` - -**算法**（源自 `wiki-graph.ts` + `graph-relevance.ts`）： -1. **读取所有 wiki 页面**，提取标题、类型、wikilink -2. **4 信号相关度计算**（每条边）： - - 直接链接（weight 3.0） - - 来源重叠（weight 4.0，基于 `sources: []` frontmatter） - - Adamic-Adar 共同邻居（weight 1.5） - - 类型亲和度（weight 1.0） -3. **Louvain 社区检测**（graphology-communities-louvain）： - - 自动聚类，计算每个社区凝聚度（实际边/可能边） - - 低凝聚度社区（<0.15）标记为警告 -4. **输出**：`graph-data.json`（nodes + edges + communities） - -**输出格式**： -```json -{ - "nodes": [{ "id": "xxx", "label": "...", "type": "entity", "linkCount": 5, "community": 0 }], - "edges": [{ "source": "xxx", "target": "yyy", "weight": 7.2 }], - "communities": [{ "id": 0, "nodeCount": 12, "cohesion": 0.24, "topNodes": ["..."] }] -} -``` - ---- - -### 6. insights — 图谱洞察 - -```bash -node ${SKILL_DIR}/skill/cli.js insights -``` - -**算法**（源自 `graph-insights.ts`，193 行）： -1. **惊人连接**（Surprising Connections）： - - 跨社区边 +3，跨类型边 +2，边缘↔枢纽耦合 +2，弱连接 +1 - - 阈值 ≥3 才输出 -2. **知识缺口**（Knowledge Gaps）： - - 孤立节点（degree ≤1） - - 稀疏社区（cohesion <0.15 且 ≥3 节点） - - 桥节点（连接 ≥3 个社区） - -**输出**：Markdown 格式洞察报告 - + entry_points: + cli: skill/dist/cli.js + mcp: skill/dist/mcp-server.js + hermes: + tags: [knowledge-base, wiki, graph-analysis, deep-research, semantic-search] --- -### 7. deep-research — 深度研究 +# llm-wiki — Skill Manifest -```bash -node ${SKILL_DIR}/skill/cli.js deep-research [--queries="q1|q2|q3"] -``` - -**流程**（源自 `deep-research.ts`，244 行）： -1. **网络搜索**：多查询并行搜索（Tavily API），URL 去重合并 -2. **LLM 综合**：搜索结果 → wiki 页面（带 `[[wikilink]]` 交叉引用） -3. **保存**：`wiki/queries/research-{slug}-{date}.md` -4. **自动消化**：研究结果自动 ingest，提取实体/概念 +Three equivalent ways to use this skill, picked by the host: -**环境变量**：`TAVILY_API_KEY`（或 `SERPER_API_KEY`） +1. **MCP (recommended)** — point your AI host (Claude Desktop, Cursor, + VS Code Copilot Chat, OpenAI Codex CLI, Hermes) at + `skill/dist/mcp-server.js`. It exposes seven `wiki_*` tools. +2. **CLI** — shell out to `node skill/dist/cli.js `. +3. **Direct library** — `require('./skill/dist/lib/...')` from your + own Node.js code. ---- +All three routes hit the same backend (`skill/src/lib/`). -### 8. lint — 健康检查 +## Install ```bash -node ${SKILL_DIR}/skill/cli.js lint +cd skill +npm install +npm run build ``` -**检查项**（源自 `lint.ts`）： -- 孤立页面（无入链且无出链） -- 断链（`[[wikilink]]` 指向不存在的页面） -- 过短页面（< 100 字） -- 语言不一致（frontmatter `lang` 与内容不符） -- 重复内容（相似度过高的页面） - ---- - -### 9. sweep-reviews — 处理审核队列 +Verify: ```bash -node ${SKILL_DIR}/skill/cli.js sweep-reviews +node dist/cli.js --help +node dist/cli.js status /path/to/wiki-project ``` -**功能**（源自 `sweep-reviews.ts`，14KB）： -- 扫描所有 `review: true` 的 wiki 页面 -- 基于规则匹配 + LLM 语义判断自动解决 -- 预定义动作：Create Page / Skip（防止 LLM 幻觉任意动作） - ---- - -### 10. status — 知识库状态 +Run the full end-to-end test suite (real HTTP server, no mocks): ```bash -node ${SKILL_DIR}/skill/cli.js status +npm run test:e2e +# → writes skill/docs/test-report.md ``` -**输出**：JSON 格式统计（页面数、实体数、概念数、源数、待审核数） - ---- - -## 配置环境变量 - -| 变量 | 用途 | 示例 | -|------|------|------| -| `OPENAI_API_KEY` | LLM API Key（OpenAI/Anthropic 兼容）| `sk-...` | -| `OPENAI_API_BASE` | 自定义 LLM 端点（Ollama/代理）| `http://localhost:11434/v1` | -| `LLM_MODEL` | 模型名称 | `gpt-4o` / `claude-3-5-sonnet` | -| `EMBEDDING_API_BASE` | 嵌入端点（可选，启用向量搜索）| `http://localhost:11434/v1` | -| `EMBEDDING_MODEL` | 嵌入模型（可选）| `text-embedding-3-small` | -| `TAVILY_API_KEY` | 深度研究搜索 API（可选）| `tvly-...` | - ---- - -## 与 llm-wiki-skill 的关键互补 +## CLI commands -本技能建议**配合** llm-wiki-skill 使用而非替代： +| Command | Purpose | +|---|---| +| `init ` | Create the wiki directory layout | +| `status ` | Page count + community count | +| `search ` | BM25 (+ optional vector) search with snippets | +| `graph ` | Build knowledge graph (4-signal relevance + Louvain) | +| `insights ` | Surprising connections + knowledge gaps | +| `lint ` | Find orphans / isolated pages | +| `ingest ` | Two-stage LLM ingest of a markdown/text source | +| `deep-research ` | Web search → LLM synthesis → auto-ingest | -| 场景 | 推荐方案 | -|------|---------| -| 日常 ingest（速度优先）| llm-wiki-skill（Shell，零开销，SHA256 缓存）| -| 高精度图谱分析（Adamic-Adar） | 本技能（graph + insights 命令，4 信号模型）| -| RRF 混合搜索 | 本技能（search 命令，BM25+向量）| -| 深度研究专项 | 本技能（deep-research 命令）| -| 基础图谱分析与可视化 | llm-wiki-skill（3 信号 + Louvain + 数字山水 HTML）| -| 中文内容源（微信/知乎/小红书）| llm-wiki-skill | -| Hermes Runtime 集成 | llm-wiki-skill（已有 HERMES.md + SessionStart hook）| -| 本技能 Hermes 集成 | 参见 HERMES.md（需手动适配）| - ---- - -## 安装 +Examples: ```bash -# 安装 CLI 依赖 -cd ${SKILL_DIR}/skill -npm install - -# 验证安装 -node cli.js --version -``` +node skill/dist/cli.js status ~/notes/my-wiki +node skill/dist/cli.js search ~/notes/my-wiki "attention mechanism" +node skill/dist/cli.js insights ~/notes/my-wiki +node skill/dist/cli.js ingest ~/notes/my-wiki ~/raw/paper.md +node skill/dist/cli.js deep-research ~/notes/my-wiki "Mixture of Experts" +``` + +## MCP tools + +Exact JSON schemas are in `skill/src/mcp-server.ts`. + +| Tool | Required args | Optional args | +|---|---|---| +| `wiki_status` | — | `project_path` | +| `wiki_search` | `query` | `project_path`, `limit` | +| `wiki_graph` | — | `project_path`, `format` (`summary`\|`json`) | +| `wiki_insights` | — | `project_path`, `max_connections`, `max_gaps` | +| `wiki_lint` | — | `project_path` | +| `wiki_ingest` | `source_file` | `project_path`, `folder_context` | +| `wiki_deep_research` | `topic` | `project_path`, `search_queries`, `auto_ingest` | + +## Configuration (env vars) + +| Variable | Purpose | +|---|---| +| `OPENAI_API_KEY` / `LLM_API_KEY` | LLM credentials | +| `LLM_BASE_URL` | OpenAI-compatible endpoint (Ollama, OpenRouter, ...) | +| `LLM_MODEL` | Model name (default `gpt-4o-mini`) | +| `LLM_PROVIDER` | `openai` / `anthropic` / `ollama` / `deepseek` | +| `TAVILY_API_KEY` | Required for `deep-research` / `wiki_deep_research` | +| `WIKI_OUTPUT_LANGUAGE` | `auto` / `English` / `Chinese` / ... | +| `WIKI_PATH` | Default project path for the MCP server | +| `EMBEDDING_*` | Optional vector search (graceful no-op if unset) | +| `SKILL_VERBOSE` | Mirror activity logs to stderr | + +## Host integration guides + +- Claude (Desktop MCP + Claude Code Skill): [`skill/docs/usage-claude.md`](skill/docs/usage-claude.md) +- Cursor (MCP): [`skill/docs/usage-cursor.md`](skill/docs/usage-cursor.md) +- VS Code Copilot Chat (MCP): [`skill/docs/usage-copilot.md`](skill/docs/usage-copilot.md) +- OpenAI Codex CLI (MCP): [`skill/docs/usage-codex.md`](skill/docs/usage-codex.md) +- Hermes (Skill or MCP): [`skill/docs/usage-hermes.md`](skill/docs/usage-hermes.md) + +## Architecture & test report + +- Architecture: [`skill/docs/architecture.md`](skill/docs/architecture.md) +- E2E test report: [`skill/docs/test-report.md`](skill/docs/test-report.md) +- Status / progress: [`skill/docs/skill-mcp-progress.md`](skill/docs/skill-mcp-progress.md) + +## Out of scope + +This is a backend-only skill. Image extraction (PDF/PPTX/DOCX), +vision-LLM captioning, the async sweep-reviews queue, and the +embedded vector index are intentionally not ported — they require +the GUI desktop app or specialized binary dependencies. Use the +upstream [nashsu/llm_wiki](https://github.com/nashsu/llm_wiki) Tauri +app for those. + +REVIEW blocks emitted by the LLM during `ingest` are still parsed and +returned to the caller (the CLI prints them as JSON; the MCP tool +includes them in the reply) so a host can act on them. diff --git a/skill/docs/skill-mcp-progress.md b/skill/docs/skill-mcp-progress.md index e30a9b8c..f7d65032 100644 --- a/skill/docs/skill-mcp-progress.md +++ b/skill/docs/skill-mcp-progress.md @@ -1,248 +1,119 @@ -# llm_wiki Node.js Skill + MCP Server — 方案与进度 +# llm_wiki Node.js Skill + MCP — Status -> 文档生成日期：2026-05-02 -> 状态：**进行中** — ingest / deep-research 实现中，PR 待更新 +> **Status: ✅ Completed** +> Last updated: 2026-05-02 ---- - -## 一、背景与目标 - -### 项目来源 - -[nashsu/llm_wiki](https://github.com/nashsu/llm_wiki) 是一个基于 Tauri v2（Rust + React/TypeScript）的桌面应用，核心功能是把本地源文件（Markdown/PDF/DOCX）通过 LLM 自动整理成结构化 Wiki。 - -### 需求 - -bid-sys 项目需要其后台核心逻辑，但 **不需要 GUI（Tauri 桌面应用）**，目标是： - -1. **Node.js Skill** — 纯命令行可调用的 wiki 管理工具 -2. **MCP Server** — 将 wiki 操作暴露为 AI 可调用的工具（供 Claude Desktop / VS Code Copilot Chat 使用） -3. **贡献 MCP** — 向 nashsu/llm_wiki 提交 PR，将 MCP 服务器作为官方插件 - ---- - -## 二、架构分析 - -### nashsu/llm_wiki 技术栈 - -``` -llm_wiki/ -├── src/ # React + TypeScript 前端（GUI 层） -│ ├── lib/ # 核心业务逻辑（纯 TypeScript）⬅ 我们需要的 -│ ├── stores/ # Zustand React 状态管理 -│ └── commands/ # Tauri IPC 桥接层 -├── src-tauri/ # Rust 后端（文件 I/O、PDF 提取、系统集成） -``` +## What this is -### 两个注入点 +A backend-only port of [nashsu/llm_wiki](https://github.com/nashsu/llm_wiki) +delivered as **one** Node.js library exposed through **three** entry +points: -所有 `src/lib/*.ts` 通过两个抽象层与 Tauri 交互： +1. `dist/cli.js` — command-line interface +2. `dist/mcp-server.js` — Model Context Protocol stdio server +3. `SKILL.md` — discovery / trigger manifest for skill-aware hosts -| 原始导入 | 功能 | Node.js 替代 | -|---------|------|-------------| -| `@/commands/fs` | 文件读写/列举 | `shims/fs-node.ts` | -| `@/stores/*` | 应用状态（LLM 配置等）| `shims/stores-node.ts` | - -Tauri HTTP 代理（`tauri-fetch.ts`）已内置 `isNodeEnv` 检测，直接降级到 `globalThis.fetch`，无需额外适配。 - ---- - -## 三、实现方案 - -### 方案选择：自包含副本（Self-Contained Copy） - -将 `src/lib/*.ts` 复制并修补到 `skill/src/lib/`，修改所有 `@/` 路径别名为相对路径，完全独立于原始项目结构。 - -**优点：** 不依赖 tsconfig 路径别名，构建简单，易于移植 -**缺点：** 需手工同步上游更新 - -### 目录结构 - -``` -skill/ -├── src/ -│ ├── cli.ts # CLI 入口（8 个命令） -│ ├── mcp-server.ts # MCP 服务器（7 个工具） -│ ├── lib/ # 从 nashsu/llm_wiki 移植的核心库 -│ │ ├── graph-relevance.ts -│ │ ├── wiki-graph.ts -│ │ ├── graph-insights.ts -│ │ ├── search.ts -│ │ ├── path-utils.ts -│ │ ├── llm-client.ts # LLM SSE 流式调用 -│ │ ├── detect-language.ts # Unicode 脚本语言检测 -│ │ ├── output-language.ts # 输出语言指令构建 -│ │ ├── frontmatter.ts # YAML frontmatter 解析器 -│ │ ├── sources-merge.ts # Frontmatter 数组字段合并 -│ │ ├── page-merge.ts # Wiki 页面内容合并（LLM） -│ │ ├── ingest-sanitize.ts # LLM 输出清理 -│ │ ├── ingest-cache.ts # SHA256 内容缓存 -│ │ ├── project-mutex.ts # 按项目路径的异步互斥锁 -│ │ ├── ingest.ts # 核心 ingest 流水线（待完成） -│ │ └── web-search.ts # Tavily 搜索 API -│ ├── shims/ # Tauri → Node.js 适配层 -│ │ ├── fs-node.ts -│ │ ├── stores-node.ts -│ │ └── embedding-stub.ts -│ └── types/ -│ └── wiki.ts -├── package.json -└── tsconfig.json - -mcp-server/ # 独立 MCP 包（用于 PR 提交） -├── src/index.ts -├── package.json -└── README.md -``` +See [`architecture.md`](./architecture.md) for the full design. --- -## 四、功能清单 - -### CLI 命令 - -| 命令 | 状态 | 说明 | -|------|------|------| -| `status` | ✅ | 统计 wiki 页面数量/类型 | -| `search ` | ✅ | BM25+RRF 全文搜索 | -| `graph` | ✅ | 构建并输出知识图谱（Louvain 社区检测）| -| `insights` | ✅ | 发现意外关联 + 知识盲点 | -| `lint` | ✅ | 检测孤立页面/断链/缺失字段 | -| `init` | ✅ | 初始化 wiki 目录结构 | -| `ingest ` | 🔄 | LLM 自动摄入源文件 → wiki 页面 | -| `deep-research ` | 🔄 | 网络搜索 → LLM 综合 → 自动摄入 | - -### MCP 工具 - -| 工具 | 状态 | 说明 | -|------|------|------| -| `wiki_status` | ✅ | 获取 wiki 统计 | -| `wiki_search` | ✅ | 搜索 wiki 页面 | -| `wiki_graph` | ✅ | 获取知识图谱 | -| `wiki_insights` | ✅ | 获取 AI 见解 | -| `wiki_lint` | ✅ | 检查 wiki 健康度 | -| `wiki_ingest` | 🔄 | 摄入源文件 | -| `wiki_deep_research` | 🔄 | 深度研究 | +## Capability matrix + +### CLI commands (`dist/cli.js`) + +| Command | Purpose | Status | +|---|---|---| +| `init ` | Create wiki dir layout | ✅ | +| `status ` | Page + community counts | ✅ | +| `search ` | BM25(+RRF) keyword search | ✅ | +| `graph ` | Build knowledge graph (Louvain) | ✅ | +| `insights ` | Surprising connections + gaps | ✅ | +| `lint ` | Orphan / broken-link check | ✅ | +| `ingest ` | Two-stage LLM ingest | ✅ | +| `deep-research ` | Web-search → synth → ingest | ✅ | + +### MCP tools (`dist/mcp-server.js`) + +| Tool | Status | +|---|---| +| `wiki_status` | ✅ | +| `wiki_search` | ✅ | +| `wiki_graph` | ✅ | +| `wiki_insights` | ✅ | +| `wiki_lint` | ✅ | +| `wiki_ingest` | ✅ | +| `wiki_deep_research` | ✅ | --- -## 五、环境变量配置 - -```bash -# LLM 配置（ingest / deep-research 必需） -export LLM_PROVIDER=openai # openai | anthropic | ollama | deepseek -export OPENAI_API_KEY=sk-... -export LLM_MODEL=gpt-4o-mini -export LLM_BASE_URL= # 自定义端点（可选） +## Validation -# 网络搜索（deep-research 必需） -export TAVILY_API_KEY=tvly-... +Real end-to-end tests run on every `npm run test:e2e`. They drive +`dist/cli.js` and `dist/mcp-server.js` as subprocesses, against a real +on-disk wiki fixture, with LLM/Tavily traffic served by a real local +HTTP server (no code-level mocks). -# 输出语言（可选，默认 auto 自动检测） -export WIKI_OUTPUT_LANGUAGE=auto # auto | English | Chinese | Japanese | ... +Latest run: **19/19 cases passed** — see [`test-report.md`](./test-report.md). -# 调试 -export SKILL_VERBOSE=1 # 输出详细日志到 stderr -``` +| Phase | Cases | +|---|---| +| Non-LLM CLI | init, status, search, graph, insights, lint | +| LLM CLI | ingest (cold), ingest (cache hit), deep-research | +| MCP | initialize, tools/list, all 7 tool calls | +| Regression | post-ingest lint | --- -## 六、依赖 - -```json -{ - "dependencies": { - "graphology": "^0.25.4", - "graphology-communities-louvain": "^2.0.0", - "@modelcontextprotocol/sdk": "^1.1.0", - "js-yaml": "^4.1.0" - } -} -``` +## Host integration guides ---- - -## 七、开发进度 - -### 已完成 - -- [x] 分析 nashsu/llm_wiki 架构，识别 Tauri 注入点 -- [x] 创建 `shims/fs-node.ts` — Tauri IPC → Node.js fs 适配 -- [x] 创建 `shims/stores-node.ts` — Zustand → 模块级状态，支持 env 配置 LLM -- [x] 创建 `shims/embedding-stub.ts` — 向量搜索优雅降级 -- [x] 移植并修补所有图谱库（graph-relevance, wiki-graph, graph-insights） -- [x] 移植搜索库（BM25+RRF，向量可选） -- [x] 移植 path-utils（纯工具函数） -- [x] 实现 CLI 6 个命令：status/search/graph/insights/lint/init -- [x] 实现 MCP 服务器 5 个工具 -- [x] npm install + tsc 构建通过 -- [x] 端到端测试：合成 wiki 数据验证所有命令 -- [x] Fork nashsu/llm_wiki → toughhou/llm_wiki -- [x] 移植 llm-client.ts（OpenAI 兼容 SSE 流式调用） -- [x] 移植 detect-language.ts（Unicode 脚本检测） -- [x] 移植 output-language.ts -- [x] 移植 frontmatter.ts（js-yaml 解析） -- [x] 移植 sources-merge.ts（数组字段合并） -- [x] 移植 page-merge.ts（LLM 辅助页面合并） -- [x] 移植 ingest-sanitize.ts（LLM 输出清洗） -- [x] 移植 ingest-cache.ts（SHA256 增量缓存） -- [x] 移植 project-mutex.ts（并发保护） -- [x] 移植 web-search.ts（Tavily API） -- [x] 提交 PR #117 到 nashsu/llm_wiki - -### 进行中 - -- [ ] 完成 ingest.ts — 两阶段 LLM 流水线（分析 → 生成 → 写文件） -- [ ] 完成 deep-research.ts — 网络搜索 → LLM 综合 → auto-ingest -- [ ] CLI 添加 ingest / deep-research 命令 -- [ ] MCP 服务器添加 wiki_ingest / wiki_deep_research 工具 -- [ ] 端到端测试（需真实 LLM API Key） -- [ ] 更新 PR #117 - -### 待完成 - -- [ ] sweep-reviews（批量审核 wiki 页面） -- [ ] 嵌入向量搜索（可选，需 embedding API） +- [`usage-claude.md`](./usage-claude.md) — Claude Desktop (MCP) + Claude Code (Skill) +- [`usage-cursor.md`](./usage-cursor.md) — Cursor MCP +- [`usage-copilot.md`](./usage-copilot.md) — VS Code GitHub Copilot Chat (MCP) +- [`usage-codex.md`](./usage-codex.md) — OpenAI Codex CLI (MCP) +- [`usage-hermes.md`](./usage-hermes.md) — Hermes (Skill or MCP) --- -## 八、PR 提交记录 +## Design decisions made along the way -| PR | 仓库 | 分支 | 状态 | -|----|------|------|------| -| #117 | nashsu/llm_wiki | feat/mcp-server | 开放中，待更新 | +1. **MCP-first, Skill-second, CLI as the safety net.** MCP is the only + protocol every target host supports natively today. +2. **Single source tree.** The previous duplicated `mcp-server/` + directory was deleted; everything lives in `skill/src/lib/`. +3. **Env-var configuration only.** No config files. Every host's + config syntax already supports passing env vars to a child process. +4. **No code-level mocks in tests.** A real local OpenAI/Tavily- + compatible HTTP server (`src/test-server/fake-llm-server.ts`) + replaces only the upstream provider so the skill code's `fetch` / + SSE parsing / file I/O paths all execute for real. --- -## 九、本地测试方法 - -```bash -cd skill && npm install && npm run build +## Out of scope (intentionally) -# 测试基础命令 -node dist/cli.js status /path/to/wiki-project -node dist/cli.js search "machine learning" /path/to/wiki-project -node dist/cli.js graph /path/to/wiki-project -node dist/cli.js insights /path/to/wiki-project -node dist/cli.js lint /path/to/wiki-project +- Image extraction (PDF/PPTX/DOCX) — needs Rust pdfium binding +- Vision-LLM caption pipeline — needs multimodal endpoint +- Sweep-reviews queue — depends on long-lived UI store +- LanceDB vector index — kept as a graceful no-op via + `shims/embedding-stub.ts` -# 测试 ingest（需 LLM API Key） -export OPENAI_API_KEY=sk-xxx -node dist/cli.js ingest /path/to/source.md /path/to/wiki-project - -# 测试 deep-research（需 LLM + Tavily） -export TAVILY_API_KEY=tvly-xxx -node dist/cli.js deep-research "transformer architecture" /path/to/wiki-project - -# 启动 MCP 服务器 -node dist/mcp-server.js -``` +REVIEW blocks emitted by the LLM are still **parsed** and surfaced to +the caller (CLI prints them as JSON, MCP renders them in the tool +reply), so a host can act on them. --- -## 十、相关资源 - -- 上游仓库：https://github.com/nashsu/llm_wiki -- 本仓库（fork）：https://github.com/toughhou/llm_wiki -- PR #117：https://github.com/nashsu/llm_wiki/pull/117 -- bid-sys 项目：https://github.com/toughhou/bid-sys +## Files reference + +| File | Role | +|---|---| +| `skill/src/lib/ingest.ts` | Two-stage LLM pipeline (analysis → generation → write) | +| `skill/src/lib/deep-research.ts` | Multi-query search → LLM synthesis → auto-ingest | +| `skill/src/lib/llm-client.ts` | OpenAI-compatible SSE streaming | +| `skill/src/lib/web-search.ts` | Tavily client (configurable base URL) | +| `skill/src/lib/page-merge.ts` | LLM-assisted merge of conflicting page versions | +| `skill/src/lib/ingest-cache.ts` | SHA256-keyed incremental cache | +| `skill/src/cli.ts` | CLI entry point | +| `skill/src/mcp-server.ts` | MCP entry point | +| `skill/src/test-server/fake-llm-server.ts` | Real local OpenAI/Tavily HTTP server | +| `skill/src/test-server/e2e.ts` | End-to-end runner that produces `test-report.md` | diff --git a/skill/package.json b/skill/package.json index dd07dde1..09e91a9f 100644 --- a/skill/package.json +++ b/skill/package.json @@ -11,7 +11,8 @@ "build": "tsc -p tsconfig.json", "typecheck": "tsc -p tsconfig.json --noEmit", "start": "node dist/cli.js", - "mcp": "node dist/mcp-server.js" + "mcp": "node dist/mcp-server.js", + "test:e2e": "npm run build && node dist/test-server/e2e.js" }, "dependencies": { "@modelcontextprotocol/sdk": "^1.0.0", diff --git a/skill/src/lib/deep-research.ts b/skill/src/lib/deep-research.ts index 0657037b..f1c2f431 100644 --- a/skill/src/lib/deep-research.ts +++ b/skill/src/lib/deep-research.ts @@ -169,7 +169,7 @@ export async function deepResearch( const pageContent = [ "---", "type: query", - `title: "Research: ${topic.replace(/"/g, '\\"')}"`, + `title: "Research: ${topic.replace(/\\/g, "\\\\").replace(/"/g, '\\"')}"`, `created: ${date}`, "origin: deep-research", "tags: [research]", diff --git a/skill/src/test-server/e2e.ts b/skill/src/test-server/e2e.ts index 11361884..29f93f2c 100644 --- a/skill/src/test-server/e2e.ts +++ b/skill/src/test-server/e2e.ts @@ -10,7 +10,7 @@ import * as fs from "fs" import * as path from "path" import * as os from "os" -import { spawn, type ChildProcessWithoutNullStreams } from "child_process" +import { spawn } from "child_process" import { startFakeServer, type FakeServerHandle } from "../test-server/fake-llm-server" const SKILL_ROOT = path.resolve(__dirname, "../..") diff --git a/skill/src/test-server/fake-llm-server.ts b/skill/src/test-server/fake-llm-server.ts index 7fb50ffe..69440b76 100644 --- a/skill/src/test-server/fake-llm-server.ts +++ b/skill/src/test-server/fake-llm-server.ts @@ -128,8 +128,10 @@ export function startFakeServer(port = 0): Promise { res.writeHead(404, { "Content-Type": "text/plain" }) res.end("not found") } catch (err) { + // Test-only server — log to stderr for debugging, return generic body. + console.error("[fake-llm-server] handler error:", err) res.writeHead(500, { "Content-Type": "text/plain" }) - res.end(String(err)) + res.end("internal server error") } }) From 2e7ed441a89405e67d65de6fc900f061e585319c Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 2 May 2026 10:39:43 +0000 Subject: [PATCH 6/6] Allow skill/docs in git (was caught by global docs/ ignore) Agent-Logs-Url: https://github.com/toughhou/llm_wiki/sessions/63c92ae1-4eba-4083-9087-deb3a07b750c Co-authored-by: toughhou <7472236+toughhou@users.noreply.github.com> --- .gitignore | 3 + skill/docs/architecture.md | 147 ++++++++++++ skill/docs/test-report.md | 459 ++++++++++++++++++++++++++++++++++++ skill/docs/usage-claude.md | 131 ++++++++++ skill/docs/usage-codex.md | 62 +++++ skill/docs/usage-copilot.md | 71 ++++++ skill/docs/usage-cursor.md | 76 ++++++ skill/docs/usage-hermes.md | 85 +++++++ 8 files changed, 1034 insertions(+) create mode 100644 skill/docs/architecture.md create mode 100644 skill/docs/test-report.md create mode 100644 skill/docs/usage-claude.md create mode 100644 skill/docs/usage-codex.md create mode 100644 skill/docs/usage-copilot.md create mode 100644 skill/docs/usage-cursor.md create mode 100644 skill/docs/usage-hermes.md diff --git a/.gitignore b/.gitignore index c4700ec6..839b0379 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,9 @@ src-tauri/gen/ # Internal docs (not shipped) docs/ +# But: the skill subproject's docs ARE part of the public skill surface. +!skill/docs/ +!skill/docs/** # Benchmark and test data (local only) tests/ diff --git a/skill/docs/architecture.md b/skill/docs/architecture.md new file mode 100644 index 00000000..f7a996c5 --- /dev/null +++ b/skill/docs/architecture.md @@ -0,0 +1,147 @@ +# Architecture — llm-wiki Skill + MCP + +## Goal + +A single backend implementation of the `nashsu/llm_wiki` knowledge-base +algorithms, reachable from **any** AI host via three equivalent entry points: + +``` + ┌──────────────────────┐ + │ skill/src/lib/* │ ← single source of truth + │ (graph, search, │ + │ ingest, deep- │ + │ research, ...) │ + └──────────┬───────────┘ + │ + ┌────────────────────┼────────────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌──────────┐ ┌─────────────┐ ┌─────────────┐ + │ cli.ts │ │ mcp-server │ │ SKILL.md │ + │ │ │ .ts │ │ / HERMES │ + │ shell │ │ stdio JSON- │ │ loader │ + │ scripts, │ │ RPC (Claude │ │ docs (any │ + │ cron, CI │ │ / Cursor / │ │ host's │ + │ │ │ Copilot / │ │ skill mech-│ + │ │ │ Codex / │ │ anism that │ + │ │ │ Hermes ...) │ │ shells out │ + └──────────┘ └─────────────┘ └─────────────┘ +``` + +## Why MCP is the primary entry point + +MCP (Model Context Protocol) is currently the only protocol supported +**natively** by the major hosts users care about — Claude Desktop, +Cursor, VS Code Copilot Chat, OpenAI Codex CLI, Continue, Hermes +runtime. A single MCP stdio server reaches them all. + +A skill manifest (Claude Skills `SKILL.md`, Hermes Skill, etc.) is just +a discovery/triggering hint that ultimately needs to *call something* — +either an MCP server or a CLI. Building MCP first means every skill +manifest can wrap the same backend without forking it. + +CLI is kept as a third entry because: + +- Not every workflow involves an LLM host (cron, CI, makefiles). +- It's the lowest-common-denominator interop format. +- It's how Hermes-style skills shell out today. + +## Why a single shared `lib/` + +The previous repo state had a duplicated `mcp-server/` tree containing a +*subset* of the `skill/lib/` files — the LLM-related libraries +(`llm-client`, `page-merge`, `ingest-cache`, etc.) had been ported to +`skill/lib/` only. That's a code-drift bomb: the next bug fix in +`graph-insights.ts` would have to be remembered twice. We collapsed it +to a single `skill/src/lib/` tree, then added a unified `dist/` with +both `cli.js` and `mcp-server.js` bin entries. + +## Directory layout + +``` +skill/ +├── src/ +│ ├── lib/ # core algorithms (Tauri-free) +│ │ ├── wiki-graph.ts +│ │ ├── graph-relevance.ts +│ │ ├── graph-insights.ts +│ │ ├── search.ts (BM25 + RRF) +│ │ ├── path-utils.ts +│ │ ├── frontmatter.ts (js-yaml) +│ │ ├── sources-merge.ts +│ │ ├── page-merge.ts (LLM-assisted page merge) +│ │ ├── ingest-cache.ts (SHA256 incremental cache) +│ │ ├── ingest-sanitize.ts +│ │ ├── ingest.ts ★ two-stage LLM pipeline +│ │ ├── deep-research.ts ★ web-search → synthesis → ingest +│ │ ├── llm-client.ts (OpenAI-compatible SSE streaming) +│ │ ├── web-search.ts (Tavily; configurable base URL) +│ │ ├── detect-language.ts +│ │ ├── output-language.ts +│ │ └── project-mutex.ts +│ ├── shims/ # Tauri → Node.js adapters +│ │ ├── fs-node.ts (read/write/list) +│ │ ├── stores-node.ts (env-driven config + activity logger) +│ │ └── embedding-stub.ts (no-op fallback) +│ ├── types/wiki.ts +│ ├── cli.ts # entry point 1: command line +│ ├── mcp-server.ts # entry point 2: MCP stdio server +│ └── test-server/ +│ ├── fake-llm-server.ts # real local OpenAI-compatible HTTP server +│ └── e2e.ts # end-to-end real test runner +├── dist/ # tsc output (committed? no — built per install) +└── docs/ # this file + usage-* + test-report.md +``` + +## Configuration model + +All runtime configuration is **env-var driven**. There are no config +files. This keeps the same code reachable from every host without +per-host config syntax. + +| Variable | Purpose | +|---|---| +| `OPENAI_API_KEY` / `LLM_API_KEY` | LLM credentials | +| `LLM_BASE_URL` | OpenAI-compatible endpoint (Ollama, OpenRouter, ...) | +| `LLM_MODEL` | Model name (default `gpt-4o-mini`) | +| `LLM_PROVIDER` | `openai` / `anthropic` / `ollama` / `deepseek` | +| `TAVILY_API_KEY` | Tavily search (deep-research only) | +| `TAVILY_BASE_URL` | Override Tavily endpoint (test-only) | +| `EMBEDDING_*` | Optional vector search (gracefully degrades to BM25 only) | +| `WIKI_OUTPUT_LANGUAGE` | `auto` / `English` / `Chinese` / ... | +| `WIKI_PATH` | Default project path for MCP server | +| `SKILL_VERBOSE` | Mirror activity log to stderr | + +## What is intentionally not ported + +| Upstream feature | Reason | +|---|---| +| Image extraction (PDF/PPTX/DOCX) | Needs Rust pdfium binding | +| Vision-LLM caption pipeline | Needs multimodal endpoint config | +| Embedding generation | Optional; has a stub for graceful no-op | +| Sweep-reviews queue | Depends on long-lived Zustand React store | +| Chrome Web Clipper | Browser extension surface, not a skill | + +REVIEW blocks emitted by the LLM are still **parsed** by `lib/ingest.ts` +and surfaced in the return value (CLI prints them as JSON; MCP renders +them in the tool reply), so a host can act on them. + +## Test strategy + +`skill/src/test-server/fake-llm-server.ts` is a real HTTP server +implementing the OpenAI-compatible Chat Completions SSE protocol and +the Tavily search REST protocol. The skill code is unchanged: real +`fetch`, real SSE parsing, real JSON decoding all execute. Only the +**upstream provider** is replaced with a deterministic local server, +which is the standard contract-test approach (vs. mocking out +`streamChat` in code, which would skip the SSE / fetch / network paths +entirely). + +`skill/src/test-server/e2e.ts` drives every CLI command and every MCP +tool against a real on-disk wiki fixture and writes the raw transcript +to `skill/docs/test-report.md`. + +Run with: +```bash +cd skill && npm run test:e2e +``` diff --git a/skill/docs/test-report.md b/skill/docs/test-report.md new file mode 100644 index 00000000..acb6ea7d --- /dev/null +++ b/skill/docs/test-report.md @@ -0,0 +1,459 @@ +# llm-wiki Skill + MCP — End-to-End Test Report + +> Generated: 2026-05-02T10:37:32.647Z +> Fixture: `/tmp/llm-wiki-e2e-XPEWdX` +> Node: v20.20.2 + +## Methodology + +All cases run **the real built CLI / MCP server** (`dist/cli.js`, `dist/mcp-server.js`) +against a real on-disk wiki fixture. LLM and Tavily traffic is served by +`dist/test-server/fake-llm-server.js`, a real local HTTP server that speaks the +OpenAI-compatible Chat Completions SSE protocol and the Tavily search REST +protocol — no code-level mocks. The skill code (`llm-client.ts`, `web-search.ts`) +runs unmodified and exercises real `fetch` / SSE parsing / JSON decoding. + +## Summary: 19/19 passed + +| # | Case | Exit | Status | Notes | +|---|------|------|--------|-------| +| 1 | init | 0 | ✅ pass | | +| 2 | status | 0 | ✅ pass | | +| 3 | search | 0 | ✅ pass | | +| 4 | graph | 0 | ✅ pass | | +| 5 | insights | 0 | ✅ pass | | +| 6 | lint | 0 | ✅ pass | | +| 7 | ingest | 0 | ✅ pass | All expected files written: wiki/sources/rnn-vs-transformer.md, wiki/concepts/recurrent-neural-network.md, wiki/entities/mamba.md | +| 8 | ingest (cache hit) | 0 | ✅ pass | Total LLM chat calls so far: 2 (cache hit should not increment beyond 2) | +| 9 | deep-research | 0 | ✅ pass | query files: research-mixture-of-experts-2026-05-02.md | mixture-of-experts page exists: true | final calls: chat=5, search=1 | +| 10 | mcp:initialize | 0 | ✅ pass | | +| 11 | mcp:tools/list | 0 | ✅ pass | tools: wiki_status, wiki_search, wiki_graph, wiki_insights, wiki_lint, wiki_ingest, wiki_deep_research | +| 12 | mcp:wiki_status | 0 | ✅ pass | | +| 13 | mcp:wiki_search | 0 | ✅ pass | | +| 14 | mcp:wiki_graph | 0 | ✅ pass | | +| 15 | mcp:wiki_insights | 0 | ✅ pass | | +| 16 | mcp:wiki_lint | 0 | ✅ pass | | +| 17 | mcp:wiki_ingest | 0 | ✅ pass | | +| 18 | mcp:wiki_deep_research | 0 | ✅ pass | | +| 19 | lint (post-ingest) | 0 | ✅ pass | | + +## Per-case detail + +### init + +``` +$ node dist/cli.js init /tmp/llm-wiki-e2e-XPEWdX/project +``` +**stdout (first 60 lines):** +``` +✓ Initialized wiki at: /tmp/llm-wiki-e2e-XPEWdX/project + +``` + +### status + +``` +$ node dist/cli.js status /tmp/llm-wiki-e2e-XPEWdX/project +``` +**stdout (first 60 lines):** +``` +Wiki: /tmp/llm-wiki-e2e-XPEWdX/project +Total pages: 6 +Communities: 3 + concept: 2 + entity: 2 + overview: 1 + source: 1 + +``` + +### search + +``` +$ node dist/cli.js search /tmp/llm-wiki-e2e-XPEWdX/project attention +``` +**stdout (first 60 lines):** +``` +# Search: "attention" + +## Attention Mechanism +**Path**: wiki/concepts/attention-mechanism.md | **Score**: 0.0164 +--- type: concept title: Attention Mechanism created: 2026-04-01 updated: 2026-04-01 tags: [ml] related: [transfor... + +## Transformer +**Path**: wiki/concepts/transformer.md | **Score**: 0.0161 +...rmer created: 2026-04-01 updated: 2026-04-01 tags: [ml, architecture] related: [attention-mechanism, bert] sources: ["intro.md"] --- # Transformer The Transformer is a... + +## Source: intro.md +**Path**: wiki/sources/intro.md | **Score**: 0.0159 +...04-01 updated: 2026-04-01 sources: ["intro.md"] tags: [] related: [transformer, attention-mechanism] --- # Source: intro.md Introduces [[transformer]] and [[attention-... + +## Index +**Path**: wiki/index.md | **Score**: 0.0156 +...: Index type: overview --- # Knowledge Base ## Concepts - [[transformer]] - [[attention-mechanism]] ## Entities - [[bert]] + +``` +**stderr (first 30 lines):** +``` +[search] "attention" | token:4 vector:0 → 4 results + +``` + +### graph + +``` +$ node dist/cli.js graph /tmp/llm-wiki-e2e-XPEWdX/project +``` +**stdout (first 60 lines):** +``` +{ + "nodes": [ + { + "id": "attention-mechanism", + "label": "Attention Mechanism", + "type": "concept", + "path": "/tmp/llm-wiki-e2e-XPEWdX/project/wiki/concepts/attention-mechanism.md", + "linkCount": 4, + "community": 0 + }, + { + "id": "transformer", + "label": "Transformer", + "type": "concept", + "path": "/tmp/llm-wiki-e2e-XPEWdX/project/wiki/concepts/transformer.md", + "linkCount": 6, + "community": 0 + }, + { + "id": "bert", + "label": "BERT", + "type": "entity", + "path": "/tmp/llm-wiki-e2e-XPEWdX/project/wiki/entities/bert.md", + "linkCount": 3, + "community": 1 + }, + { + "id": "orphan-thing", + "label": "Orphan Thing", + "type": "entity", + "path": "/tmp/llm-wiki-e2e-XPEWdX/project/wiki/entities/orphan-thing.md", + "linkCount": 0, + "community": 2 + }, + { + "id": "index", + "label": "Index", + "type": "overview", + "path": "/tmp/llm-wiki-e2e-XPEWdX/project/wiki/index.md", + "linkCount": 3, + "community": 1 + }, + { + "id": "intro", + "label": "Source: intro.md", + "type": "source", + "path": "/tmp/llm-wiki-e2e-XPEWdX/project/wiki/sources/intro.md", + "linkCount": 2, + "community": 0 + } + ], + "edges": [ + { + "source": "attention-mechanism", + "target": "transformer", + "weight": 14.329401401273703 + }, + { + "source": "transformer", + "target": "bert", +``` +**stderr (first 30 lines):** +``` +Building graph: /tmp/llm-wiki-e2e-XPEWdX/project + +✓ 6 nodes, 7 edges, 3 communities + +``` + +### insights + +``` +$ node dist/cli.js insights /tmp/llm-wiki-e2e-XPEWdX/project +``` +**stdout (first 60 lines):** +``` +# Wiki Insights + +## Surprising Connections + +### Transformer ↔ BERT +- **Score**: 4 | **Why**: crosses community boundary, different types + +### Source: intro.md ↔ Transformer +- **Score**: 4 | **Why**: connects source to concept, peripheral node links to hub + +### Source: intro.md ↔ Attention Mechanism +- **Score**: 4 | **Why**: connects source to concept, peripheral node links to hub + +## Knowledge Gaps + +### 1 isolated page +**Type**: isolated-node +Orphan Thing +💡 These pages have few or no connections. Consider adding [[wikilinks]] to related pages. + +``` + +### lint + +``` +$ node dist/cli.js lint /tmp/llm-wiki-e2e-XPEWdX/project +``` +**stdout (first 60 lines):** +``` +[orphan] Orphan Thing (orphan-thing.md) + +✓ 6 pages checked — 1 issue(s) + +``` + +### ingest + +``` +$ node dist/cli.js ingest /tmp/llm-wiki-e2e-XPEWdX/project /tmp/llm-wiki-e2e-XPEWdX/raw/rnn-vs-transformer.md +``` + +**Notes**: All expected files written: wiki/sources/rnn-vs-transformer.md, wiki/concepts/recurrent-neural-network.md, wiki/entities/mamba.md + +**stdout (first 60 lines):** +``` +{ + "status": "success", + "cached": false, + "pages": [ + "wiki/sources/rnn-vs-transformer.md", + "wiki/concepts/recurrent-neural-network.md", + "wiki/entities/mamba.md", + "wiki/log.md" + ], + "reviews_pending": 1, + "reviews": [ + { + "type": "suggestion", + "title": "Add Linear Attention page", + "description": "Linear attention deserves its own page." + } + ], + "warnings": [], + "hard_failures": [] +} + +``` +**stderr (first 30 lines):** +``` +Ingesting: /tmp/llm-wiki-e2e-XPEWdX/raw/rnn-vs-transformer.md → /tmp/llm-wiki-e2e-XPEWdX/project +✓ ingested — 4 files written, 1 review item(s), 0 warning(s) + +``` + +### ingest (cache hit) + +``` +$ node dist/cli.js ingest /tmp/llm-wiki-e2e-XPEWdX/project /tmp/llm-wiki-e2e-XPEWdX/raw/rnn-vs-transformer.md +``` + +**Notes**: Total LLM chat calls so far: 2 (cache hit should not increment beyond 2) + +**stdout (first 60 lines):** +``` +{ + "status": "success", + "cached": true, + "pages": [ + "wiki/sources/rnn-vs-transformer.md", + "wiki/concepts/recurrent-neural-network.md", + "wiki/entities/mamba.md", + "wiki/log.md" + ], + "reviews_pending": 0, + "reviews": [], + "warnings": [], + "hard_failures": [] +} + +``` +**stderr (first 30 lines):** +``` +Ingesting: /tmp/llm-wiki-e2e-XPEWdX/raw/rnn-vs-transformer.md → /tmp/llm-wiki-e2e-XPEWdX/project +✓ cache HIT — 4 files unchanged + +``` + +### deep-research + +``` +$ node dist/cli.js deep-research /tmp/llm-wiki-e2e-XPEWdX/project Mixture of Experts +``` + +**Notes**: query files: research-mixture-of-experts-2026-05-02.md | mixture-of-experts page exists: true | final calls: chat=5, search=1 + +**stdout (first 60 lines):** +``` +{ + "status": "success", + "topic": "Mixture of Experts", + "saved_path": "wiki/queries/research-mixture-of-experts-2026-05-02.md", + "web_result_count": 2, + "ingested": true, + "ingested_files": [ + "wiki/sources/research-mixture-of-experts-2026-05-02.md", + "wiki/concepts/mixture-of-experts.md" + ], + "warnings": [] +} + +``` +**stderr (first 30 lines):** +``` +Researching: "Mixture of Experts" → /tmp/llm-wiki-e2e-XPEWdX/project +✓ saved wiki/queries/research-mixture-of-experts-2026-05-02.md (2 sources, 2 pages ingested) + +``` + +### mcp:initialize + +``` +$ MCP initialize +``` +**stdout (first 60 lines):** +``` +{"result":{"protocolVersion":"2024-11-05","capabilities":{"tools":{}},"serverInfo":{"name":"llm-wiki","version":"0.4.6-mcp"}},"jsonrpc":"2.0","id":1} +``` + +### mcp:tools/list + +``` +$ MCP tools/list +``` + +**Notes**: tools: wiki_status, wiki_search, wiki_graph, wiki_insights, wiki_lint, wiki_ingest, wiki_deep_research + +**stdout (first 60 lines):** +``` +{"result":{"tools":[{"name":"wiki_status","description":"Get page count and type breakdown for a wiki project. Returns statistics about the knowledge base.","inputSchema":{"type":"object","properties":{"project_path":{"type":"string","description":"Absolute path to the wiki project directory (contains wiki/ subdirectory)"}},"required":[]}},{"name":"wiki_search","description":"Search wiki pages using BM25 keyword matching with optional vector search (RRF fusion). Returns ranked results with snippets.","inputSchema":{"type":"object","properties":{"query":{"type":"string","description":"Search query (supports Chinese and English)"},"project_path":{"type":"string","description":"Path to wiki project (defaults to WIKI_PATH env var)"},"limit":{"type":"number","description":"Max results to return (default: 10)"}},"required":["query"]}},{"name":"wiki_graph","description":"Build knowledge graph from wiki pages: wikilinks, type-based edges, Louvain community detection. Returns nodes, edges, and community clusters.","inputSchema":{"type":"object","properties":{"project_path":{"type":"string","description":"Path to wiki project"},"format":{"type":"string","enum":["json","summary"],"description":"Output format: 'json' for full graph data, 'summary' for human-readable overview (default: summary)"}},"required":[]}},{"name":"wiki_insights","description":"Analyze wiki graph structure to find surprising cross-community connections and knowledge gaps (isolated pages, sparse clusters, bridge nodes).","inputSchema":{"type":"object","properties":{"project_path":{"type":"string","description":"Path to wiki project"},"max_connections":{"type":"number","description":"Max surprising connections to return (default: 5)"},"max_gaps":{"type":"number","description":"Max knowledge gaps to return (default: 8)"}},"required":[]}},{"name":"wiki_lint","description":"Structural lint of wiki pages: find orphaned pages (no links), no-outlinks, and connectivity issues.","inputSchema":{"type":"object","prop +``` + +### mcp:wiki_status + +``` +$ MCP tools/call wiki_status +``` +**stdout (first 60 lines):** +``` +{"result":{"content":[{"type":"text","text":"Wiki: /tmp/llm-wiki-e2e-XPEWdX/project\nTotal pages: 12\nCommunities: 5\n concept: 4\n entity: 3\n source: 3\n overview: 1\n other: 1"}]},"jsonrpc":"2.0","id":3} +``` + +### mcp:wiki_search + +``` +$ MCP tools/call wiki_search +``` +**stdout (first 60 lines):** +``` +{"result":{"content":[{"type":"text","text":"# Search: \"transformer\"\n\n## Transformer\n**Path**: wiki/concepts/transformer.md | **Score**: 0.0164\n--- type: concept title: Transformer created: 2026-04-01 updated: 2026-04-01 tags: [ml, architecture] related: [atte...\n\n## Source: rnn-vs-transformer.md\n**Path**: wiki/sources/rnn-vs-transformer.md | **Score**: 0.0161\n--- type: source title: \"Source: rnn-vs-transformer.md\" created: 2026-05-02 updated: 2026-05-02 sources: [\"rnn-vs-transformer.md\"] ...\n\n## Recurrent Neural Network\n**Path**: wiki/concepts/recurrent-neural-network.md | **Score**: 0.0159\n...work created: 2026-05-02 updated: 2026-05-02 tags: [ml, architecture] related: [transformer] sources: [\"rnn-vs-transformer.md\"] --- # Recurrent Neural Network RNNs proce...\n\n## Research: Mixture of Experts\n**Path**: wiki/queries/research-mixture-of-experts-2026-05-02.md | **Score**: 0.0156\n...(MoE) MoE architectures route tokens to specialized expert sub-networks [1]. [[transformer]] models like Switch Transformer demonstrate sparse expert routing [2]. ## Re...\n\n## Attention Mechanism\n**Path**: wiki/concepts/attention-mechanism.md | **Score**: 0.0154\n...ttention Mechanism created: 2026-04-01 updated: 2026-04-01 tags: [ml] related: [transformer] sources: [\"intro.md\"] --- # Attention Mechanism Attention lets a model focus...\n\n## Mixture of Experts\n**Path**: wiki/concepts/mixture-of-experts.md | **Score**: 0.0152\n...Mixture of Experts created: 2026-05-02 updated: 2026-05-02 tags: [ml] related: [transformer] sources: [\"research-mixture-of-experts-2026-05-02.md\"] --- # Mixture of Expert...\n\n## BERT\n**Path**: wiki/entities/bert.md | **Score**: 0.0149\n...title: BERT created: 2026-04-02 updated: 2026-04-02 tags: [ml, model] related: [transformer] sources: [\"bert-paper.md\"] --- # BERT BERT is a [[transformer]]-based langua...\n\n## Mamba\n**Path**: wiki/entities/mamba.md | **Score**: 0.0147\n...05-02 updated: 2026-05-02 tags: [ml, mod +``` + +### mcp:wiki_graph + +``` +$ MCP tools/call wiki_graph +``` +**stdout (first 60 lines):** +``` +{"result":{"content":[{"type":"text","text":"# Knowledge Graph Summary\n\n**Nodes**: 12 | **Edges**: 13 | **Communities**: 5\n\n## Node Types\n- concept: 4\n- entity: 3\n- source: 3\n- overview: 1\n- other: 1\n\n## Top Communities\n### Community 1 (6 pages, cohesion: 0.53)\nKey pages: Transformer, Attention Mechanism, BERT, Index, Source: intro.md\n### Community 2 (3 pages, cohesion: 1.00)\nKey pages: Recurrent Neural Network, Source: rnn-vs-transformer.md, Mamba\n### Community 3 (1 pages, cohesion: 0.00)\nKey pages: Orphan Thing\n### Community 4 (1 pages, cohesion: 0.00)\nKey pages: log\n### Community 5 (1 pages, cohesion: 0.00)\nKey pages: Source: MoE research\n\n## Top Hubs (by link count)\n- Transformer (concept, 9 links)\n- Attention Mechanism (concept, 4 links)\n- Recurrent Neural Network (concept, 3 links)\n- BERT (entity, 3 links)\n- Index (overview, 3 links)\n- Source: rnn-vs-transformer.md (source, 3 links)\n- Mamba (entity, 2 links)\n- Source: intro.md (source, 2 links)\n- Mixture of Experts (concept, 1 links)\n- Orphan Thing (entity, 0 links)"}]},"jsonrpc":"2.0","id":5} +``` + +### mcp:wiki_insights + +``` +$ MCP tools/call wiki_insights +``` +**stdout (first 60 lines):** +``` +{"result":{"content":[{"type":"text","text":"# Wiki Insights\n\n## Surprising Connections\n\n### Source: rnn-vs-transformer.md ↔ Transformer\n- Score: 5 | crosses community boundary, connects source to concept\n\n### Source: intro.md ↔ Transformer\n- Score: 4 | connects source to concept, peripheral node links to hub\n\n### Recurrent Neural Network ↔ Transformer\n- Score: 3 | crosses community boundary\n\n## Knowledge Gaps\n\n### 3 isolated pages\nMixture of Experts, Orphan Thing, Source: MoE research\n💡 These pages have few or no connections. Consider adding [[wikilinks]] to related pages.\n"}]},"jsonrpc":"2.0","id":6} +``` + +### mcp:wiki_lint + +``` +$ MCP tools/call wiki_lint +``` +**stdout (first 60 lines):** +``` +{"result":{"content":[{"type":"text","text":"Found 3 issue(s) in 12 pages:\n\n[isolated] Mixture of Experts — only 1 link(s)\n[orphan] Orphan Thing (orphan-thing.md)\n[orphan] Source: MoE research (research-mixture-of-experts-2026-05-02.md)"}]},"jsonrpc":"2.0","id":7} +``` + +### mcp:wiki_ingest + +``` +$ MCP tools/call wiki_ingest +``` +**stdout (first 60 lines):** +``` +{"result":{"content":[{"type":"text","text":"✓ Ingested \"moe-deep-dive.md\" — 1 files written\n - wiki/sources/moe-deep-dive.md"}]},"jsonrpc":"2.0","id":8} +``` + +### mcp:wiki_deep_research + +``` +$ MCP tools/call wiki_deep_research +``` +**stdout (first 60 lines):** +``` +{"result":{"content":[{"type":"text","text":"✓ Deep research on \"RLHF\" complete\n Saved: wiki/queries/research-rlhf-2026-05-02.md\n Web results: 1\n Auto-ingested 2 wiki page(s)\n - wiki/sources/research-rlhf-2026-05-02.md\n - wiki/concepts/rlhf.md"}]},"jsonrpc":"2.0","id":9} +``` + +### lint (post-ingest) + +``` +$ node dist/cli.js lint /tmp/llm-wiki-e2e-XPEWdX/project +``` +**stdout (first 60 lines):** +``` +[isolated] RLHF — 1 link(s) +[orphan] Orphan Thing (orphan-thing.md) +[orphan] Source: MoE research (research-mixture-of-experts-2026-05-02.md) +[orphan] Source: RLHF research (research-rlhf-2026-05-02.md) +[isolated] Source: MoE deep dive — 1 link(s) + +✓ 15 pages checked — 5 issue(s) + +``` + +## Final wiki snapshot (file tree) + +``` +wiki/ + concepts/ + attention-mechanism.md + mixture-of-experts.md + recurrent-neural-network.md + rlhf.md + transformer.md + entities/ + bert.md + mamba.md + orphan-thing.md + index.md + log.md + queries/ + research-mixture-of-experts-2026-05-02.md + research-rlhf-2026-05-02.md + sources/ + intro.md + moe-deep-dive.md + research-mixture-of-experts-2026-05-02.md + research-rlhf-2026-05-02.md + rnn-vs-transformer.md + synthesis/ +``` diff --git a/skill/docs/usage-claude.md b/skill/docs/usage-claude.md new file mode 100644 index 00000000..79cc625e --- /dev/null +++ b/skill/docs/usage-claude.md @@ -0,0 +1,131 @@ +# Using llm-wiki with Claude + +Two integration paths: + +1. **Claude Desktop via MCP** — the recommended approach. Claude calls + `wiki_*` tools natively in chat. +2. **Claude Code as a Skill** — drop-in `SKILL.md` discovery so + Claude shells out to the CLI when it detects wiki-related intent. + +--- + +## 1. Claude Desktop (MCP) + +### Install + +```bash +git clone https://github.com/toughhou/llm_wiki.git +cd llm_wiki/skill +npm install +npm run build +``` + +### Configure + +Edit `~/Library/Application Support/Claude/claude_desktop_config.json` +(macOS) / `%APPDATA%\Claude\claude_desktop_config.json` (Windows): + +```json +{ + "mcpServers": { + "llm-wiki": { + "command": "node", + "args": ["/absolute/path/to/llm_wiki/skill/dist/mcp-server.js"], + "env": { + "WIKI_PATH": "/Users/me/notes/my-wiki", + "OPENAI_API_KEY": "sk-...", + "LLM_MODEL": "gpt-4o-mini", + "TAVILY_API_KEY": "tvly-..." + } + } + } +} +``` + +Restart Claude Desktop. You should see a 🔧 indicator showing the +`llm-wiki` server is connected with 7 tools. + +### Available tools + +| Tool | Purpose | +|---|---| +| `wiki_status` | Page count + type breakdown | +| `wiki_search` | BM25(+RRF) search | +| `wiki_graph` | Knowledge graph (Louvain communities) | +| `wiki_insights` | Surprising connections + knowledge gaps | +| `wiki_lint` | Find orphans / broken links | +| `wiki_ingest` | Two-stage LLM ingest of a source file | +| `wiki_deep_research` | Web search → synthesis → auto-ingest | + +### Example prompts + +> "Use wiki_status to tell me how big my knowledge base is." +> +> "Search the wiki for anything about transformers, then run insights to +> see what's missing." +> +> "Ingest the file `~/Downloads/attention-is-all-you-need.md` into my +> wiki." +> +> "Run deep research on 'sparse Mixture of Experts' and add it to the +> wiki." + +--- + +## 2. Claude Code (Skill) + +Claude Code looks for a `SKILL.md` at the repo root. The repo's +`SKILL.md` already declares the trigger conditions (knowledge base, +wiki, graph analysis, deep research) and lists the CLI commands. + +### Install + +```bash +cd llm_wiki/skill +npm install +npm run build +# Optionally, link globally so `llm-wiki` is on PATH: +npm link +``` + +### Use + +In Claude Code: + +``` +> 帮我看一下 ~/notes/my-wiki 知识库的状态 +``` + +Claude Code reads the `SKILL.md`, recognizes the trigger, and shells +out to: + +```bash +node /path/to/skill/dist/cli.js status ~/notes/my-wiki +``` + +The output (page count, communities, type breakdown) is folded back +into Claude's context. + +### CLI commands available to Claude Code + +``` +llm-wiki status +llm-wiki search +llm-wiki graph +llm-wiki insights +llm-wiki lint +llm-wiki init +llm-wiki ingest +llm-wiki deep-research +``` + +--- + +## Troubleshooting + +- **MCP tool not appearing**: check `~/Library/Logs/Claude/mcp-server-llm-wiki.log` + for startup errors. The server logs `llm-wiki MCP server vX started` + on success. +- **`No LLM configured`**: set `OPENAI_API_KEY` (or `LLM_API_KEY`) and + `LLM_BASE_URL` in the `env` block, not just in your shell. +- **`No web search results`** during deep research: set `TAVILY_API_KEY`. diff --git a/skill/docs/usage-codex.md b/skill/docs/usage-codex.md new file mode 100644 index 00000000..2b44f47e --- /dev/null +++ b/skill/docs/usage-codex.md @@ -0,0 +1,62 @@ +# Using llm-wiki with OpenAI Codex CLI + +The OpenAI Codex CLI (`codex` / `codex chat`) supports MCP servers via +its `~/.codex/config.toml`. + +## Install + +```bash +git clone https://github.com/toughhou/llm_wiki.git +cd llm_wiki/skill +npm install +npm run build +``` + +## Configure + +Add an `[mcp_servers.llm-wiki]` block to `~/.codex/config.toml`: + +```toml +[mcp_servers.llm-wiki] +command = "node" +args = ["/absolute/path/to/llm_wiki/skill/dist/mcp-server.js"] + +[mcp_servers.llm-wiki.env] +WIKI_PATH = "/Users/me/notes/my-wiki" +OPENAI_API_KEY = "sk-..." +LLM_MODEL = "gpt-4o-mini" +TAVILY_API_KEY = "tvly-..." +``` + +Run `codex mcp list` to verify the server starts and exposes the 7 +`wiki_*` tools. + +## Use + +``` +$ codex chat +> Use llm-wiki to search my wiki for "transformer" and then run insights. +``` + +Codex resolves `llm-wiki:wiki_search` and `llm-wiki:wiki_insights`, +calls them in sequence, and folds the results back into the chat +context. + +## CLI alternative (no MCP) + +If you'd rather have Codex shell out to the CLI (e.g. inside a longer +shell pipeline), `codex` can call `node dist/cli.js …` directly: + +``` +> Run `node /path/to/skill/dist/cli.js status /Users/me/notes/my-wiki` +> and summarize the output. +``` + +This works without any MCP configuration but loses the typed-tool +semantics — Codex sees raw stdout and has to parse it. + +## Tool reference + +`wiki_status`, `wiki_search`, `wiki_graph`, `wiki_insights`, +`wiki_lint`, `wiki_ingest`, `wiki_deep_research` — see +[`architecture.md`](./architecture.md). diff --git a/skill/docs/usage-copilot.md b/skill/docs/usage-copilot.md new file mode 100644 index 00000000..0ec03e7d --- /dev/null +++ b/skill/docs/usage-copilot.md @@ -0,0 +1,71 @@ +# Using llm-wiki with VS Code GitHub Copilot Chat + +GitHub Copilot Chat in VS Code supports MCP servers (Copilot Chat +≥ 0.27 with MCP enabled). + +## Install + +```bash +git clone https://github.com/toughhou/llm_wiki.git +cd llm_wiki/skill +npm install +npm run build +``` + +## Configure + +Edit `.vscode/mcp.json` in your workspace (recommended) or the user- +level VS Code MCP config: + +```json +{ + "servers": { + "llm-wiki": { + "type": "stdio", + "command": "node", + "args": ["${workspaceFolder}/skill/dist/mcp-server.js"], + "env": { + "WIKI_PATH": "${workspaceFolder}/.wiki", + "OPENAI_API_KEY": "${env:OPENAI_API_KEY}", + "LLM_MODEL": "gpt-4o-mini", + "TAVILY_API_KEY": "${env:TAVILY_API_KEY}" + } + } + } +} +``` + +`${env:VAR}` expansion lets you keep secrets in your shell environment +rather than the workspace settings file. + +Reload VS Code. Open Copilot Chat → click the tools icon → enable the +`llm-wiki` server. You should see 7 `wiki_*` tools listed. + +## Use + +In Copilot Chat (Agent mode): + +> "@workspace use the llm-wiki tools to ingest `docs/api-reference.md`, +> then show me the resulting wiki status." + +Copilot Chat will: + +1. Call `wiki_ingest` with `source_file=` +2. Call `wiki_status` +3. Format the responses into the chat reply. + +## Notes for `@workspace` flows + +- The MCP server's `WIKI_PATH` defaults to `process.cwd()`. When VS + Code spawns the server it inherits the workspace cwd, so for a + one-wiki-per-repo setup you can omit `WIKI_PATH` and just put a + `wiki/` folder at the repo root. +- For multiple separate wikis, pass `project_path` explicitly in the + prompt: *"Ingest X into the wiki at /Users/me/notes/work-wiki"*. The + tool argument overrides the env-var default. + +## Tool reference + +`wiki_status`, `wiki_search`, `wiki_graph`, `wiki_insights`, +`wiki_lint`, `wiki_ingest`, `wiki_deep_research` — see +[`architecture.md`](./architecture.md). diff --git a/skill/docs/usage-cursor.md b/skill/docs/usage-cursor.md new file mode 100644 index 00000000..16a8c902 --- /dev/null +++ b/skill/docs/usage-cursor.md @@ -0,0 +1,76 @@ +# Using llm-wiki with Cursor + +Cursor supports MCP servers natively (Settings → Features → MCP). + +## Install + +```bash +git clone https://github.com/toughhou/llm_wiki.git +cd llm_wiki/skill +npm install +npm run build +``` + +## Configure + +Open `~/.cursor/mcp.json` (create if missing): + +```json +{ + "mcpServers": { + "llm-wiki": { + "command": "node", + "args": ["/absolute/path/to/llm_wiki/skill/dist/mcp-server.js"], + "env": { + "WIKI_PATH": "/Users/me/notes/my-wiki", + "OPENAI_API_KEY": "sk-...", + "LLM_MODEL": "gpt-4o-mini", + "TAVILY_API_KEY": "tvly-..." + } + } + } +} +``` + +Open Cursor → `Cmd+,` → search for `MCP` → click **Refresh**. The +server should appear with 7 tools enabled. + +## Per-project configuration + +If you keep one wiki per project, use a project-local +`.cursor/mcp.json` instead and point `WIKI_PATH` to a folder inside the +project: + +```json +{ + "mcpServers": { + "llm-wiki": { + "command": "node", + "args": ["/absolute/path/to/llm_wiki/skill/dist/mcp-server.js"], + "env": { + "WIKI_PATH": "${workspaceFolder}/.wiki" + } + } + } +} +``` + +## Use it from Cursor Composer + +Open Composer (`Cmd+I`) and try: + +> "Use the llm-wiki MCP server: search the wiki for `attention`, then +> run insights on it." + +> "Ingest `docs/architecture.md` into the wiki." + +Cursor will route the call to the MCP server, stream the result back +into the composer panel, and let you act on it. + +## Tool reference + +Same 7 tools as Claude Desktop: +`wiki_status`, `wiki_search`, `wiki_graph`, `wiki_insights`, +`wiki_lint`, `wiki_ingest`, `wiki_deep_research`. + +See [`architecture.md`](./architecture.md) for the full env-var list. diff --git a/skill/docs/usage-hermes.md b/skill/docs/usage-hermes.md new file mode 100644 index 00000000..6b053c10 --- /dev/null +++ b/skill/docs/usage-hermes.md @@ -0,0 +1,85 @@ +# Using llm-wiki with Hermes + +Hermes Skill Runtime supports two integration styles for this project: + +1. **Hermes Skill (CLI shell-out)** — the existing `SKILL.md` / + `HERMES.md` model: Hermes loads the skill manifest, recognizes + triggers, and shells out to `node skill/dist/cli.js …`. +2. **Hermes MCP client** — Hermes recent versions support MCP servers + the same way Claude Desktop does. + +## 1. Hermes Skill (CLI) + +### Install + +```bash +git clone https://github.com/toughhou/llm_wiki.git +cd llm_wiki +bash install.sh --platform hermes # if you've set this up; otherwise: +cd skill && npm install && npm run build +``` + +The repo's root `SKILL.md` is the manifest Hermes reads. Trigger +phrases include "知识库", "wiki", "graph analysis", "deep research", +"知识图谱", etc. + +### Environment + +Export these before launching Hermes (or put them in your shell +profile): + +```bash +export OPENAI_API_KEY=sk-... +export LLM_MODEL=gpt-4o-mini +export TAVILY_API_KEY=tvly-... # for deep-research +export WIKI_OUTPUT_LANGUAGE=auto # or English / Chinese / ... +``` + +### Use + +In a Hermes session: + +``` +> 帮我把 ~/raw/paper.md 这篇论文消化进 ~/wiki 知识库 +``` + +Hermes routes to the skill, which executes: + +```bash +node /path/to/skill/dist/cli.js ingest ~/wiki ~/raw/paper.md +``` + +The JSON output (status, generated pages, review items, warnings) is +returned to Hermes for follow-up reasoning. + +## 2. Hermes MCP + +If your Hermes version supports MCP, register the server: + +```yaml +# ~/.hermes/mcp.yaml (path may vary by Hermes version) +servers: + llm-wiki: + command: node + args: ["/absolute/path/to/llm_wiki/skill/dist/mcp-server.js"] + env: + WIKI_PATH: /Users/me/wiki + OPENAI_API_KEY: sk-... + LLM_MODEL: gpt-4o-mini + TAVILY_API_KEY: tvly-... +``` + +The same 7 `wiki_*` tools become available as native Hermes tool +calls, exactly as in Claude Desktop / Cursor / Copilot Chat. + +## Choosing between Skill and MCP + +| If you want to ... | Use | +|---|---| +| Compose with shell pipelines, cron, makefiles | CLI / Skill | +| Have Hermes call typed tools with structured args | MCP | +| Run on systems without Node.js available to MCP host | CLI / Skill | +| Get streaming feedback in the agent UI | MCP | + +Both routes hit the same `skill/src/lib/` core, so there's no feature +difference — pick whichever fits the workflow.