Skip to content

feat(chat): add optional character RAG with bilingual starter knowledge - #59

Merged
Lucas1479 merged 1 commit into
mainfrom
codex/optional-character-rag
Sep 6, 2026
Merged

feat(chat): add optional character RAG with bilingual starter knowledge#59
Lucas1479 merged 1 commit into
mainfrom
codex/optional-character-rag

Conversation

@Lucas1479

@Lucas1479 Lucas1479 commented Sep 6, 2026

Copy link
Copy Markdown
Member

Main Chat can miss Kurisu's canonical BBS handle, especially across Chinese/Japanese spellings. This adds optional local character retrieval shared by all eight Chat routes, with a sourced bilingual starter corpus. RAG is off by default and the static persona is unchanged. Based on current public main d5e1d7f, including #55.

  • Reuse FAISS + multilingual E5 and JSON source data. Build from a file or directory; include 15 topics / 30 Chinese and Japanese records. Personal data, index binaries and weights are not published.
  • Configure enablement, index directory, top-k and threshold through .env or Settings. Show applied configuration, loading/failure state and last retrieval summary. CLI diagnostics show rejected candidates as well as accepted hits.
  • Label retrieved passages as fallible current-turn reference data, preserve the original user message/history and existing fallback behavior, and reuse the cache-only model-loading boundary. Accepted excerpts go to the selected model, including remote APIs.
  • Add the optional rag extra without changing CPU/cu124/ROCm build selection; add a dedicated CPU retrieval CI job. Update both READMEs, setup/evaluation docs and source-archive selection.

Validation: 107 targeted Python tests passed (1 skipped); all 32 real E5 retrieval development cases passed; 15 Electron tests and TypeScript/Vite build passed. Ruff, architecture views, clean core environment verification and source-release checks passed. Clean model-less core regression: 1,837 passed, 12 skipped. Remote CI is tracked by the checks below.

The evaluation records 100 live Flash requests across successive corpus iterations, not 100 persona passes. Known limitations include irrelevant retrieval for a Chinese greeting, occasional masculine voice, unprompted Christina reactions and invented shared history. Some failures occur without any retrieved reference. These are disclosed; this PR does not claim to solve default-off persona consistency or justify enabling RAG by default.

Related to #56; keep the issue open for contributor retesting and separate persona-baseline follow-up. Setup: optional character knowledge.


Main Chat 对红莉栖 BBS 网名及中日文写法的识别存在缺口。这次接入各聊天路径共用的可选本地检索,并提供有来源的中日文基础资料;默认关闭,固定 persona 不变。分支基于包含 #55 的公开主线 d5e1d7f

复用 FAISS/E5/JSON 基础方式,支持知识目录和文件,附 15 个主题、30 条资料。开关、索引目录、top-k 和阈值可通过 .env 或 Settings 配置;显示实际生效值、加载状态和最近检索摘要,CLI 可查看被过滤的候选。资料作为明确标注的本轮参考传入,保留原始用户输入和历史;命中文字会发给所选模型,包括远程 API。模型仅在显式准备时下载,聊天运行时使用本地缓存。

新增可选 rag 依赖和 CPU 检索 CI,保持现有 CPU/cu124/ROCm 构建选择,补齐中英文 README、配置及评估说明。未上传旧个人库、索引或模型。

验证:107 项相关 Python 测试通过、1 项跳过;32 条真实检索开发用例通过;15 项 Electron 测试、前端构建、Ruff、架构视图、干净 core 环境及源码发布检查通过。干净且未安装模型依赖的 core 完整回归:1,837 项通过、12 项跳过。远程 CI 状态见下方检查。

评估记录了多个资料迭代中的 100 次真实 Flash 调用,不是 100 次角色测试全通过。中文闲聊误召回、偶发男性口吻、无端反应“克里斯蒂娜”和虚构共同经历等仍有反例,部分出现在没有检索资料时。这次先完成 RAG 和资料,人设基线另行处理;#56 保持开启,邀请贡献者重建索引并复测。

@Lucas1479
Lucas1479 merged commit 1b70c07 into main Sep 6, 2026
7 checks passed
@Lucas1479
Lucas1479 deleted the codex/optional-character-rag branch September 6, 2026 10:29
Lucas1479 added a commit that referenced this pull request Sep 7, 2026
…jects (#60)

The old 148-entry local corpus contains useful character facts alongside
duplicates, incorrect member numbers and unverified anecdotes. This
data-only follow-up to #59 audits every legacy row and expands the
public corpus from **30 to 52 Chinese/Japanese records, covering 26
topics**.

The update corrects Moeka/Ruka/Faris to 005/006/007, adds the Science
publication, chopsticks/cooking, father relationship, PhoneWave, Okabe's
phrase, supporting characters and the lab building, and deduplicates
already covered facts. Twenty old rows have core facts already covered;
verified portions of 16 more are incorporated/corrected; 112 are
deferred for sourcing or narrative context, not declared universally
false. The original local files remain unchanged. Sources and all row
dispositions are documented.

The father entry uses the maintainer's suggested subject anchoring:
`私(牧瀬紅莉栖)の父親はドクター中鉢。` Other reviewed wording stays intact. A blanket
first-person rewrite was tested but added four unrelated-query false
hits. No runtime code, static persona, RAG wrapper, default-off setting,
threshold, top-k, or dependencies change.

Validation:

- 70 targeted Python tests passed, including all 67 development
retrieval cases and release-tooling tests; Ruff, architecture views and
source-release validation passed.
- Additional held-out retrieval improved from 2/18 with the 30-entry
corpus to 11/18; remaining failures and six pre-existing unrelated-query
false hits are explicitly recorded.
- 153 live Flash requests across corpus comparison and follow-up wording
tests. Central facts improved, but these are not 153 persona passes.
RAG-off and first-person arms both produced an explicit `俺` in the
focused experiment; shared-history fabrication and unsupported
elaboration remain.

[Audit, method, results and
limitations](https://github.com/Code-Amadeus/Amadeus/blob/codex/character-knowledge-expansion/docs/character_rag_curation.md).
Related to #56; this does not close its default-off persona acceptance
criterion.

---

这是 #59 后续的纯资料更新:逐条审看旧本地 148 条,去重、核对及修正后,公开库从 **30 条扩为中日文 52 条、26
个主题**。20 条的核心事实已有覆盖,16 条中的可核实事实融入,112 条待来源或剧情上下文确认;不代表判定它们全部错误,旧本地文件未改动。


修正萌郁/琉华/菲利斯的编号,补充论文期刊、筷子与料理、父亲、电话微波炉、角色及场所等事实。父亲条目采用明确的“私(牧瀬紅莉栖)の父親は…”写法;全库统一改第一人称新增了四个无关问题误召回,因此没有直接采用。默认
prompt、RAG 包装、开关、阈值、top-k 和依赖不变。

70 项相关 Python 测试通过,包含 67 条真实检索开发回归;额外问法从 2/18 提升至 11/18,失败情况完整保留。另做 153
次真实模型调用,确认事实覆盖改善,但男性口吻、虚构共同经历及附加解释仍有反例,不宣称人物表现全部修好。建议作为默认关闭 RAG
的资料更新审阅,#56 继续开启。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant