feat(kb): [WIP]overhaul knowledge base workflows and retrieval

[lxfight]

Improve the knowledge base feature across backend storage, retrieval quality, upload/import workflows, dashboard usability, and regression coverage.

Changes

Modifications / 改动点

• Add and harden KB document workflows, including batch delete, document search, upload/import progress, partial failure reporting, and safer rollback behavior.

• Improve retrieval behavior with kb_ids filtering, HNSW support, cosine/IP FAISS migration, parallel dense retrieval, sparse retrieval fixes, RRF cleanup, and BM25 memory protection.

• Add storage and migration safeguards, including FAISS index backup, external ID preservation, incremental index_type migration, KB-scoped stats, cascade deletion, generated kb_id column, name index, and initialization retry/locking.

• Preserve more Markdown structure during chunking.

• Improve dashboard KB pages, including overview/detail/document layouts, document search, upload UX, validation feedback, failed upload surfacing, task status handling, and localized text.

• Add dashboard tests and consolidate Python KB regression coverage into tests/test_knowledge_base.py.

• This is NOT a breaking change. / 这不是一个破坏性变更。

Screenshots or Test Results / 运行截图或测试结果

• uv run ruff format .
• uv run ruff check .
• uv run pytest tests/test_knowledge_base.py -q
• node --test dashboard/tests/knowledgeBase.test.mjs

---

Checklist / 检查清单

• 😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
  / 如果 PR 中有新加入的功能，已经通过 Issue / 邮件等方式和作者讨论过。

• 👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
  / 我的更改经过了良好的测试，并已在上方提供了“验证步骤”和“运行截图”。

• 🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in requirements.txt and pyproject.toml.
  / 我确保没有引入新依赖库，或者引入了新依赖库的同时将其添加到 requirements.txt 和 pyproject.toml 文件相应位置。

• 😮 My changes do not introduce malicious code.
  / 我的更改没有引入恶意代码。

[sourcery-ai]

Sorry, we are unable to review this pull request

The GitHub API does not allow us to fetch diffs exceeding 20000 lines

[gemini-code-assist]

Code Review

This pull request introduces comprehensive upgrades to the knowledge base ingestion, storage, and retrieval pipelines, including FTS5-based document searching, a thread-safe embedding cache, automatic FAISS index migration from L2 to Inner Product, background ingestion task tracking, and index consistency check/repair tools. Review feedback identified critical issues: a swallowed RuntimeError during single-KB dense retrieval due to asyncio.gather catching exceptions, potential SQLite database locking during concurrent batch deletions, a potential TypeError in FAISS index saving when the path is None, and a potential AttributeError in task serialization if timestamp fields are missing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

In _dense_retrieve, asyncio.gather is called with return_exceptions=True. This means any RuntimeError raised inside _retrieve_one (which is explicitly raised when len(kb_ids) == 1) will be caught and returned as an exception object in results_per_kb instead of being raised to the caller. The loop then simply logs the exception and continues, effectively swallowing the error and returning an empty list [] instead of raising the intended RuntimeError.

We should check if len(kb_ids) == 1 and re-raise the exception if it is an instance of Exception.

Suggested change

	results_per_kb = await asyncio.gather(*tasks, return_exceptions=True)
	all_results: list[Result] = []
	for result in results_per_kb:
	if isinstance(result, Exception):
	logger.error(f"稠密检索异常: {result}", exc_info=True)
	continue
	all_results.extend(result)
	results_per_kb = await asyncio.gather(*tasks, return_exceptions=True)
	all_results: list[Result] = []
	for result in results_per_kb:
	if isinstance(result, Exception):
	if len(kb_ids) == 1:
	raise result
	logger.error(f"稠密检索异常: {result}", exc_info=True)
	continue
	all_results.extend(result)

[gemini-code-assist]

In delete_documents_by_ids, asyncio.gather is used to concurrently execute _delete_one for all candidate documents. Since _delete_one performs write operations on the SQLite database (via delete_documents), running them concurrently without any rate-limiting or serialization will likely cause SQLite "database is locked" (OperationalError) errors under load or with many documents. It is safer to execute them sequentially or use a semaphore to limit concurrency.

Suggested change

	vec_results = await asyncio.gather(
	*[_delete_one(doc_id) for doc_id in candidates],
	)
	vec_results = []
	for doc_id in candidates:
	vec_results.append(await _delete_one(doc_id))

[gemini-code-assist]

In _save_index_locked, faiss.write_index is called with self.path. However, self.path can be None (as defined in __init__ where path: str | None = None). If self.path is None, calling write_index will raise a TypeError. A guard should be added to ensure self.path is not None before attempting to write the index.

Suggested change

	if self.index is None:
	return
	faiss.write_index(self.index, self.path)
	await asyncio.to_thread(faiss.write_index, self.index, self.path)
	if self.index is None or self.path is None:
	return
	await asyncio.to_thread(faiss.write_index, self.index, self.path)

[gemini-code-assist]

In _task_to_dict, task.created_at and task.updated_at are converted to ISO format using .isoformat(). If either of these fields is None (e.g., due to legacy database records or corrupted states), this will raise an AttributeError. Adding a None-check ensures robust defensive programming.

Suggested change

	"created_at": task.created_at.isoformat(),
	"updated_at": task.updated_at.isoformat(),
	"created_at": task.created_at.isoformat() if task.created_at else None,
	"updated_at": task.updated_at.isoformat() if task.updated_at else None,