Skip to content

fix(search): keep session logs out of retrieval - #3018

Open
r266-tech wants to merge 2 commits into
volcengine:mainfrom
r266-tech:fix/search-exclude-session-logs-2989
Open

fix(search): keep session logs out of retrieval#3018
r266-tech wants to merge 2 commits into
volcengine:mainfrom
r266-tech:fix/search-exclude-session-logs-2989

Conversation

@r266-tech

@r266-tech r266-tech commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Filter legacy and canonical session transcripts plus generated L0/L1 sidecars before quick, global, and recursive retrieval.
  • Normalize encoded URIs with bounded work, fail closed on incomplete normalization, and expose incomplete replacement scans through QueryResult.truncated provenance.
  • Page past filtered vector hits within one request-wide replacement budget while preserving each directory's existing first search.
  • Thread child-search offsets through the retrieval proxy, manager, and backend.

Fixes #2989.

Tests

  • ruff check openviking/retrieve/hierarchical_retriever.py openviking/storage/vikingdb_manager.py openviking/storage/viking_vector_index_backend.py openviking_cli/retrieve/types.py tests/storage/test_hierarchical_retriever_session_logs.py
  • python -m py_compile openviking/retrieve/hierarchical_retriever.py openviking/storage/vikingdb_manager.py openviking/storage/viking_vector_index_backend.py openviking_cli/retrieve/types.py tests/storage/test_hierarchical_retriever_session_logs.py
  • Dependency-isolated current-main behavior harness: 5 groups passed (URI policy, replacement paging, quick/image candidate budgets, recursive paging, and offset forwarding).

Full upstream pytest was not available in the no-clone maintenance environment; GitHub CI is the integration check.

@huangruiteng

Copy link
Copy Markdown
Collaborator

@r266-tech @qin-ctx 我按当前 origin/main@4295dfdef0e8f8ba2f4cd26eb381552fe3b95d29 重新核过了这个问题,结论是:问题仍然有效,但需要基于当前主干收敛后再合并。

  • 当前主干已经在 index_resource 入口跳过新的 session URI 索引;但 prune_orphans 只会删除源文件已不存在的 L2 记录。旧库中的 messages.jsonl 文件仍然存在,所以历史向量仍可能被 find 召回,不能直接把 [Bug] v0.4.7 search/find and search/search rank raw messages.jsonl above real memory #2989 当成已被主干覆盖。
  • PR 头 77d6396f4c042b302e41d86aa46890dd722a9b56 的聚焦测试已复跑:54 passedtests/storage/test_hierarchical_retriever_session_logs.py)。过滤谓词、补量分页和边界用例方向是成立的。
  • 但该分支目前与主干冲突;主干的 hierarchical_retriever.py 已新增异步 rerank、图片检索和 rerank 输入 token 预算等路径。直接保留旧版 retriever 的大段修改会回退这些能力。
  • offset 透传链路当前主干已经具备,因此 rebase 后不需要继续保留这部分重复改动。

建议下一版只保留一个窄修复:把“内部 session transcript 不可召回”做成可复用 URI/索引策略,在当前主干的 quick/global/children 查询结果上过滤历史向量,并按需分页补足 limit;同时保留现有异步、多模态、token-budget 行为和对应回归测试。这样既覆盖旧索引,又避免重写整个检索器。

请按这个方向 rebase/refine;更新后我会继续在本 PR 上复核。

@r266-tech
r266-tech force-pushed the fix/search-exclude-session-logs-2989 branch from 77d6396 to 15c994c Compare July 22, 2026 18:05
@r266-tech

Copy link
Copy Markdown
Contributor Author

已按这个方向基于当前 main@09cd4782 重做并更新了本 PR,新的 head 是 15c994c,现在是单提交且已无冲突。

  • 修复只作用于 retrieval 边界:quick/global/children 都会过滤旧索引里的 session transcript 与生成的 L0/L1 sidecar;现有 async rerank、image query 的 50-candidate budget 和 rerank token budget 路径都保留。
  • URI 匹配覆盖 canonical/legacy 形态,并对 percent-decoding 做有界、fail-closed 处理;归一化不完整会传播为 QueryResult.truncated,不会静默放行。
  • 补量分页保留每个目录原有的首个查询,只让额外 replacement pages 消耗整次请求共享的上限,避免递归场景按父目录放大。
  • child offset 只沿 proxy/manager/backend 现有链路透传;没有扩展 retrieval config,也没有加入额外 manager override。

本地已通过 Ruff、py_compile 和 5 组 dependency-isolated current-main 行为回归;完整 upstream pytest 交给本次 GitHub CI 验证。

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head 15c994c967b0202de0936f7ab1a8ace3d7a2c0b2 against current main@09cd4782df3c8a443c3785eebb7fbd7365628eed. The rework is now conflict-free and preserves the current async rerank, image candidate, and rerank-token-budget paths. The bounded URI classification, request-wide replacement-page budget, child offset propagation, and QueryResult.truncated plumbing are coherent. Two test-compatibility fixes are still required before approval:

  1. The new child paging path always passes offset=. The existing DummyStorage / QuickSearchStorage / DirectChildProxy test doubles in tests/retrieve/test_hierarchical_retriever_rerank.py do not accept that keyword, so three existing tests now fail with TypeError. These same three nodes pass 3/3 on exact current main. Please add an optional offset: int = 0 to the affected doubles (they may discard it).
  2. test_filtered_search_reports_scan_cap fails on this exact head because the project logger writes the warning but does not propagate it into caplog; the assertion sees an empty capture. Please assert through a patched retriever_module.logger.warning (or another logger-compatible capture) instead.

The exact failing nodes are:

  • tests/storage/test_hierarchical_retriever_session_logs.py::test_filtered_search_reports_scan_cap
  • tests/retrieve/test_hierarchical_retriever_rerank.py::test_retrieve_uses_rerank_scores_in_thinking_mode
  • tests/retrieve/test_hierarchical_retriever_rerank.py::test_retrieve_falls_back_to_vector_scores_when_rerank_returns_none
  • tests/retrieve/test_hierarchical_retriever_rerank.py::test_score_propagation_alpha_uses_configured_weight

I applied only those test-double/capture adjustments locally and the related retrieval set passed 106/106. Please also run Ruff fix/format: the exact head currently reports an unsorted private-name import in the new test file and would reformat two lines in hierarchical_retriever.py. After the local test adjustments plus those mechanical fixes, Ruff check, format check, and git diff --check all pass.

@r266-tech

Copy link
Copy Markdown
Contributor Author

已在 efe883e 处理这轮反馈:

  • 三个现有 child-search test double 现在都接受 offset,并按 results[offset : offset + limit] 返回;带调用记录的两个 double 也会记录 offset。
  • scan-cap 回归改为 patch 项目 logger 的 warning,断言该预算告警恰好调用一次,不再依赖 caplog 传播。
  • 已按仓库配置完成 Ruff import/format 修正。

本地通过 Ruff check、Ruff format check、三个修改文件的 py_compile 和 offset 签名静态核验。新的 GitHub integration check 正在运行。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

[Bug] v0.4.7 search/find and search/search rank raw messages.jsonl above real memory

2 participants