Repository navigation
feat(docreader): 上传 HTML 只索引正文,与 URL 导入路径对齐(开关,默认关) - #4424
Merged
lyingbug merged 3 commits intoOct 17, 2026
Merged
Conversation
docreader 的「上传 HTML 文件」路径(html_parser.py)把整页转 Markdown, 导航栏、侧栏、页脚都会进入索引并被当正文召回;而「URL 导入」路径 (web_parser.py)早已用 trafilatura 抽正文。同一产品两条路径行为不一致。 按仓库既有开关模式(DocReaderConfig + DOCREADER_* 环境变量)新增 DOCREADER_HTML_MAIN_CONTENT,默认 false:默认仍整页转换,索引输出逐字节 不变,升级不改变既有文档的解析结果。开启后改用 trafilatura 抽正文; 抽取结果 < 120 字符、或不足整页转换的 15% 时回退整页转换,表单页、 纯列表页、短公告等没有明确主体的页面不会被吃掉。 - docreader/config.py:新增 DOCREADER_HTML_MAIN_CONTENT(默认 false), 并进入 dump_config - docreader/parser/html_parser.py:关闭时走原路径;开启时抽正文 + 双阈值回退 - docreader/tests:新增 6 个正文抽取用例、3 个配置用例,夹具见 docreader/testdata/html_main_content/ - docker-compose.yml / .env.example / docreader/README.md / website-docs: 登记该开关
Contributor
Author
7 of 16 tasks
lyingbug
approved these changes
Oct 17, 2026
lyingbug
left a comment
Collaborator
There was a problem hiding this comment.
已复核此前评审结论及最新提交,相关 CI 检查全部通过,同意合入。
lyingbug
approved these changes
Oct 17, 2026
lyingbug
left a comment
Collaborator
There was a problem hiding this comment.
已复核与最新 main 的合并:冲突仅为 .env.example 和 docker-compose.yml 中相邻配置项,已完整保留 HTML 正文抽取与 XLSX 百分比开关。Compose 配置校验及 65 项相关 Python 回归测试通过,等待本次提交的 CI 完成后合入。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题:同一个 docreader 里,「上传 HTML」和「URL 导入」两条路径行为不一致
docreader/parser/web_parser.py):Playwright 抓取 →trafilatura.extract()抽正文 → 抽取为空时回退可见文本;docreader/parser/html_parser.py):MHTMLParser.html_to_markdown()整页转换 → 导航栏、侧栏、页脚、搜索框、Cookie 提示全部进入索引,检索时被当正文召回。docreader 本来就依赖 trafilatura(
pyproject.toml/uv.lock钉 2.0.0),URL 路径一直在用,只是文件路径没用上。改动
新增
DOCREADER_HTML_MAIN_CONTENT(DocReaderConfig,默认 false),沿用仓库既有开关模式(与DOCREADER_ODL_MARKDOWN_WITH_HTML同形):回退策略(两层):表单页/纯列表页/短公告没有明确主体 → 抽取结果过短 → 回退整页转换;长页只抽到一小段 → 不足全量转换的 15% → 回退整页转换。回退即完整保留旧行为,不丢内容。
为什么下限取 250:trafilatura 在自己的
MIN_EXTRACTED_SIZE(250)以下会退化成纯文本渲染,那会丢掉标题层级、表格和链接目标 —— 落在两个阈值之间的页面就不是"少掉 chrome"而是"丢掉结构"。把下限卡在 trafilatura 自己的地板上,让整页转换在那一段接管。(实测:仓库自带的docreader/testdata/test.html抽取只有 151 字符,取 120 时会丢表格和全部<h2>;取 250 时回退整页转换、结构完整。)为什么是开关,而不是"两条路径共用一个函数":
extract_markdown_from_html()用with_metadata=True, include_images=True:元数据头(title:/url:/date:)会被当正文索引,而本地上传文件没有 URL;上传路径一直extract_images=False(图片交给 Go 侧image_resolver.go)。直接复用会改变上传路径的语义;internal/types/knowledgebase.go的ImageAttrsEnabled注释),docreader 侧的同形实现就是DOCREADER_*环境变量。真正该共用的是策略(抽正文 + 失败回退),不是函数。
测试
docreader/tests/test_html_parser.py(新增HTMLMainContentTest,6 例)+test_config.py(3 例),夹具docreader/testdata/html_main_content/*.html。逐字节一致性:把
origin/main的解析器与改后默认解析器分别跑 20 个构造 HTML(空/纯脚本/表格/实体/BOM/UTF-16/windows-1252/畸形/深嵌套/大导航/长文…)+ 全部仓库夹具,SHA-256 全等;完整测试套件 296 → 305 例,失败面不变。已知限制(本 PR 不处理,欢迎维护者定夺)
<h1>+ 单个约 250–400 字段落)会把正文输出两遍(fast/no_fallback/favor_precision/deduplicate均无效),开启后这类页面会索引重复正文,且虚高的长度可能绕过比例阈值;include_images=False(URL 路径是True),开启后正文内的图片引用不进 Markdown(远程图片仍由 Go 侧解析)。兼容性
默认关闭 → 升级后索引内容不变。开启方式:
DOCREADER_HTML_MAIN_CONTENT=true(已加入docker-compose.yml/.env.example/docreader/README.md/website-docs)。后续如需 per-upload override,可按pdf_force_scanned/odl_markdown_with_html的先例加parser_engine_overrides(本 PR 不含)。