Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
256 changes: 248 additions & 8 deletions docs/FileProcessingPipeline-zh.md

Large diffs are not rendered by default.

261 changes: 254 additions & 7 deletions docs/FileProcessingPipeline.md

Large diffs are not rendered by default.

17 changes: 9 additions & 8 deletions docs/ParserDebugCLI-zh.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Parser CLI Debuger使用指南

本工具用于本地调试 LightRAG 注册表中的任意内容解析引擎(内置 `native` / `legacy` / `mineru` / `docling`,以及通过 `lightrag.parsers` entry point 注册的第三方引擎,见 `docs/ThirdPartyParser-zh.md`),针对**单个文件**触发与 pipeline worker 相同的注册表派发路径(`get_parser(engine).parse(...)`),并把解析产物(sidecar 与 raw 缓存)输出到一个**扁平目录布局**——与生产入库目录相比,区别仅在于:
本工具用于本地调试 LightRAG 注册表中的任意内容解析引擎(内置 `native` / `legacy` / `mineru` / `docling` / `paddleocr_vl`,以及通过 `lightrag.parsers` entry point 注册的第三方引擎,见 `docs/ThirdPartyParser-zh.md`),针对**单个文件**触发与 pipeline worker 相同的注册表派发路径(`get_parser(engine).parse(...)`),并把解析产物(sidecar 与 raw 缓存)输出到一个**扁平目录布局**——与生产入库目录相比,区别仅在于:

- **无 `__parsed__/` 中间层**:产物直接落在指定父目录下,便于查看;
- **源文件不会被归档**:源文件保留在原位置(生产路径会把源文件移到 `<INPUT_DIR>/__parsed__/`);
- **raw 缓存只看目录是否存在**:`mineru` / `docling` 的 raw 目录非空即视为有效,跳过 `_manifest.json` 校验。
- **raw 缓存只看目录是否存在**:`mineru` / `docling` / `paddleocr_vl` 的 raw 目录非空即视为有效,跳过 `_manifest.json` 校验。

其余流程(IR 构建、sidecar 写入、对 `full_docs` 的同步逻辑)与生产入库完全一致,便于排查解析阶段问题。

Expand All @@ -22,10 +22,10 @@ python -m lightrag.parser.cli <input_file> \
| 参数 | 说明 |
|---|---|
| `input_file` | 待解析的源文件路径(位置参数,必填)。文件必须实际存在。 |
| `--engine` | 必填,可选值来自注册表:内置 `native`(仅 `.docx`,本地解析)/ `legacy`(纯文本抽取,无 sidecar)/ `mineru`(PDF/办公文档,调 MinerU 服务)/ `docling`(PDF/办公文档,调 docling-serve),以及任何已注册的第三方引擎。 |
| `--engine` | 必填,可选值来自注册表:内置 `native`(本地结构化解析)/ `legacy`(纯文本抽取,无 sidecar)/ `mineru`(PDF/办公文档,调 MinerU 服务)/ `docling`(PDF/办公文档,调 docling-serve)/ `paddleocr_vl`(PDF/Office/图片,调 PaddleOCR-VL 服务),以及任何已注册的第三方引擎。 |
| `-o / --sidecar-parent-dir` | sidecar 与 raw 目录的父目录,默认 = 源文件所在目录。 |
| `--doc-id` | 自定义文档 ID,默认 `doc-<md5(源文件绝对路径)>`(同一文件多次跑结果稳定)。 |
| `--force-reparse` | 仅对外部服务引擎(`mineru` / `docling` 及继承 `ExternalParserBase` 的第三方引擎)生效:清空 raw 目录、强制重新下载与解析。默认行为是 raw 目录非空即复用。 |
| `--force-reparse` | 仅对外部服务引擎(`mineru` / `docling` / `paddleocr_vl` 及继承 `ExternalParserBase` 的第三方引擎)生效:清空 raw 目录、强制重新下载与解析。默认行为是 raw 目录非空即复用。 |
| `--preview N` | 解析完成后打印前 N 个 block 的预览(headings + 内容片段),默认 5;`0` 关闭。对无 sidecar 的引擎(如 `legacy`),改为打印解析文本的前 400 字符。 |

## 输出目录布局
Expand All @@ -41,7 +41,7 @@ python -m lightrag.parser.cli <input_file> \
│ ├── sample.tables.json # 表格 sidecar(若 IR 含 tables)
│ ├── sample.drawings.json # 图纸/图片 sidecar(若 IR 含 drawings)
│ └── sample.equations.json # 公式 sidecar(若 IR 含 equations)
└── sample.pdf.<engine>_raw/ # ← mineru / docling 的 raw 缓存(native 无此目录)
└── sample.pdf.<engine>_raw/ # ← mineru / docling / paddleocr_vl 的 raw 缓存(native 无此目录)
├── _manifest.json # 由引擎下载流程写入;CLI 缓存校验不读
└── <bundle files> # 引擎特定 raw 产物(content_list.json / *.json / 资产等)
```
Expand Down Expand Up @@ -96,12 +96,13 @@ python -m lightrag.parser.cli ./inputs/workspace/sample.pdf \

## 环境变量

`mineru` / `docling` 引擎在 **缓存未命中**(首次解析或 `--force-reparse`)时会调用外部服务,所需环境变量与生产入库一致:
`mineru` / `docling` / `paddleocr_vl` 引擎在 **缓存未命中**(首次解析或 `--force-reparse`)时会调用外部服务,所需环境变量与生产入库一致:

- **MinerU**:`MINERU_API_MODE`(`local` / `official`)、`MINERU_API_TOKEN`、`MINERU_LOCAL_ENDPOINT` 或 `MINERU_OFFICIAL_ENDPOINT`,可选 `MINERU_ENGINE_VERSION` / `MINERU_MODEL_VERSION` / `MINERU_POLL_INTERVAL_SECONDS` / `MINERU_MAX_POLLS`。
- **Docling**:`DOCLING_ENDPOINT`,可选 `DOCLING_ENGINE_VERSION` / `DOCLING_DO_OCR` / `DOCLING_FORCE_OCR` / `DOCLING_OCR_ENGINE` / `DOCLING_OCR_PRESET` / `DOCLING_OCR_LANG` / `DOCLING_DO_FORMULA_ENRICHMENT` / `DOCLING_POLL_INTERVAL_SECONDS` / `DOCLING_MAX_POLLS`。
- **PaddleOCR-VL**:`PADDLEOCR_VL_API_MODE`(`official` / `local`)、`PADDLEOCR_VL_API_TOKEN` 或 `PADDLEOCR_VL_LOCAL_ENDPOINT`,可选 `PADDLEOCR_VL_OFFICIAL_ENDPOINT` / `PADDLEOCR_VL_POLL_INTERVAL_SECONDS` / `PADDLEOCR_VL_MAX_POLLS` 以及请求参数环境变量。

详见 [FileProcessingConfiguration-zh.md](./FileProcessingConfiguration-zh.md)。
详见 [FileProcessingPipeline-zh.md](./FileProcessingPipeline-zh.md)。

**缓存命中**时(raw 目录已存在且非空,且未传 `--force-reparse`)无需任何外部服务环境变量——可用于离线复现解析输出。

Expand All @@ -111,7 +112,7 @@ python -m lightrag.parser.cli ./inputs/workspace/sample.pdf \
|---|---|
| `error: input file does not exist: ...` | 检查 `input_file` 路径,必须是已存在的文件(不是 raw 目录)。 |
| raw 目录存在但 sidecar 内容仍是旧的 | 默认会**复用** raw 重建 sidecar。如果 raw 本身就过期或被替换,加 `--force-reparse` 清空重下。 |
| MinerU 报 `MINERU_API_TOKEN` 缺失 / Docling 连接 `DOCLING_ENDPOINT` 失败 | 缓存未命中触发了外部服务调用——核对对应环境变量;或确认 raw 目录是否非空(命中缓存时无需服务)。 |
| MinerU 报 `MINERU_API_TOKEN` 缺失 / Docling 连接 `DOCLING_ENDPOINT` 失败 / PaddleOCR-VL 报 endpoint 或 token 缺失 | 缓存未命中触发了外部服务调用——核对对应环境变量;或确认 raw 目录是否非空(命中缓存时无需服务)。 |
| 源文件被意外移动 | 不应发生:CLI 已 mock 归档函数。若复现请提 issue(可能是 pipeline 内增加了新的归档调用点)。 |
| docling 报 `produced zero blocks` | docling raw 中的主 JSON 内容不可解析或为空。检查 raw 目录的 `*.json` 是否合法。 |

Expand Down
17 changes: 9 additions & 8 deletions docs/ParserDebugCLI.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Parser CLI Debugger Guide

This tool is used to locally debug any parsing engine in LightRAG's registry (the built-in `native` / `legacy` / `mineru` / `docling`, plus third-party engines registered via the `lightrag.parsers` entry point — see `docs/ThirdPartyParser-zh.md`). It drives the same registry dispatch path as the pipeline worker (`get_parser(engine).parse(...)`) for a **single file** and outputs the parsing artifacts (sidecar and raw cache) into a **flat directory layout**. Compared with the production ingestion directory, the only differences are:
This tool is used to locally debug any parsing engine in LightRAG's registry (the built-in `native` / `legacy` / `mineru` / `docling` / `paddleocr_vl`, plus third-party engines registered via the `lightrag.parsers` entry point — see `docs/ThirdPartyParser.md`). It drives the same registry dispatch path as the pipeline worker (`get_parser(engine).parse(...)`) for a **single file** and outputs the parsing artifacts (sidecar and raw cache) into a **flat directory layout**. Compared with the production ingestion directory, the only differences are:

- **No `__parsed__/` intermediate layer**: artifacts land directly under the specified parent directory for easy inspection;
- **The source file is not archived**: the source file stays at its original location (the production path moves the source file to `<INPUT_DIR>/__parsed__/`);
- **Raw cache validity only checks directory existence**: any non-empty `mineru` / `docling` raw directory is considered valid, skipping `_manifest.json` validation.
- **Raw cache validity only checks directory existence**: any non-empty `mineru` / `docling` / `paddleocr_vl` raw directory is considered valid, skipping `_manifest.json` validation.

The rest of the flow (IR construction, sidecar writing, `full_docs` synchronization logic) is identical to production ingestion, making it convenient for troubleshooting parsing-stage issues.

Expand All @@ -22,10 +22,10 @@ python -m lightrag.parser.cli <input_file> \
| Argument | Description |
|---|---|
| `input_file` | Path to the source file to parse (positional argument, required). The file must actually exist. |
| `--engine` | Required; choices come from the registry: built-in `native` (only `.docx`, local parsing) / `legacy` (plain-text extraction, no sidecar) / `mineru` (PDF/Office documents, calls MinerU service) / `docling` (PDF/Office documents, calls docling-serve), plus any registered third-party engine. |
| `--engine` | Required; choices come from the registry: built-in `native` (local structured parsing) / `legacy` (plain-text extraction, no sidecar) / `mineru` (PDF/Office documents, calls MinerU service) / `docling` (PDF/Office documents, calls docling-serve) / `paddleocr_vl` (PDF/Office/images, calls PaddleOCR-VL service), plus any registered third-party engine. |
| `-o / --sidecar-parent-dir` | Parent directory of the sidecar and raw directories. Defaults to the directory containing the source file. |
| `--doc-id` | Custom document ID. Defaults to `doc-<md5(absolute path of source file)>` (stable across multiple runs on the same file). |
| `--force-reparse` | Effective only for external-service engines (`mineru` / `docling` and third-party engines subclassing `ExternalParserBase`): clears the raw directory and forces re-download and re-parse. By default, a non-empty raw directory is reused. |
| `--force-reparse` | Effective only for external-service engines (`mineru` / `docling` / `paddleocr_vl` and third-party engines subclassing `ExternalParserBase`): clears the raw directory and forces re-download and re-parse. By default, a non-empty raw directory is reused. |
| `--preview N` | After parsing completes, prints a preview of the first N blocks (headings + content snippets). Default 5; `0` disables it. For engines without a sidecar (e.g. `legacy`), prints the first 400 characters of the extracted text instead. |

## Output Directory Layout
Expand All @@ -41,7 +41,7 @@ Taking input `./inputs/workspace/sample.pdf` + the default sidecar parent direct
│ ├── sample.tables.json # table sidecar (if IR contains tables)
│ ├── sample.drawings.json # drawing/image sidecar (if IR contains drawings)
│ └── sample.equations.json # equation sidecar (if IR contains equations)
└── sample.pdf.<engine>_raw/ # ← raw cache for mineru / docling (native has no such directory)
└── sample.pdf.<engine>_raw/ # ← raw cache for mineru / docling / paddleocr_vl (native has no such directory)
├── _manifest.json # written by the engine download flow; not read by CLI cache validation
└── <bundle files> # engine-specific raw artifacts (content_list.json / *.json / assets, etc.)
```
Expand Down Expand Up @@ -96,12 +96,13 @@ python -m lightrag.parser.cli ./inputs/workspace/sample.pdf \

## Environment Variables

The `mineru` / `docling` engines call external services when the **cache misses** (first parse or `--force-reparse`); the required environment variables are identical to production ingestion:
The `mineru` / `docling` / `paddleocr_vl` engines call external services when the **cache misses** (first parse or `--force-reparse`); the required environment variables are identical to production ingestion:

- **MinerU**: `MINERU_API_MODE` (`local` / `official`), `MINERU_API_TOKEN`, `MINERU_LOCAL_ENDPOINT` or `MINERU_OFFICIAL_ENDPOINT`, optional `MINERU_ENGINE_VERSION` / `MINERU_MODEL_VERSION` / `MINERU_POLL_INTERVAL_SECONDS` / `MINERU_MAX_POLLS`.
- **Docling**: `DOCLING_ENDPOINT`, optional `DOCLING_ENGINE_VERSION` / `DOCLING_DO_OCR` / `DOCLING_FORCE_OCR` / `DOCLING_OCR_ENGINE` / `DOCLING_OCR_PRESET` / `DOCLING_OCR_LANG` / `DOCLING_DO_FORMULA_ENRICHMENT` / `DOCLING_POLL_INTERVAL_SECONDS` / `DOCLING_MAX_POLLS`.
- **PaddleOCR-VL**: `PADDLEOCR_VL_API_MODE` (`official` / `local`), `PADDLEOCR_VL_API_TOKEN` or `PADDLEOCR_VL_LOCAL_ENDPOINT`, optional `PADDLEOCR_VL_OFFICIAL_ENDPOINT` / `PADDLEOCR_VL_POLL_INTERVAL_SECONDS` / `PADDLEOCR_VL_MAX_POLLS` and request-parameter environment variables.

See [FileProcessingConfiguration.md](./FileProcessingConfiguration.md) for details.
See [FileProcessingPipeline.md](./FileProcessingPipeline.md) for details.

When the **cache is hit** (the raw directory already exists and is non-empty, and `--force-reparse` is not passed), no external service environment variables are needed — this can be used to offline-reproduce parsing output.

Expand All @@ -111,7 +112,7 @@ When the **cache is hit** (the raw directory already exists and is non-empty, an
|---|---|
| `error: input file does not exist: ...` | Check the `input_file` path; it must be an existing file (not a raw directory). |
| Raw directory exists but sidecar content is still stale | The default behavior is to **reuse** raw and regenerate sidecar. If the raw itself is outdated or has been replaced, add `--force-reparse` to clear and re-download. |
| MinerU reports `MINERU_API_TOKEN` missing / Docling fails to connect to `DOCLING_ENDPOINT` | A cache miss triggered an external service call — verify the corresponding environment variables; or confirm whether the raw directory is non-empty (no service needed when the cache hits). |
| MinerU reports `MINERU_API_TOKEN` missing / Docling fails to connect to `DOCLING_ENDPOINT` / PaddleOCR-VL reports a missing endpoint or token | A cache miss triggered an external service call — verify the corresponding environment variables; or confirm whether the raw directory is non-empty (no service needed when the cache hits). |
| Source file is unexpectedly moved | Should not happen: the CLI has mocked the archive function. If reproducible, please file an issue (a new archive call site may have been added in the pipeline). |
| docling reports `produced zero blocks` | The main JSON content in docling raw is unparseable or empty. Check whether the `*.json` files in the raw directory are valid. |

Expand Down
8 changes: 4 additions & 4 deletions docs/ThirdPartyParser-zh.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 第三方 Parser 引擎开发与注册指南

LightRAG 的解析层通过统一的 `BaseParser` 契约 + 中央引擎注册表(`lightrag/parser/registry.py`)派发所有解析引擎。内置引擎(`native` / `legacy` / `mineru` / `docling`)与第三方引擎走完全相同的派发路径:pipeline worker 与调试 CLI 都通过 `get_parser(engine).parse(ParseContext(...))` 驱动,**没有任何针对内置引擎的特判**。因此第三方包只需做两件事:
LightRAG 的解析层通过统一的 `BaseParser` 契约 + 中央引擎注册表(`lightrag/parser/registry.py`)派发所有解析引擎。内置引擎(`native` / `legacy` / `mineru` / `docling` / `paddleocr_vl`)与第三方引擎走完全相同的派发路径:pipeline worker 与调试 CLI 都通过 `get_parser(engine).parse(ParseContext(...))` 驱动,**没有任何针对内置引擎的特判**。因此第三方包只需做两件事:

1. **实现**一个 `BaseParser` 子类;
2. **注册**一个 `ParserSpec`(推荐通过 `lightrag.parsers` entry point 自动发现)。
Expand Down Expand Up @@ -113,7 +113,7 @@ class MyExternalParser(ExternalParserBase):
def build_ir(self, raw_dir, document_name) -> IRDoc: ...
```

可选覆写 `validate_ir`(构建后校验,如零 block 报错)。参考实现:`lightrag/parser/external/mineru/parser.py`、`.../docling/parser.py`。
可选覆写 `validate_ir`(构建后校验,如零 block 报错)。参考实现:`lightrag/parser/external/mineru/parser.py`、`.../docling/parser.py`、`.../paddleocr_vl/parser.py`

### 2.3 失败语义(重要)

Expand All @@ -140,7 +140,7 @@ register_parser(ParserSpec(

| 字段 | 必填 | 说明 |
|---|---|---|
| `engine_name` | ✓ | 注册表键,也是 `--engine` / 文件名 hint / `LIGHTRAG_PARSER` 里的引擎名。**与已有名字相同会覆盖原注册**(包括内置引擎)——除非有意替换实现,请勿与 `native/legacy/mineru/docling` 撞名。 |
| `engine_name` | ✓ | 注册表键,也是 `--engine` / 文件名 hint / `LIGHTRAG_PARSER` 里的引擎名。**与已有名字相同会覆盖原注册**(包括内置引擎)——除非有意替换实现,请勿与 `native/legacy/mineru/docling/paddleocr_vl` 撞名。 |
| `impl` | ✓ | `"module:Class"` 字符串。注册表只在文档实际解析时才 import 它,**注册阶段绝不能提前 import 实现**(保持能力查询 import-cheap,这是注册表的设计不变量)。 |
| `suffixes` | ✓ | 该引擎能处理的扩展名(小写无点)。用于路由校验与 worker 端后缀守门。 |
| `queue_group` | | 并发池分组,默认 `"native"`(共享 native 池)。独立池填唯一组名。 |
Expand All @@ -151,7 +151,7 @@ register_parser(ParserSpec(
### 并发模型

- pipeline 每批为**每个 `queue_group` 建一条队列 + 一组 worker**;
- 组的 worker 数:内置组(`native`/`mineru`/`docling`)由 LightRAG 实例字段 `max_parallel_parse_*` 决定(支持构造参数覆盖);第三方独立组取该组唯一 owner spec 的 `concurrency`;一个组**只能有一个**声明了 `concurrency` 的 spec,多个会在批启动时报错;
- 组的 worker 数:内置组(`native`/`mineru`/`docling`/`paddleocr_vl`)由 LightRAG 实例字段 `max_parallel_parse_*` 决定(支持构造参数覆盖);第三方独立组取该组唯一 owner spec 的 `concurrency`;一个组**只能有一个**声明了 `concurrency` 的 spec,多个会在批启动时报错;
- `queue_group="native"` 蹭内置池时,`concurrency` 不生效(池大小由 `max_parallel_parse_native` 决定,被忽略的 spec 级 `concurrency` 会在批启动时记录 warning 日志)——本地轻量引擎(如 legacy)适合这种方式,外部服务引擎建议独立组以免慢请求拖住本地解析。

## 4. 注册:entry point 自动发现(推荐)
Expand Down
Loading
Loading