[Feat] Support multi-device inference and OCR batch inference #3923

Bobholamovic · 2025-04-28T15:07:46Z

大部分OCR类产线支持batch size>1。
新增并行推理支持：内建多卡推理能力；提供多卡、多实例推理应用代码示例；新增和更新文档。

…mputing

…ovic/PaddleX into feat/parallel_computing

paddle-bot · 2025-04-28T15:07:59Z

Thanks for your contribution!

…olamovic/PaddleX into feat/optimize_ppstructurev3

…tructurev3" This reverts commit 6698697, reversing changes made to 6d96879.

Bobholamovic · 2025-05-06T11:32:21Z

docs/module_usage/instructions/model_python_API.en.md

@@ -39,7 +39,6 @@ In short, just three steps:
    * `use_hpip`：`bool` type, whether to enable the high-performance inference plugin;
    * `hpi_config`：`dict | None` type, high-performance inference configuration;
    * _`inference hyperparameters`_: used to set common inference hyperparameters. Please refer to specific model description document for details.
-  * Return Value: `BasePredictor` type.


实际返回的类型并不是BasePredictor，而是一个单下划线开头的内部类，故去除这里的错误说明。

Bobholamovic · 2025-05-06T11:33:12Z

docs/pipeline_deploy/high_performance_inference.en.md

@@ -4,7 +4,7 @@ comments: true

 # PaddleX High-Performance Inference Guide

-In real production environments, many applications impose strict performance metrics—especially in response time—on deployment strategies to ensure system efficiency and a smooth user experience. To address this, PaddleX offers a high-performance inference plugin that, through automatic configuration and multi-backend inference capabilities, enables users to significantly accelerate model inference without concerning themselves with complex configurations and low-level details.
+In real production environments, many applications impose strict performance metrics—especially in response time—on deployment strategies to ensure system efficiency and a smooth user experience. To address this, PaddleX offers a high-performance inference plugin that, through automatic configuration and multi-backend inference capabilities, enables users to significantly accelerate model inference without concerning themselves with complex configurations and low-level details. In addition to supporting inference acceleration on pipelines, the PaddleX high-performance inference plugin can also be used to accelerate inference when modules are used standalone.


基本上，各种“部署”相关的概念是适用于产线的，但高性能推理比较特别，也适用于模块，所以这里特别提了一下。

Bobholamovic · 2025-05-06T11:36:24Z

paddlex/configs/pipelines/PP-StructureV3.yaml

@@ -1,6 +1,8 @@

 pipeline_name: PP-StructureV3

+batch_size: 8


默认使用测试中表现最好的配置

Bobholamovic · 2025-05-06T11:37:40Z

paddlex/inference/models/common/static_infer.py

@@ -833,7 +833,7 @@ def _build_ui_runtime(self, backend, backend_config, ui_option=None):
                    for name, shapes in backend_config.dynamic_shapes.items():
                        ui_option.trt_option.set_shape(name, *shapes)
                else:
-                    logging.warning(
+                    logging.info(


这里不应该是警告，而是预期的行为，所以用info

Bobholamovic · 2025-05-06T11:38:15Z

paddlex/inference/models/base/predictor/base_predictor.py

@@ -335,6 +335,8 @@ def _prepare_pp_option(
            device_info = None
        if pp_option is None:
            pp_option = PaddlePredictorOption(model_name=self.model_name)
+        elif pp_option.model_name is None:


支持自动注入model_name，从而允许产线也通过pp_option设置一些基本配置

Bobholamovic · 2025-05-06T11:38:59Z

paddlex/inference/models/text_detection/processors.py

@@ -105,6 +104,8 @@ def resize_image_type1(self, img):
            resize_w = ori_w * resize_h / ori_h
            N = math.ceil(resize_w / 32)
            resize_w = N * 32
+        if resize_h == ori_h and resize_w == ori_w:


对于不resize的情况直接返回原图，减小开销

Bobholamovic · 2025-05-06T11:39:55Z

paddlex/inference/pipelines/__init__.py

-        config["use_hpip"] = use_hpip
-    if hpi_config is not None:
-        config["hpi_config"] = hpi_config
+    if use_hpip is None:


这里原本的逻辑有点问题，可能导致配置文件中的设置不生效，这里修复了

Bobholamovic · 2025-05-06T11:40:25Z

paddlex/modules/base/trainer.py

@@ -21,7 +21,7 @@
    set_env_for_device,
    update_device_num,
 )
-from ...utils.flags import FLAGS_json_format_model, DISABLE_CINN_MODEL_WL
+from ...utils.flags import DISABLE_CINN_MODEL_WL, FLAGS_json_format_model


linter自动修复

Bobholamovic · 2025-05-06T11:41:08Z

paddlex/inference/pipelines/layout_parsing/xycut_enhanced/xycuts.py

-                block.region_label not in mask_labels
-                and block.secondary_direction == cut_direction
-            ):
+    if len(all_boxes) > 0:


缺少对边界情况all_boxes为空的处理

Bobholamovic · 2025-05-06T11:49:32Z

paddlex/inference/pipelines/base.py

@@ -101,7 +101,9 @@ def create_model(self, config: Dict, **kwargs) -> BasePredictor:
            model_dir=model_dir,
            device=self.device,
            batch_size=config.get("batch_size", 1),
-            pp_option=self.pp_option,
+            pp_option=(
+                self.pp_option.copy() if self.pp_option is not None else self.pp_option


PaddlePredictorOption和Predictor是组合关系（predictor对pp_option的生命周期负责）而不是聚合关系，同时采用依赖注入的方式，为了防止predictor对pp_option的原地修改导致一个产线的不同模型错误地共享trt dynamic shape等信息，每次创建模型时对pp_option作一次拷贝

Bobholamovic added 13 commits March 20, 2025 22:55

feat/parallel_computing

1b6d11e

Rename functions and parallelize read operations

ce01e2f

Add full support for predictor processors

2bdae90

Merge branch 'develop' into feat/parallel_computing

6243f45

Optimize for OCR

aa16d50

Merge remote-tracking branch 'official/develop' into feat/parallel_co…

346cf5b

…mputing

Remove optimization for rotate_image

1dc9d5e

Fix bug

b5a8ecd

Merge remote-tracking branch 'official/develop' into feat/parallel_co…

c43df36

…mputing

Merge remote-tracking branch 'official/develop' into feat/parallel_co…

caf91c1

…mputing

Merge branch 'feat/parallel_computing' of https://github.com/Bobholam…

551e289

…ovic/PaddleX into feat/parallel_computing

Merge branch 'develop' into feat/parallel_computing

90c5170

Support multi-device inference and pipeline batch inference

a524e6a

Bobholamovic changed the title ~~[Fix] Support multi-device inference and OCR batch inference~~ [Feat] Support multi-device inference and OCR batch inference Apr 28, 2025

Bobholamovic added the wip label Apr 28, 2025

Bobholamovic and others added 14 commits April 29, 2025 00:20

Fix hpip bug

fdc923b

Fix hpip bug

6e7e7fb

Fix hpip bug

0e6382c

warning->info

53f7d82

Fix table recognition v2 bugs

6d96879

Merge branch 'feat/parallel_computing' into feat/optimize_ppstructurev3

6698697

Support seal recognition and PP-StructureV3

a40876c

Update doc

983d87f

Fix

0b4621e

PaddlePredictorOption supports copy

d400339

Fix OCR bug

c458d94

No parallel if iterable has only one element

5b35046

Add quick return for text det resize

3e9736a

Fix xycuts bug

c266c60

Bobholamovic and others added 9 commits April 30, 2025 15:02

Merge branch 'feat/optimize_ppstructurev3' of https://github.com/Bobh…

9401d6d

…olamovic/PaddleX into feat/optimize_ppstructurev3

Revert "Merge branch 'feat/parallel_computing' into feat/optimize_pps…

25ab205

…tructurev3" This reverts commit 6698697, reversing changes made to 6d96879.

Fix table bug

c7fa3dc

Cancel execution permission for regular files in ultra-infer

3bac9ea

More pipelines support multi-gpu inference

4f7d590

PP-StructureV3 by default uses bs8

cbccae0

Reset ultra-infer

ba9ffe8

Add parallel inference docs

a91c416

Add parallel inference doc to mkdocs.yml

6e0db1e

Bobholamovic removed the wip label May 6, 2025

Bobholamovic commented May 6, 2025

View reviewed changes

Bobholamovic requested a review from cuicheng01 May 6, 2025 11:55

Bobholamovic added 13 commits May 6, 2025 20:03

Update doc

95b433a

Update opset_version description

dfc8833

Fix serving docs

8c3bfb1

PP-FormulaNet-L supports paddle_fp16

b4d8ee9

/workspace -> /app

40479d2

Unset paddle_fp16

7bced71

Add FAQ in hpi doc

4b8a6c2

Fix docs

3b488d7

Update docs

40e684c

Merge branch 'develop' into feat/optimize_ppstructurev3

0ba3194

Update PP-StructureV3 interface

76d94a7

Add note on edge deployment

10eb0aa

Polish docs

0fc4f9a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feat] Support multi-device inference and OCR batch inference #3923

[Feat] Support multi-device inference and OCR batch inference #3923

Bobholamovic commented Apr 28, 2025 •

edited

Loading

paddle-bot bot commented Apr 28, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

Bobholamovic May 6, 2025

[Feat] Support multi-device inference and OCR batch inference #3923

Are you sure you want to change the base?

[Feat] Support multi-device inference and OCR batch inference #3923

Conversation

Bobholamovic commented Apr 28, 2025 • edited Loading

paddle-bot bot commented Apr 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bobholamovic commented Apr 28, 2025 •

edited

Loading