Docker容器中使用Qwen3-VL会出现推理异常

[bullet46]

一. 系统环境与版本
使用例程：https://github.com/sophgo/sophon-demo/tree/release/sample/Qwen3-VL

=============== Demo Completed ===============
(base) linaro@bm1684:/data$ bm_version
SophonSDK version: v23.09 LTS-SP3
sophon-soc-libsophon : 0.5.1
sophon-soc-libsophon-dev : 0.5.1
sophon-mw-soc-sophon-ffmpeg : 0.10.0
sophon-mw-soc-sophon-opencv : 0.10.0
BL2 v2.7(release): Built : 08:54:37, Jan 22 2025
BL31 v2.7(release): Built : 08:54:37, Jan 22 2025
U-Boot 2022.10 (Jan 23 2025 - 18:56:00 +0800) Sophon BM1684X
KernelVersion : Linux bm1684 5.4.217 #41 SMP Fri Sep 12 18:03:23 CST 2025 aarch64 aarch64 aarch64 GNU/Linux
HWVersion: 0x00
MCUVersion: 0x09
(base) linaro@bm1684:/data$ docker --version
Docker version 19.03.8, build afacb8b7f0

二. 测试步骤

1. 使用Docker容器(自制)拉起Qwen3-VL模型，在容器进程1中，提供模型推理服务。
2. 询问问题，此时表现正常。
3. 在另一个样机进程2中，调用YUV转JPEG的函数。
4. 再次询问问题，模型输出为循环的id为0的Token，一直在打感叹号。
5. 即使退出进程2，模型推理也无法恢复。

三. 测试现象

四. 补充测试

1. 使用例程https://github.com/sophgo/LLM-TPU/tree/main/models/Qwen3_VL未出现问题。
2. 不使用Docker，使用pyenv环境启动旧例程，也表现正常。

[bullet46]

测试感觉和Docker环境有强相关。Docker版本是当时样机自带的，Docker环境也是在样机板上构建的，具体Dockerfile如附件。

Dockerfile.python310，用于提供Python3.10环境

# Dockerfile.base
FROM rdsource.tp-link.com.cn:8088/arm64v8/ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive \
    TZ=Asia/Shanghai \
    PYTHONIOENCODING=utf-8

USER root

# 1. 复制编译 Python 必须的文件 (只复制 apt 源和 python 源码)
COPY source/source.list /tmp/source/source.list
COPY source/Python-3.10.12.tgz /tmp/source/Python-3.10.12.tgz

# 2. 安装编译工具 -> 编译 Python -> 清理编译工具以减小体积
RUN cp /tmp/source/source.list /etc/apt/sources.list && \
    apt-get update && \
    # 安装编译 Python 3.10 所需的底层依赖 (这是编译必须的，不能省)
    apt-get install -y --no-install-recommends \
    build-essential zlib1g-dev libncurses5-dev libncursesw5-dev \
    libsqlite3-dev liblzma-dev tk-dev uuid-dev libgdbm-dev libnss3-dev libssl-dev \
    libreadline-dev libffi-dev libbz2-dev wget curl && \
    cd /tmp/source && \
    tar -xzf Python-3.10.12.tgz && \
    cd Python-3.10.12 && \
    ./configure --enable-optimizations --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib" && \
    make -j$(nproc) && \
    make altinstall && \
    ldconfig && \
    # === 清理阶段 ===
    cd / && \
    rm -rf /tmp/source && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Dockerfile.app，用于运行APP

# 使用刚才构建的基础镜像
FROM bm1684-python310:v1.0

ENV DEBIAN_FRONTEND=noninteractive \
    TZ=Asia/Shanghai \
    PIP_NO_CACHE_DIR=1 \
    PIP_INDEX_URL=http://rdsource.tp-link.com.cn/pypi/simple/ \
    PIP_TRUSTED_HOST=rdsource.tp-link.com.cn

USER root

# 1. 复制运行时需要的文件: Sophon驱动 + Whl包 + 源文件
COPY source /tmp/source
COPY whl /tmp/whl

RUN cp /tmp/source/source.list /etc/apt/sources.list && \
    apt-get update && \
    # === 安装运行时系统库 ===
    apt-get install -y --no-install-recommends \
    libgomp1 libopenblas-dev \
    # === 安装 Sophon 及中间件 ===
    /tmp/source/sophon-libsophon_0.5.1-LTS_arm64.deb \
    /tmp/source/sophon-libsophon-dev_0.5.1-LTS_arm64.deb \
    /tmp/source/sophon-mw-sophon-ffmpeg_0.11.0_arm64.deb \
    /tmp/source/sophon-mw-sophon-ffmpeg-dev_0.11.0_arm64.deb \
    /tmp/source/sophon-mw-sophon-opencv_0.11.0_arm64.deb \
    /tmp/source/sophon-mw-sophon-opencv-dev_0.11.0_arm64.deb && \
    # === 清理 ===
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# 3. 安装 Python 业务包
RUN pip3.10 install --no-cache-dir --upgrade pip && \
    pip3.10 install --no-cache-dir -r /tmp/whl/requirements.txt && \
    pip3.10 install --no-cache-dir /tmp/whl/silk2.tools.logger-1.0.2-py3-none-any.whl && \
    pip3.10 install --no-cache-dir /tmp/whl/sophon_arm-3.10.7-py3-none-any.whl && \
    # 清理所有临时文件
    rm -rf /root/.cache/pip /tmp/whl /tmp/source

CMD ["/bin/bash"]

requirements.txt版本

aiofiles==24.1.0
annotated-doc==0.0.4
annotated-types==0.7.0
anyio==4.12.0
av==12.3.0
brotli==1.2.0
certifi==2025.11.12
charset-normalizer==3.4.4
click==8.3.1
dfss==1.10.4
exceptiongroup==1.3.1
fastapi==0.123.3
ffmpy==1.0.0
filelock==3.20.0
fsspec==2025.10.0
gradio==6.0.2
gradio_client==2.0.1
groovy==0.1.2
h11==0.16.0
hf-xet==1.2.0
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.36.0
idna==3.11
Jinja2==3.1.6
Logbook==1.9.2
markdown-it-py==4.0.0
MarkupSafe==3.0.3
mdurl==0.1.2
mpmath==1.3.0
networkx==3.4.2
numpy==1.24.4
opencv-python-headless==4.11.0.86
orjson==3.11.4
packaging==24.2
pandas==2.3.3
pillow==12.0.0
psutil==5.9.1
pybind11==3.0.1
pydantic==2.12.4
pydantic_core==2.41.5
pydub==0.25.1
Pygments==2.19.2
python-dateutil==2.9.0.post0
python-multipart==0.0.20
pytz==2025.2
PyYAML==6.0.3
qwen-vl-utils==0.0.14
regex==2025.11.3
requests==2.32.3
rich==14.2.0
safehttpx==0.1.7
safetensors==0.7.0
semantic-version==2.10.0
shellingham==1.5.4
six==1.17.0
starlette==0.50.0
sympy==1.14.0
tokenizers==0.22.1
tomlkit==0.13.3
torch==2.4.1
torchaudio==0.13.1
torchvision==0.19.1
tqdm==4.67.1
transformers==4.57.1
typer==0.20.0
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2025.2
urllib3==2.5.0
uvicorn==0.38.0

容器启动命令

# 使用脚本 https://github.com/sophgo/sophon-demo/blob/release/sample/Qwen3-VL/
docker run -it --rm --privileged=true -p 8000:8000 -v /data/Qwen3-VL/:/data/Qwen3-VL/   -w /data/Qwen3-VL/python   bm1684-app:v1.0   python3.10 -m qwen3_vl -m /data/Qwen3-VL/models/BM1684X/qwen3-vl-4b-instruct_w4bf16_seq2048_bm1684x_1dev_20251026_141347.bmodel   -c /data/Qwen3-VL/config

[bullet46]

最小触发例程

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#include "bm_jpeg_interface.h"
#include "bmcv_api_ext_c.h"
#include "bmlib_runtime.h"

static int s_sdk_init = 0;
static bm_handle_t s_bm_handle;

void Sophon_sdk_init(void)
{
    bm_status_t bm_status = 0;

    if (s_sdk_init)
    {
        printf("bm_dev_request has been init!!\n");
        return;
    }

    bm_status = bm_dev_request(&s_bm_handle, 0);
    if (bm_status)
    {
        printf("bm_dev_request failed!! bm_status: %d\n", bm_status);
        return ;
    }

    // memset(g_sophon_decoder, 0, sizeof(g_sophon_decoder));
    // for (i = 0; i < MAX_DECODER_NUM; i++)
    // {
    //     pthread_mutex_init(&g_sophon_decoder[i].yuv_status_mutex, NULL);
    //     g_sophon_decoder[i].player_id = -1;
    // }

    // g_thread_running = TRUE;
    // if (0 != pthread_create(&s_sws_pthread, NULL, scale_yuv_thread, NULL))
    // {
    //     SOPHON_ERR_PRINT("create scale_yuv_thread failed.\n");
    //     goto _init_failed;
    // }

    // s_dec_yuv.data = (unsigned char *)malloc(JPEG_DEC_YUV_BUFFER_SIZE);
    // if (s_dec_yuv.data == NULL)
    // {
    //     SOPHON_ERR_PRINT("malloc failed! size: %d\n", JPEG_DEC_YUV_BUFFER_SIZE);
    //     goto _init_failed;
    // }
    // s_dec_yuv.size = JPEG_DEC_YUV_BUFFER_SIZE;
    s_sdk_init = 1;
    printf("bm_dev_request success!! \n");
    return;
}

int Sophon_jpeg_encode(void *p_input, int width, int height, int Qfactor, int fmt, void *p_output, unsigned int *jpg_size)
{
    int ret = 0;
    size_t jpeg_size = 0;
    int q_factor = (int)Qfactor;
    int image_num = 1;
    bm_image src = {0};
    bm_status_t bm_status;
    bm_image_format_ext format = FORMAT_NV12;
    void* jpeg_data = NULL;
    unsigned char *in_ptr[2] = {p_input, p_input + height * width};

    if (NULL == p_input || NULL == p_output || NULL == jpg_size || !s_sdk_init)
    {
        printf("Invalid param !!\n");
        return -1;
    }

    bm_status = bm_image_create(s_bm_handle, height, width, format, DATA_TYPE_EXT_1N_BYTE, &src, NULL);
    if (bm_status)
    {
        printf("bm_image_create failed!!  width*height[%d * %d] bm_status: %d\n", width, height, bm_status);
        return -1;
    }
    bm_status = bm_image_alloc_dev_mem(src, BMCV_HEAP_ANY);
    if (bm_status)
    {
        printf("bm_image_alloc_dev_mem failed!! bm_status: %d\n", bm_status);
        ret = -1;
        goto _destroy_image;
    }
    bm_status = bm_image_copy_host_to_device(src, (void**)in_ptr);
    if (bm_status)
    {
        printf("bm_image_copy_host_to_device failed!! bm_status: %d\n", bm_status);
        ret = -1;
        goto _destroy_image;
    }

    bm_status = bmcv_image_jpeg_enc(s_bm_handle, image_num, &src, &jpeg_data, &jpeg_size, q_factor, 0);
    if (bm_status)
    {
        printf("bmcv_image_jpeg_enc failed!! bm_status: %d\n", bm_status);
        ret = -1;
        goto _destroy_image;
    }
    memcpy(p_output, jpeg_data, jpeg_size);
    *jpg_size = (unsigned int)jpeg_size;

_destroy_image:
    if (jpeg_data)
    {
        free(jpeg_data);
    }
    bm_image_destroy(src);
    return ret;
}

int main()
{
    printf("=============== JPEG Encoding Demo ===============\n");
    
    // 1. 初始化SDK
    Sophon_sdk_init();
    if (!s_sdk_init) {
        printf("SDK initialization failed!\n");
        return -1;
    }
    
    // 2. 设置参数
    const char* input_yuv_file = "/data/test.yuv";
    const char* output_jpg_file = "/tmp/test_output.jpg";
    int width = 512;
    int height = 288;
    int qfactor = 80;  // JPEG质量因子
    int fmt = 0;       // 格式参数（根据需要调整）
    
    // 3. 分配内存
    size_t yuv_size = width * height * 3 / 2;  // NV12格式大小
    size_t jpg_buffer_size = width * height * 3; // JPEG缓冲区（预留足够空间）
    
    unsigned char* yuv_data = (unsigned char*)malloc(yuv_size);
    unsigned char* jpg_data = (unsigned char*)malloc(jpg_buffer_size);
    unsigned int jpg_size = 0;
    
    if (!yuv_data || !jpg_data) {
        printf("Memory allocation failed!\n");
        if (yuv_data) free(yuv_data);
        if (jpg_data) free(jpg_data);
        return -1;
    }
    
    // 4. 读取YUV文件
    FILE* fp = fopen(input_yuv_file, "rb");
    if (!fp) {
        printf("Failed to open input file: %s\n", input_yuv_file);
        free(yuv_data);
        free(jpg_data);
        return -1;
    }
    
    size_t read_size = fread(yuv_data, 1, yuv_size, fp);
    fclose(fp);
    
    if (read_size != yuv_size) {
        printf("Failed to read complete YUV data. Expected: %zu, Read: %zu\n", yuv_size, read_size);
        free(yuv_data);
        free(jpg_data);
        return -1;
    }
    
    printf("Successfully read YUV data from %s, size: %zu bytes\n", input_yuv_file, read_size);
    
    // 5. 调用JPEG编码函数
    printf("Starting JPEG encoding...\n");
    int ret = Sophon_jpeg_encode(yuv_data, width, height, qfactor, fmt, jpg_data, &jpg_size);
    
    if (ret == 0 && jpg_size > 0) {
        printf("JPEG encoding successful! Output size: %u bytes\n", jpg_size);
        
        // 6. 保存JPEG文件
        FILE* out_fp = fopen(output_jpg_file, "wb");
        if (out_fp) {
            size_t written = fwrite(jpg_data, 1, jpg_size, out_fp);
            fclose(out_fp);
            
            if (written == jpg_size) {
                printf("JPEG file saved successfully to %s\n", output_jpg_file);
            } else {
                printf("Failed to write complete JPEG file. Written: %zu, Expected: %u\n", written, jpg_size);
                ret = -1;
            }
        } else {
            printf("Failed to create output file: %s\n", output_jpg_file);
            ret = -1;
        }
    } else {
        printf("JPEG encoding failed! Return code: %d\n", ret);
    }
    
    // 7. 清理内存
    free(yuv_data);
    free(jpg_data);
    
    printf("=============== Demo Completed ===============\n");
    return ret;
}

例程使用方法

1. 将DEMO与test.yuv拷贝至/data目录下。
2. 授予权限，并运行DEMO。
(base) linaro@bm1684:/data$ ./DEMO
=============== JPEG Encoding Demo ===============
bm_dev_request success!!
Successfully read YUV data from /data/test.yuv, size: 221184 bytes
Starting JPEG encoding...
Open /dev/jpu successfully, device index = 0, jpu fd = 12, vpp fd = 13
JPEG encoding successful! Output size: 18632 bytes
JPEG file saved successfully to /tmp/test_output.jpg
=============== Demo Completed ===============
如出现success，则证明转JPEG成功

受权限管控，YUV的图片附件我这边无法上传，请您那边试试其它方法换一张图片。

在不触发这个问题前，Docker推理一直是正常的，理论上来说Docker容器级隔离，异常应该更轻点。
但测试下来样机进程确实有影响Docker容器的风险，在2台样机上都复现了此问题。
可能会影响我们这边的容器化评估，请帮忙排查，谢谢。

[bullet46]

另补测试，确实和Docker相关，使用如下例程
https://github.com/sophgo/LLM-TPU/tree/main/models/Qwen3_VL

在推理中调用./DEMO仍然会持续输出token id 0

[bullet46]

使用非特权模式，仅映射bm-tpu0和bmdev-ctl，也会出现问题
docker run -it --rm --device /dev/bm-tpu0 --device /dev/bmdev-ctl --ipc=host -p 8000:8000 -v /data/Qwen3-VL/:/data/Qwen3-VL/ -w /data/Qwen3-VL/Qwen3_VL/cpp_demo my-new-image:v2.0 ./pipeline -m /data/Qwen3-VL/models/BM1684X/qwen3-vl-4b-instruct_w4bf16_seq2048_bm1684x_1dev_20251026_141347.bmodel -c /data/Qwen3-VL/config

[bullet46]

收到供应商回复，使用-v映射opt、etc等路径至容器内，可以解决此问题。
实测加了这部分映射后容器可正常工作，后续压力测试，反馈结果。

docker run -it --rm \
--privileged=true \
-v /opt/sophon:/opt/sophon \
-v /dev:/dev \
-v /etc/profile.d:/etc/profile.d \
-v /etc/profile:/etc/profile \
-v /etc/ld.so.conf.d:/etc/ld.so.conf.d \
-p 8000:8000 \
-v /data/Qwen3-VL/:/data/Qwen3-VL/ \
-w /data/Qwen3-VL/python \
bm1684-app:v1.0 \
python3.10 -m qwen3_vl -m /data/Qwen3-VL/models/BM1684X/qwen3-vl-4b-instruct_w4bf16_seq2048_bm1684x_1dev_20251026_141347.bmodel -c /data/Qwen3-VL/config

原因猜测：
映射后会把Docker中安装的libsophon覆盖了，直接使用样机的libsopho，能避免一些环境冲突、设备抢占问题：

(base) linaro@bm1684:/opt/sophon$ ls
libsophon-0.5.1  libsophon-current  sophon-ffmpeg-latest  sophon-ffmpeg_0.10.0  sophon-opencv-latest  sophon-opencv_0.10.0

不过这样无法实现完全的容器化隔离，但由于libsophon更新频率比较低，可绑定固件版本来发布Docker镜像。