Error Messages "section_type: PCIe error" are observed when set "tp=2" on model "DeepSeek-R1-Distill-Qwen-7B"

[jetang0202]

OEM customer would like to know why this error message came up with loading model "DeepSeek-R1-Distill-Qwen-7B" and set "tp=2" as below. And how to fix this problem.

---

[16004.940106] {468}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[16004.940109] {468}[Hardware Error]: It has been corrected by h/w and requires no further action
[16004.940111] {468}[Hardware Error]: event severity: corrected
[16004.940112] {468}[Hardware Error]: Error 0, type: corrected
[16004.940114] {468}[Hardware Error]: section_type: PCIe error
[16004.940115] {468}[Hardware Error]: port_type: 5, upstream switch port
[16004.940117] {468}[Hardware Error]: version: 3.0
[16004.940118] {468}[Hardware Error]: command: 0x0147, status: 0x0010
[16004.940120] {468}[Hardware Error]: device_id: 0000:1a:00.0
[16004.940121] {468}[Hardware Error]: slot: 0
[16004.940123] {468}[Hardware Error]: secondary_bus: 0x1b
[16004.940124] {468}[Hardware Error]: vendor_id: 0x11f8, device_id: 0x5000
[16004.940125] {468}[Hardware Error]: class_code: 060400
[16004.940126] {468}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x000b
[16004.940128] {468}[Hardware Error]: aer_cor_status: 0x00002000, aer_cor_mask: 0x00000000
[16004.940129] {468}[Hardware Error]: aer_uncor_status: 0x00100000, aer_uncor_mask: 0x00180000
[16004.940131] {468}[Hardware Error]: aer_uncor_severity: 0x00463030
[16004.940132] {468}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000
.
.
.

---

According to triage result, if same model with setting as tp=1 or tp=4 then no error message and model could execute without problem.

[hzdzkjdxyjs]

兄弟，你提供的信息也太少了，启动的时候几块GPU，型号是啥，驱动是啥，docker是啥，启动的指令是啥，你不说完全，研发没法复现的

[jetang0202]

此問題只發生在客戶獨特設計，單張BMG卡上，有2顆P5- (minus) 的GPUs。並且搭配PCIe 擴充卡，以實現多卡多GPUs高計算流量設計(up to 8卡，16GPUs)。
但是問題目前是發生在，單卡(w/ 2 GPUs) 再加上PCIe 擴充卡的情況下。所以沒有這樣的硬件組合下，問題無法複現。詳細軟件內容我再請客戶提供。

完整錯誤訊息如下:
[16004.940106] {468}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
[16004.940109] {468}[Hardware Error]: It has been corrected by h/w and requires no further action
[16004.940111] {468}[Hardware Error]: event severity: corrected
[16004.940112] {468}[Hardware Error]: Error 0, type: corrected
[16004.940114] {468}[Hardware Error]: section_type: PCIe error
[16004.940115] {468}[Hardware Error]: port_type: 5, upstream switch port
[16004.940117] {468}[Hardware Error]: version: 3.0
[16004.940118] {468}[Hardware Error]: command: 0x0147, status: 0x0010
[16004.940120] {468}[Hardware Error]: device_id: 0000:1a:00.0
[16004.940121] {468}[Hardware Error]: slot: 0
[16004.940123] {468}[Hardware Error]: secondary_bus: 0x1b
[16004.940124] {468}[Hardware Error]: vendor_id: 0x11f8, device_id: 0x5000
[16004.940125] {468}[Hardware Error]: class_code: 060400
[16004.940126] {468}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x000b
[16004.940128] {468}[Hardware Error]: aer_cor_status: 0x00002000, aer_cor_mask: 0x00000000
[16004.940129] {468}[Hardware Error]: aer_uncor_status: 0x00100000, aer_uncor_mask: 0x00180000
[16004.940131] {468}[Hardware Error]: aer_uncor_severity: 0x00463030
[16004.940132] {468}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000
[16004.940134] {468}[Hardware Error]: Error 1, type: corrected
[16004.940135] {468}[Hardware Error]: section_type: PCIe error
[16004.940136] {468}[Hardware Error]: port_type: 6, downstream switch port
[16004.940137] {468}[Hardware Error]: version: 3.0
[16004.940139] {468}[Hardware Error]: command: 0x0547, status: 0x0010
[16004.940140] {468}[Hardware Error]: device_id: 0000:1b:00.0
[16004.940141] {468}[Hardware Error]: slot: 1
[16004.940142] {468}[Hardware Error]: secondary_bus: 0x1c
[16004.940144] {468}[Hardware Error]: vendor_id: 0x11f8, device_id: 0x5000
[16004.940145] {468}[Hardware Error]: class_code: 060400
[16004.940146] {468}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x000b
[16004.940147] {468}[Hardware Error]: aer_cor_status: 0x00002000, aer_cor_mask: 0x00000000
[16004.940149] {468}[Hardware Error]: aer_uncor_status: 0x00100000, aer_uncor_mask: 0x00180020
[16004.940150] {468}[Hardware Error]: aer_uncor_severity: 0x00463010
[16004.940151] {468}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000
[16004.940152] {468}[Hardware Error]: Error 2, type: corrected
[16004.940154] {468}[Hardware Error]: section_type: PCIe error
[16004.940154] {468}[Hardware Error]: port_type: 6, downstream switch port
[16004.940156] {468}[Hardware Error]: version: 3.0
[16004.940157] {468}[Hardware Error]: command: 0x0547, status: 0x0010
[16004.940158] {468}[Hardware Error]: device_id: 0000:1b:01.0
[16004.940159] {468}[Hardware Error]: slot: 2
[16004.940160] {468}[Hardware Error]: secondary_bus: 0x20
[16004.940161] {468}[Hardware Error]: vendor_id: 0x11f8, device_id: 0x5000
[16004.940162] {468}[Hardware Error]: class_code: 060400
[16004.940164] {468}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0003
[16004.940165] {468}[Hardware Error]: aer_cor_status: 0x00003000, aer_cor_mask: 0x00000000
[16004.940166] {468}[Hardware Error]: aer_uncor_status: 0x00100000, aer_uncor_mask: 0x00180020
[16004.940167] {468}[Hardware Error]: aer_uncor_severity: 0x00463010
[16004.940168] {468}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000
[16004.940170] {468}[Hardware Error]: Error 3, type: corrected
[16004.940171] {468}[Hardware Error]: section_type: PCIe error
[16004.940172] {468}[Hardware Error]: port_type: 6, downstream switch port
[16004.940173] {468}[Hardware Error]: version: 3.0
[16004.940174] {468}[Hardware Error]: command: 0x0147, status: 0x0010
[16004.940175] {468}[Hardware Error]: device_id: 0000:1d:02.0
[16004.940177] {468}[Hardware Error]: slot: 0
[16004.940178] {468}[Hardware Error]: secondary_bus: 0x1f
[16004.940179] {468}[Hardware Error]: vendor_id: 0x8086, device_id: 0xe2f1
[16004.940180] {468}[Hardware Error]: class_code: 060400
[16004.940181] {468}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0003
[16004.940182] {468}[Hardware Error]: aer_cor_status: 0x00002000, aer_cor_mask: 0x00000000
[16004.940183] {468}[Hardware Error]: aer_uncor_status: 0x00100000, aer_uncor_mask: 0x00100000
[16004.940185] {468}[Hardware Error]: aer_uncor_severity: 0x00441010
[16004.940186] {468}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000
[16004.940187] {468}[Hardware Error]: Error 4, type: corrected
[16004.940188] {468}[Hardware Error]: section_type: PCIe error
[16004.940189] {468}[Hardware Error]: port_type: 5, upstream switch port
[16004.940190] {468}[Hardware Error]: version: 3.0
[16004.940192] {468}[Hardware Error]: command: 0x0147, status: 0x0011
[16004.940193] {468}[Hardware Error]: device_id: 0000:20:00.0
[16004.940194] {468}[Hardware Error]: slot: 0
[16004.940195] {468}[Hardware Error]: secondary_bus: 0x21
[16004.940196] {468}[Hardware Error]: vendor_id: 0x8086, device_id: 0xe2ff
[16004.940197] {468}[Hardware Error]: class_code: 060400
[16004.940198] {468}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0003
[16004.940200] {468}[Hardware Error]: aer_cor_status: 0x00000001, aer_cor_mask: 0x00000000
[16004.940201] {468}[Hardware Error]: aer_uncor_status: 0x00000000, aer_uncor_mask: 0x00180000
[16004.940202] {468}[Hardware Error]: aer_uncor_severity: 0x00463030
[16004.940203] {468}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000
[16004.940205] {468}[Hardware Error]: Error 5, type: corrected
[16004.940206] {468}[Hardware Error]: section_type: PCIe error
[16004.940207] {468}[Hardware Error]: port_type: 6, downstream switch port
[16004.940208] {468}[Hardware Error]: version: 3.0
[16004.940209] {468}[Hardware Error]: command: 0x0147, status: 0x0010
[16004.940210] {468}[Hardware Error]: device_id: 0000:21:02.0
[16004.940212] {468}[Hardware Error]: slot: 0
[16004.940213] {468}[Hardware Error]: secondary_bus: 0x23
[16004.940214] {468}[Hardware Error]: vendor_id: 0x8086, device_id: 0xe2f1
[16004.940215] {468}[Hardware Error]: class_code: 060400
[16004.940216] {468}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0003
[16004.940217] {468}[Hardware Error]: aer_cor_status: 0x00002000, aer_cor_mask: 0x00000000
[16004.940219] {468}[Hardware Error]: aer_uncor_status: 0x00100000, aer_uncor_mask: 0x00100000
[16004.940220] {468}[Hardware Error]: aer_uncor_severity: 0x00441010
[16004.940221] {468}[Hardware Error]: TLP Header: 00000000 00000000 00000000 00000000
[16004.940249] pcieport 0000:1a:00.0: AER: aer_status: 0x00002000, aer_mask: 0x00000000
[16004.940252] pcieport 0000:1a:00.0: [13] NonFatalErr
[16004.940255] pcieport 0000:1a:00.0: AER: aer_layer=Transaction Layer, aer_agent=Receiver ID
[16004.940261] pcieport 0000:1b:00.0: AER: aer_status: 0x00002000, aer_mask: 0x00000000
[16004.940263] pcieport 0000:1b:00.0: [13] NonFatalErr
[16004.940265] pcieport 0000:1b:00.0: AER: aer_layer=Transaction Layer, aer_agent=Receiver ID
[16004.940271] pcieport 0000:1b:01.0: AER: aer_status: 0x00003000, aer_mask: 0x00000000
[16004.940273] pcieport 0000:1b:01.0: [12] Timeout
[16004.940275] pcieport 0000:1b:01.0: [13] NonFatalErr
[16004.940276] pcieport 0000:1b:01.0: AER: aer_layer=Data Link Layer, aer_agent=Transmitter ID
[16004.940283] pcieport 0000:1d:02.0: AER: aer_status: 0x00002000, aer_mask: 0x00000000
[16004.940285] pcieport 0000:1d:02.0: [13] NonFatalErr
[16004.940286] pcieport 0000:1d:02.0: AER: aer_layer=Transaction Layer, aer_agent=Receiver ID
[16004.940293] pcieport 0000:20:00.0: AER: aer_status: 0x00000001, aer_mask: 0x00000000
[16004.940295] pcieport 0000:20:00.0: [ 0] RxErr (First)
[16004.940297] pcieport 0000:20:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
[16004.940304] pcieport 0000:21:02.0: AER: aer_status: 0x00002000, aer_mask: 0x00000000
[16004.940306] pcieport 0000:21:02.0: [13] NonFatalErr
[16004.940308] pcieport 0000:21:02.0: AER: aer_layer=Transaction Layer, aer_agent=Receiver ID

[hzdzkjdxyjs]

兄弟，首先我不是Intel的哈，我只是小菜鸟哈，B60 48G的卡，我们也有，我们也用过，没有这种报错，但是可以和你分享一些细节

1.bios安装，在使用B60前，确定您的bios支持B60Pro的设计，因为他是双芯卡，在启动的时候，您需要在bios中设置几个参数，这非常重要
开启Re-SizeBAR

将Pcie的通道改为x8x8，if bios支持，Auto也是可以的，注意服务器一般有两个sock，请把两边插了显卡的Pcie通道改为x8x8

2.安装驱动和linux，安装标准版的linux25.04就可以了，再安装对应的GPU驱动，安装完后请您运行xpu-smi discovery确认您的卡的数量，因为B60Pro是双芯卡，实际上的数量应该是*2的
例如我装了一张B60 Pro xpu-smi下应该是两张（这非常重要，有可能您显示的就只是一张，因为您没有改bios）

3.安装对应的Docker，您要使用本项目的docker来启动服务
docker pull intel/llm-scaler-vllm:1.1-preview

4.导入环境变量正式启动服务
例如脚本
sudo docker run -td
--privileged
--net=host
--device=/dev/dri
--name=DeepSeek-R1-Distill-Qwen-7B
-v /root/models:/llm/models/
--shm-size="32g"
--entrypoint /bin/bash
intel/llm-scaler-vllm:1.1-preview

ZE_AFFINITY_MASK也是个非常重要的参数，就像cuda类 device=“cude：0，1”这是指定GPU的，B60Pro是双芯卡，所以0，1分别代表他的两张芯片

export ZE_AFFINITY_MASK=0,1
export TORCH_LLM_ALLREDUCE=1
export VLLM_USE_V1=1
export CCL_ZE_IPC_EXCHANGE=pidfd
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
python3 -m vllm.entrypoints.openai.api_server
--model /llm/models/DeepSeek-R1-Distill-Qwen-7B
--served-model-name DeepSeek-R1-Distill-Qwen-7B
--dtype=float16
--enforce-eager
--port 8000
--host 0.0.0.0
--trust-remote-code
--gpu-memory-util=0.9
--no-enable-prefix-caching
--max-num-batched-tokens=8192
--disable-log-requests
--max-model-len=8192
--block-size 64
--quantization FP8
-tp=2