deepseek 4卡启动报错

[OliviaSnail]

启动命令：CUDA_VISIBLE_DEVICES=0,1,2,3 grpst start server.mar --inference_conf=conf/inference_deepseek-r1-distill-qwen.yml --mpi_np=4

报错信息：
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: len <= remaining_buffer_size (/workspace/tensorrt_llm/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQAImplJIT/cubinObj.cpp:34)
1 0x7fdd6f903370 tensorrt_llm::common::throwRuntimeError(char const*, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 95
2 0x7fdd106eb24c tensorrt_llm::kernels::jit::CubinObj::CubinObj(void const*, unsigned long) + 412
3 0x7fdd107017d4 tensorrt_llm::kernels::jit::CubinObjRegistryTemplate<tensorrt_llm::kernels::XQAKernelFullHashKey, tensorrt_llm::kernels::XQAKernelFullHasher>::CubinObjRegistryTemplate(void const*, unsigned long) + 292
4 0x7fdd10701132 tensorrt_llm::kernels::DecoderXQARunner::Resource::Resource(void const*, unsigned long) + 50
5 0x7fdd0405e149 tensorrt_llm::plugins::GPTAttentionPluginCommon::GPTAttentionPluginCommon(void const*, unsigned long) + 1193
6 0x7fdd04095232 tensorrt_llm::plugins::GPTAttentionPlugin::GPTAttentionPlugin(void const*, unsigned long) + 18
7 0x7fdd040952b2 tensorrt_llm::plugins::GPTAttentionPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 50
8 0x7fdcf5aa6b5b /usr/local/tensorrt/lib/libnvinfer.so.10(+0x11deb5b) [0x7fdcf5aa6b5b]
9 0x7fdcf5aa345e /usr/local/tensorrt/lib/libnvinfer.so.10(+0x11db45e) [0x7fdcf5aa345e]
10 0x7fdcf59d62b7 /usr/local/tensorrt/lib/libnvinfer.so.10(+0x110e2b7) [0x7fdcf59d62b7]
11 0x7fdcf59d4e6a /usr/local/tensorrt/lib/libnvinfer.so.10(+0x110ce6a) [0x7fdcf59d4e6a]
12 0x7fdcf59eca77 /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1124a77) [0x7fdcf59eca77]
13 0x7fdcf59f05b6 /usr/local/tensorrt/lib/libnvinfer.so.10(+0x11285b6) [0x7fdcf59f05b6]
14 0x7fdcf59f0b06 /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1128b06) [0x7fdcf59f0b06]
15 0x7fdcf5a27fc7 /usr/local/tensorrt/lib/libnvinfer.so.10(+0x115ffc7) [0x7fdcf5a27fc7]
16 0x7fdcf5a28bd8 /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1160bd8) [0x7fdcf5a28bd8]
17 0x7fdcf5a28cdb /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1160cdb) [0x7fdcf5a28cdb]
18 0x7fdd1252f275 tensorrt_llm::runtime::TllmRuntime::TllmRuntime(tensorrt_llm::runtime::RawEngine const&, nvinfer1::ILogger*, float, bool) + 1413
19 0x7fdd1298d428 tensorrt_llm::batch_manager::TrtGptModelInflightBatching::TrtGptModelInflightBatching(std::shared_ptrnvinfer1::ILogger, tensorrt_llm::runtime::ModelConfig const&, tensorrt_llm::runtime::WorldConfig const&, tensorrt_llm::runtime::RawEngine const&, bool, tensorrt_llm::batch_manager::TrtGptModelOptionalParams const&) + 1304
20 0x7fdd1291151e tensorrt_llm::batch_manager::TrtGptModelFactory::create(tensorrt_llm::runtime::RawEngine const&, tensorrt_llm::runtime::ModelConfig const&, tensorrt_llm::runtime::WorldConfig const&, tensorrt_llm::batch_manager::TrtGptModelType, tensorrt_llm::batch_manager::TrtGptModelOptionalParams const&) + 526
21 0x7fdd12a28029 tensorrt_llm::executor::Executor::Impl::createModel(tensorrt_llm::runtime::RawEngine const&, tensorrt_llm::runtime::ModelConfig const&, tensorrt_llm::runtime::WorldConfig const&, tensorrt_llm::executor::ExecutorConfig const&) + 185
22 0x7fdd12a286bd tensorrt_llm::executor::Executor::Impl::loadModel(std::optionalstd::filesystem::__cxx11::path const&, std::optional<std::basic_string_view<unsigned char, std::char_traits > > const&, tensorrt_llm::runtime::GptJsonConfig const&, tensorrt_llm::executor::ExecutorConfig const&, bool, std::optional<std::map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, tensorrt_llm::executor::Tensor, std::less<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, tensorrt_llm::executor::Tensor> > > > const&) + 1229
23 0x7fdd12a2990a tensorrt_llm::executor::Executor::Impl::Impl(std::filesystem::__cxx11::path const&, std::optionalstd::filesystem::__cxx11::path const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&) + 2474
24 0x7fdd12a0f757 tensorrt_llm::executor::Executor::Executor(std::filesystem::__cxx11::path const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&) + 87
25 0x7fdd6f977bfa std::__detail::_MakeUniq<tensorrt_llm::executor::Executor>::__single_object std::make_unique<tensorrt_llm::executor::Executor, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&, tensorrt_llm::executor::ModelType&, tensorrt_llm::executor::ExecutorConfig&>(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&, tensorrt_llm::executor::ModelType&, tensorrt_llm::executor::ExecutorConfig&) + 154
26 0x7fdd6f974966 netease::grps::TrtLlmModelInstance::TrtLlmModelInstance(netease::grps::TrtLlmModelState*, netease::grps::LLMStyler*, netease::grps::MultiInstanceTokenizer*, netease::grps::VIT*) + 3542
27 0x7fdd73427eae netease::grps::TrtllmInferer::Load() + 4286
28 0x55b7cafefe2e ./bin/grps_server(+0xf2e2e) [0x55b7cafefe2e]
29 0x55b7caff1e2c ./bin/grps_server(+0xf4e2c) [0x55b7caff1e2c]
30 0x55b7caf82a7a ./bin/grps_server(+0x85a7a) [0x55b7caf82a7a]
31 0x7fdd6ff9a1ca /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7fdd6ff9a1ca]
32 0x7fdd6ff9a28b __libc_start_main + 139
33 0x55b7caf87865 ./bin/grps_server(+0x8a865) [0x55b7caf87865]

模型在TensorRT-LLM单独测过没有问题，麻烦看下。

[zhaocc1106]

模型在TensorRT-LLM单独测过没有问题，麻烦看下。

辛苦发一下卡的型号，以及单独测试没问题的TensorRT-LLM版本号~。

[zhaocc1106]

另外引擎文件是否在新的环境中重新进行了构建？

[OliviaSnail]

a30 8卡，单独测试没问题的是v.0.17.0哈。模型没有在新环境里构建，是在TensorRT-LLM的docker里构建的。

[zhaocc1106]

a30 8卡，单独测试没问题的是v.0.17.0哈。模型没有在新环境里构建，是在TensorRT-LLM的docker里构建的。

当前master分支的镜像是v0.16.0哈，在新环境里面重新构建一下试试？

[OliviaSnail]

在新环境重新构建了模型，执行run.py结果也返回正常。启动依然报错，是否与mpi有关？
堆栈信息：
==== backtrace (tid: 105372) ====
0 /opt/hpcx/ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7fdb1228f614]
1 /opt/hpcx/ucx/lib/libucs.so.0(+0x3680c) [0x7fdb1228f80c]
2 /opt/hpcx/ucx/lib/libucs.so.0(+0x36a48) [0x7fdb1228fa48]
3 /usr/local/lib/libjemalloc.so(+0x947ec) [0x7fdc184467ec]
4 /usr/local/lib/libjemalloc.so(+0x24a00) [0x7fdc183d6a00]
5 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(_ZN12tensorrt_llm7plugins18GPTAttentionPluginD0Ev+0xcd) [0x7fdba8e9793d]
6 /usr/local/tensorrt/lib/libnvinfer.so.10(+0x11de2a2) [0x7fdb9a8a62a2]
7 /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1122be2) [0x7fdb9a7eabe2]
8 /usr/local/tensorrt/lib/libnvinfer.so.10(+0xbeb21a) [0x7fdb9a2b321a]
9 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNSt23_Sp_counted_ptr_inplaceIN12tensorrt_llm7runtime11TllmRuntimeESaIvELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x282) [0x7fdbb72dc3d2]
10 ./bin/grps_server(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x57) [0x557d7f29a9b7]
11 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(+0x85f410) [0x7fdbb533b410]
12 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager18TrtGptModelFactory6createERKNS_7runtime9RawEngineERKNS2_11ModelConfigERKNS2_11WorldConfigENS0_15TrtGptModelTypeERKNS0_25TrtGptModelOptionalParamsE+0x20e) [0x7fdbb771151e]
13 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm8executor8Executor4Impl11createModelERKNS_7runtime9RawEngineERKNS3_11ModelConfigERKNS3_11WorldConfigERKNS0_14ExecutorConfigE+0xb9) [0x7fdbb7828029]
14 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm8executor8Executor4Impl9loadModelERKSt8optionalINSt10filesystem7__cxx114pathEERKS3_ISt17basic_string_viewIhSt11char_traitsIhEEERKNS_7runtime13GptJsonConfigERKNS0_14ExecutorConfigEbRKS3_ISt3mapINSt7__cxx1112basic_stringIcSB_IcESaIcEEENS0_6TensorESt4lessIST_ESaISt4pairIKST_SU_EEEE+0x4cd) [0x7fdbb78286bd]
15 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm8executor8Executor4ImplC1ERKNSt10filesystem7__cxx114pathERKSt8optionalIS5_ENS0_9ModelTypeERKNS0_14ExecutorConfigE+0x9aa) [0x7fdbb782990a]
16 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm8executor8ExecutorC1ERKNSt10filesystem7__cxx114pathENS0_9ModelTypeERKNS0_14ExecutorConfigE+0x57) [0x7fdbb780f757]
17 ./lib/libgrps_tensorrtllm_common.so(ZSt11make_uniqueIN12tensorrt_llm8executor8ExecutorEJRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS1_9ModelTypeERNS1_14ExecutorConfigEEENSt8__detail9_MakeUniqIT_E15__single_objectEDpOT0+0x9a) [0x7fdc14777bfa]
18 ./lib/libgrps_tensorrtllm_common.so(_ZN7netease4grps19TrtLlmModelInstanceC1EPNS0_16TrtLlmModelStateEPNS0_9LLMStylerEPNS0_22MultiInstanceTokenizerEPNS0_3VITE+0xdd6) [0x7fdc14774966]
19 ./lib/libgrps-server-customized.so(_ZN7netease4grps13TrtllmInferer4LoadEv+0x10be) [0x7fdc18211eae]
20 ./bin/grps_server(+0xf2e2e) [0x557d7f2f5e2e]
21 ./bin/grps_server(+0xf4e2c) [0x557d7f2f7e2c]
22 ./bin/grps_server(+0x85a7a) [0x557d7f288a7a]
23 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7fdc14d9a1ca]
24 /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7fdc14d9a28b]
25 ./bin/grps_server(+0x8a865) [0x557d7f28d865]

Stack trace (most recent call last):
#23 Object "", at 0xffffffffffffffff, in

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

---

mpirun noticed that process rank 1 with PID 0 on node a30-gpu0030 exited on signal 11 (Segmentation fault).

[zhaocc1106]

抱歉我这边没有对应的卡型号，测试过A10多卡推理是正常的。
方便的话提供下下面目录的日志，我再看下：
/root/.grps/my_grps/nohup.out
/root/.grps/my_grps/logs

[OliviaSnail]

给我个邮箱我发您吧

[zhaocc1106]

给我个邮箱我发您吧

450959507@qq.com