The speed of MXPredCreate is very slow for mxnetv1.0.0/V100/cuda8.0/cudnn5.0/driver410

[IamMegatron2025]

hello, I have an issue. When I use mxnetv1.0.0/V100/cuda10.0/cudnn7.5/driver410 to run a inference for my model using c++ api, the time of MXPredCreate seems to be normal, but when I use mxnetv1.0.0/V100/cuda8.0/cudnn5.0/driver410, the time of MXPredCreate is very slow. The below is test results:

case1: mxnetv1.0.0/V100/cuda10.0/cudnn7.5/driver410:  215133ms(the first run)/8227/8622/8517
case2: mxnetv1.0.0/V100/cuda8.0/cudnn5.0/driver410: 378909ms(the first run)/413323ms/415299ms/413514ms

I have two question.
first: why the MXPredCreate speed of the case2 is still very slow after the first run?
second: why the MXPredCreate speed of the case2 is two times slower than case1? Because the cuda8 is outdated？

Anyone can give some advises?

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Performance

[lanking520]

@IvyGongoogle Based on my understanding, the first call for inference would copy model info from CPU to GPU which would expect a long time to take.

I am not quite sure the reason about the second question, but keep your driver and lib up-to-date is always recommended.

• @leleamol if you have some context to put in here