Depoloy with Humanoid Robot Got Cuda Error

[ChuangYan2]

Hi, XMobility Dev Team,

I'm currently deploy the trt model on humanoid robot running scenaio in Isaacsim. But I got unknow error, could you please have a look what's the cause for it and how to solve it?

(Is that because the image_input not same as the one you using for carter, hmm not sure)

Thank you for your help.

Best,
Chuang

[x_mobility_navigator-1] [04/03/2025-15:47:59] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::30] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[x_mobility_navigator-1] [04/03/2025-15:47:59] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::30] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[x_mobility_navigator-1] [04/03/2025-15:47:59] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::30] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[x_mobility_navigator-1] [04/03/2025-15:47:59] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::30] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[x_mobility_navigator-1] [04/03/2025-15:47:59] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::30] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[x_mobility_navigator-1] [04/03/2025-15:47:59] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::30] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[x_mobility_navigator-1] [04/03/2025-15:47:59] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::30] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[x_mobility_navigator-1] [04/03/2025-15:47:59] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::30] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT) again, ignoring...

[nv-liuw]

@ChuangYan2 Can you provide more details on which humanoid are you using?

Likely, as noted in README, you need to update the image size in onnx_conversion script to match the robot camera.

[ChuangYan2]

@nv-liuw Thanks for your quickly reply.

• I am using Unitree G1 is Isaacsim.

I rebuild the onnx file with modifying the correct image size, but it still happens to same cuda error.

hmm, any suggestion futher?

I log following data,
[x_mobility_navigator-1] [INFO] [1743714974.704324300] [x_mobility_navigator]: Action: shape=(6,), dtype=float32, nbytes=24
[x_mobility_navigator-1] [INFO] [1743714974.704477451] [x_mobility_navigator]: History: shape=(1, 1024), dtype=float32, nbytes=4096
[x_mobility_navigator-1] [INFO] [1743714974.704629623] [x_mobility_navigator]: Sample: shape=(1, 512), dtype=float32, nbytes=2048
[x_mobility_navigator-1] [INFO] [1743714974.706573157] [x_mobility_navigator]: Image shape: (3, 720, 1280), dtype: float32
[x_mobility_navigator-1] [INFO] [1743714974.706738657] [x_mobility_navigator]: Route shape: (19, 4), dtype: float32
[x_mobility_navigator-1] [INFO] [1743714974.706887375] [x_mobility_navigator]: Speed shape: (1, 1), dtype: float32
[x_mobility_navigator-1] [ERROR] [1743714974.707190090] [x_mobility_navigator]: CUDA memory allocation failed: cuMemAlloc failed: an illegal memory access was encountered
[x_mobility_navigator-1] [ERROR] [1743714974.707621074] [x_mobility_navigator]: Inference failed: cuMemAlloc failed: an illegal memory access was encountered

thank you for your help.
Best,
Chuang

[nv-liuw]

Hi @ChuangYan2 , happy to debug together.

Can you share your USD for Unitree G1 in Isaac sim?

Also, it be helpful if you could print out the TRT engine's tensor info using something like:

engine_path = '/tmp/x_mobility.engine'
with open(engine_path, "rb") as f:
    engine_data = f.read()

# Create a TensorRT runtime
runtime = trt.Runtime(trt.Logger(trt.Logger.INFO))
engine = runtime.deserialize_cuda_engine(engine_data)

for idx in range(engine.num_io_tensors):
    name = engine.get_tensor_name(idx) 
    print(name)
    print(engine.get_tensor_dtype(name))
    print(engine.get_tensor_shape(name))