Getting started on DGX Spark

[michaelfuckner]

Hi,

I have a DGX Spark running DGX OS 7.5.0 (which basically is Ubuntu 24.04 LTS) and I don't know how to get started. I created a new user, added him to sudo and docker, created a venv, installed mlc scripts via pip and then...? detect os looks fine, but I can't get any benchmark from the docs running.

(mlc) molli123@spark-7994:~$ mlcr detect,os -j
[2026-05-07 10:18:33,480 script_utils.py:  88 INFO ] - * mlcr detect,os
[2026-05-07 10:18:33,484 module.py      :1037 INFO ] -      ! load /home/molli123/MLC/repos/local/cache/detect-os_aac747bd/mlc-cached-state.json
[2026-05-07 10:18:33,485 module.py      :1901 INFO ] - {
  "return": 0,
  "env": {
    "MLC_USER_RUN_DIR": "/home/molli123",
    "MLC_HOST_OS_TYPE": "linux",
    "MLC_HOST_OS_BITS": "64",
    "MLC_HOST_OS_FLAVOR": "ubuntu",
    "MLC_HOST_OS_FLAVOR_LIKE": "debian",
    "MLC_HOST_OS_VERSION": "24.04",
    "MLC_HOST_OS_KERNEL_VERSION": "6.17.0-1014-nvidia",
    "MLC_HOST_OS_GLIBC_VERSION": "2.39",
    "MLC_HOST_OS_MACHINE": "aarch64",
    "MLC_HOST_OS_PACKAGE_MANAGER": "apt",
    "MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
    "MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
    "+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
      "/usr/local/lib/aarch64-linux-gnu",
      "/lib/aarch64-linux-gnu",
      "/usr/lib/aarch64-linux-gnu",
      "/usr/local/lib",
      "/lib",
      "/usr/lib",
      "/usr/aarch64-linux-gnu/lib"
    ],
    "MLC_HOST_PLATFORM_FLAVOR": "aarch64",
    "MLC_HOST_PYTHON_BITS": "64",
    "MLC_HOST_SYSTEM_NAME": "spark-7994",
    "MLC_HOST_FILESYSTEMS": "ext4 vfat"
  },
  "new_env": {
    "MLC_HOST_OS_TYPE": "linux",
    "MLC_HOST_OS_BITS": "64",
    "MLC_HOST_OS_FLAVOR": "ubuntu",
    "MLC_HOST_OS_FLAVOR_LIKE": "debian",
    "MLC_HOST_OS_VERSION": "24.04",
    "MLC_HOST_OS_KERNEL_VERSION": "6.17.0-1014-nvidia",
    "MLC_HOST_OS_GLIBC_VERSION": "2.39",
    "MLC_HOST_OS_MACHINE": "aarch64",
    "MLC_HOST_OS_PACKAGE_MANAGER": "apt",
    "MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
    "MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
    "+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
      "/usr/local/lib/aarch64-linux-gnu",
      "/lib/aarch64-linux-gnu",
      "/usr/lib/aarch64-linux-gnu",
      "/usr/local/lib",
      "/lib",
      "/usr/lib",
      "/usr/aarch64-linux-gnu/lib"
    ],
    "MLC_HOST_PLATFORM_FLAVOR": "aarch64",
    "MLC_HOST_PYTHON_BITS": "64",
    "MLC_HOST_SYSTEM_NAME": "spark-7994",
    "MLC_HOST_FILESYSTEMS": "ext4 vfat"
  },
  "state": {
    "os_uname_machine": "aarch64",
    "os_uname_all": "Linux spark-7994 6.17.0-1014-nvidia #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 17 19:01:40 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux"
  },
  "new_state": {
    "os_uname_machine": "aarch64",
    "os_uname_all": "Linux spark-7994 6.17.0-1014-nvidia #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 17 19:01:40 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux"
  },
  "deps": []
}
(mlc) molli123@spark-7994:~$

I believe this looks fine, but the Benchmark does not run:

(mlc) molli123@spark-7994:~$ mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only    --model=resnet50    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=valid    --device=cuda    --quiet
[2026-05-07 10:42:29,705 script_utils.py:  88 INFO ] - * mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only
[2026-05-07 10:42:29,706 script_utils.py:  88 INFO ] -   * mlcr detect,os
[2026-05-07 10:42:29,710 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/detect-os_aac747bd/mlc-cached-state.json
[2026-05-07 10:42:29,711 script_utils.py:  88 INFO ] -   * mlcr detect,cpu
[2026-05-07 10:42:29,712 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/detect-cpu_ccbf2776/mlc-cached-state.json
[2026-05-07 10:42:29,714 script_utils.py:  88 INFO ] -   * mlcr get,python3
[2026-05-07 10:42:29,715 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/get-python3_09388c6a/mlc-cached-state.json
[2026-05-07 10:42:29,719 script_utils.py:  88 INFO ] -   * mlcr get,mlcommons,inference,src
[2026-05-07 10:42:29,720 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/get-mlperf-inference-src_49bff69c/mlc-cached-state.json
[2026-05-07 10:42:29,722 script_utils.py:  88 INFO ] -   * mlcr get,sut,description
[2026-05-07 10:42:29,723 script_utils.py:  88 INFO ] -     * mlcr detect,os
[2026-05-07 10:42:29,724 module.py      :1037 INFO ] -          ! load /home/molli123/MLC/repos/local/cache/detect-os_aac747bd/mlc-cached-state.json
[2026-05-07 10:42:29,725 script_utils.py:  88 INFO ] -     * mlcr detect,cpu
[2026-05-07 10:42:29,726 module.py      :1037 INFO ] -          ! load /home/molli123/MLC/repos/local/cache/detect-cpu_ccbf2776/mlc-cached-state.json
[2026-05-07 10:42:29,727 script_utils.py:  88 INFO ] -     * mlcr get,python3
[2026-05-07 10:42:29,728 module.py      :1037 INFO ] -          ! load /home/molli123/MLC/repos/local/cache/get-python3_09388c6a/mlc-cached-state.json
[2026-05-07 10:42:29,736 script_utils.py:  88 INFO ] -     * mlcr get,compiler
[2026-05-07 10:42:29,737 module.py      :1037 INFO ] -          ! load /home/molli123/MLC/repos/local/cache/get-llvm_1df55db8/mlc-cached-state.json
[2026-05-07 10:42:29,737 script_utils.py:  88 INFO ] -     * mlcr get,cuda-devices
[2026-05-07 10:42:29,737 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py.
Error : no scripts were found with tags: ['get', 'cuda-devices'] (when variations ignored)
[2026-05-07 10:42:29,738 main.py        : 148 ERROR] -   at /home/molli123/mlc/lib/python3.12/site-packages/mlc/script_action.py:308 in call_script_module_function
[2026-05-07 10:42:29,738 main.py        : 178 ERROR] - Failed script: run-mlperf,inference,_full,_r5.1-dev,_performance-only
[2026-05-07 10:42:29,738 main.py        : 179 ERROR] - To rerun just the failed part: mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only --model=resnet50 --implementation=nvidia --framework=tensorrt --category=datacenter --scenario=Offline --execution_mode=valid --device=cuda
[2026-05-07 10:42:29,738 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-07 10:42:29,738 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-07 10:42:29,743 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev 97b03ddb5
(mlc) molli123@spark-7994:~$

whatever Benchmark I try to run, It fails. Any idea how to get started?

[arjunsuresh]

Hi @michaelfuckner There was a bug with a recently checked in code related to Intel GPU and that's why you got this error. If you do mlc pull repo and retry the command this error shouldn't happen.

I have recently run the below command using Nvidia 6.0 code - not yet updated in the docs page. I have tried only resnet50 and bert-99 on RTX 4090 - for any other model, we might need to ask copilot or similar to do any needed fixes as no one has worked on them to integrate with the automations.

The below command is also using --docker - to do the run on docker. We can avoid this if the Host has all dependencies to run Nvidia TensorRT but not tested for.

mlcr run-mlperf,inference,_full,_r6.1-dev,_performance-only    --model=resnet50    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=valid    --device=cuda    --quiet  --docker  

[michaelfuckner]

Hi,

I updated mlc and ran your command:

(mlc) molli123@spark-7994:~$ mlc pull repo
[2026-05-08 09:04:10,630 repo_action.py : 379 INFO ] - Repository mlperf-automations already exists at /home/molli123/MLC/repos/mlcommons@mlperf-automations. Checking for local changes...
[2026-05-08 09:04:10,633 repo_action.py : 400 INFO ] - No local changes detected. Pulling latest changes...
Already up to date.
[2026-05-08 09:04:11,004 repo_action.py : 404 INFO ] - Repository successfully pulled.
[2026-05-08 09:04:11,004 repo_action.py : 419 INFO ] - Registering the repo in repos.json
(mlc) molli123@spark-7994:~$ mlcr run-mlperf,inference,_full,_r6.1-dev,_performance-only    --model=resnet50    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=valid    --device=cuda    --quiet  --docker
[2026-05-08 09:04:37,781 script_utils.py:  88 INFO ] - * mlcr run-mlperf,inference,_full,_r6.1-dev,_performance-only
[2026-05-08 09:04:37,781 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py.
Error : no scripts were found with tags: ['run-mlperf', 'inference']

variation tags ['full', 'r6.1-dev', 'performance-only'] are not matching for the found script run-mlperf-inference-app with variations dict_keys(['r2.1', 'r3.0', 'r3.1', 'r4.0', 'r4.0-dev', 'r4.1', 'r4.1-dev', 'r5.0', 'r5.0-dev', 'r5.1', 'r5.1-dev', 'r6.0-dev', 'all-modes', 'accuracy-only', 'find-performance', 'performance-and-accuracy', 'performance-only', 'populate-readme', 'submission', 'full', 'short', 'all-scenarios', 'compliance', 'scc24-base', 'scc24-main'])

[2026-05-08 09:04:37,782 main.py        : 148 ERROR] -   at /home/molli123/mlc/lib/python3.12/site-packages/mlc/script_action.py:308 in call_script_module_function
[2026-05-08 09:04:37,782 main.py        : 178 ERROR] - Failed script: run-mlperf,inference,_full,_r6.1-dev,_performance-only
[2026-05-08 09:04:37,782 main.py        : 179 ERROR] - To rerun just the failed part: mlcr run-mlperf,inference,_full,_r6.1-dev,_performance-only --model=resnet50 --implementation=nvidia --framework=tensorrt --category=datacenter --scenario=Offline --execution_mode=valid --device=cuda --docker=True
[2026-05-08 09:04:37,782 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-08 09:04:37,782 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-08 09:04:37,787 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev cc725e1c9
(mlc) molli123@spark-7994:~$ 

6.0 fails as well:

command:

mlcr run-mlperf,inference,_full,_r6.0-dev,_performance-only    --model=resnet50    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=valid    --device=cuda    --quiet  --docker

Log end:

148.7 Successfully installed MarkupSafe-3.0.3 appdirs-1.4.4 mako-1.3.12 platformdirs-4.9.6 pycuda-2024.1 pytools-2026.1 siphash24-1.8 typing-extensions-4.15.0
148.9 [2026-05-08 07:07:15,148 customize.py   : 152 INFO ] -                         Detected version: 2024.1
148.9 [2026-05-08 07:07:15,174 module.py      :1870 INFO ] -                     - cache UID: af53ac9378f04b2a
148.9 [2026-05-08 07:07:15,176 script_utils.py:  88 INFO ] -             * mlcr get,nvidia,mitten
148.9 [2026-05-08 07:07:15,192 script_utils.py:  88 INFO ] -               * mlcr detect,os
148.9 [2026-05-08 07:07:15,193 module.py      :1037 INFO ] -                    ! load /home/mlcuser/MLC/repos/local/cache/detect-os_30bc6fa8/mlc-cached-state.json
149.0 [2026-05-08 07:07:15,195 script_utils.py:  88 INFO ] -               * mlcr get,python3
149.0 [2026-05-08 07:07:15,196 module.py      :1037 INFO ] -                    ! load /home/mlcuser/MLC/repos/local/cache/get-python3_51b0e04f/mlc-cached-state.json
149.0 [2026-05-08 07:07:15,217 script_utils.py:  88 INFO ] -               * mlcr get,generic-python-lib,_pycuda
149.0 [2026-05-08 07:07:15,230 script_utils.py:  88 INFO ] -                 * mlcr detect,os
149.0 [2026-05-08 07:07:15,231 module.py      :1037 INFO ] -                      ! load /home/mlcuser/MLC/repos/local/cache/detect-os_30bc6fa8/mlc-cached-state.json
149.0 [2026-05-08 07:07:15,234 script_utils.py:  88 INFO ] -                 * mlcr detect,cpu
149.0 [2026-05-08 07:07:15,235 module.py      :1037 INFO ] -                      ! load /home/mlcuser/MLC/repos/local/cache/detect-cpu_011052a8/mlc-cached-state.json
149.0 [2026-05-08 07:07:15,240 script_utils.py:  88 INFO ] -                 * mlcr get,python3
149.0 [2026-05-08 07:07:15,241 module.py      :1037 INFO ] -                      ! load /home/mlcuser/MLC/repos/local/cache/get-python3_51b0e04f/mlc-cached-state.json
149.0 [2026-05-08 07:07:15,288 script_utils.py:  88 INFO ] -                 * mlcr get,generic-python-lib,_pip
149.0 [2026-05-08 07:07:15,291 script_utils.py:  88 INFO ] -                   * mlcr get,python3
149.0 [2026-05-08 07:07:15,292 module.py      :1037 INFO ] -                        ! load /home/mlcuser/MLC/repos/local/cache/get-python3_51b0e04f/mlc-cached-state.json
149.1 /home/mlcuser/venv/mlcflow/bin/python3 /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
149.1 [2026-05-08 07:07:15,360 customize.py   : 152 INFO ] -                           Detected version: 26.1.1
149.1 [2026-05-08 07:07:15,362 script_utils.py:  88 INFO ] -                     * mlcr get,python3
149.1 [2026-05-08 07:07:15,363 module.py      :1037 INFO ] -                          ! load /home/mlcuser/MLC/repos/local/cache/get-python3_51b0e04f/mlc-cached-state.json
149.1 [2026-05-08 07:07:15,363 module.py      :1037 INFO ] -                          ! load /home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_ce40315c/mlc-cached-state.json
149.1 [2026-05-08 07:07:15,373 script_utils.py:  88 INFO ] -                 * mlcr get,cuda
149.2 [2026-05-08 07:07:15,397 script_utils.py:  88 INFO ] -                   * mlcr detect,os
149.2 [2026-05-08 07:07:15,399 module.py      :1037 INFO ] -                        ! load /home/mlcuser/MLC/repos/local/cache/detect-os_30bc6fa8/mlc-cached-state.json
149.2 [2026-05-08 07:07:15,410 module.py      :4104 INFO ] -                       # Requested paths: /home/mlcuser/venv/mlcflow/bin:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/bin:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/amazon/efa/bin:/opt/tensorrt/bin:/home/mlcuser/.local/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda:/opt/mlc-installs/cuda/install/bin
149.2 [2026-05-08 07:07:15,420 module.py      :3822 INFO ] -                       * /usr/local/cuda/bin/nvcc
149.2 [2026-05-08 07:07:15,457 customize.py   : 107 INFO ] -                           Detected version: 12.8
149.2 [2026-05-08 07:07:15,457 module.py      :4170 INFO ] -                       # Found artifact in /usr/local/cuda/bin/nvcc
149.2 [2026-05-08 07:07:15,492 customize.py   : 107 INFO ] -                       Detected version: 12.8
149.3 [2026-05-08 07:07:15,518 module.py      :1870 INFO ] -                     - cache UID: 2f3526a6dccb499f
149.3 [2026-05-08 07:07:15,518 module.py      :1924 INFO ] -                     * ENV[CUDA_HOME]: /usr/local/cuda
149.3 [2026-05-08 07:07:15,518 module.py      :1924 INFO ] -                     * ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
149.3 [2026-05-08 07:07:15,518 module.py      :1924 INFO ] -                     * ENV[MLC_CUDA_VERSION]: 12.8
149.3 [2026-05-08 07:07:15,518 module.py      :1924 INFO ] -                     * ENV[MLC_CUDA_VERSION_STRING]: cu128
149.3 [2026-05-08 07:07:15,518 module.py      :1924 INFO ] -                     * ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
149.3 [2026-05-08 07:07:15,525 module.py      :3953 INFO ] -                     - Searching for versions:  == 2024.1 >= 2024.1
149.4 [2026-05-08 07:07:15,628 customize.py   : 152 INFO ] -                             Detected version: 2024.1
149.5 [2026-05-08 07:07:15,723 module.py      :1870 INFO ] -                     - cache UID: 6ecb7a042db34000
149.5 [2026-05-08 07:07:15,726 script_utils.py:  88 INFO ] -               * mlcr get,git,_repo.https://github.com/NVIDIA/mitten,_patch
149.5 [2026-05-08 07:07:15,739 script_utils.py:  88 INFO ] -                 * mlcr detect,os
149.5 [2026-05-08 07:07:15,740 module.py      :1037 INFO ] -                      ! load /home/mlcuser/MLC/repos/local/cache/detect-os_30bc6fa8/mlc-cached-state.json
149.5 /home/mlcuser/MLC/repos/local/cache/get-git-repo_nvidia-mitten-s_2533c87c
149.5 rm -rf repo
149.5 ******************************************************
149.5 Current directory: /home/mlcuser/MLC/repos/local/cache/get-git-repo_nvidia-mitten-s_2533c87c
149.5
149.5 Cloning mitten from https://github.com/NVIDIA/mitten
149.5
149.5 git clone  --recurse-submodules https://github.com/NVIDIA/mitten --depth 5 repo
149.5
149.5 Cloning into 'repo'...
150.2
150.2 Updating submodules after checkout...
150.2
150.2 Applying patch /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-nvidia-mitten/patch/numpy-mitten-v5.0.patch
150.2 error: patch failed: setup.cfg:21
150.2 error: setup.cfg: patch does not apply
150.2 [2026-05-08 07:07:16,455 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py.
150.2 Error : Native run script failed inside MLC script (name = get-git-repo, return code = 256)
150.2
150.2
150.2 [2026-05-08 07:07:16,455 main.py        : 148 ERROR] -   at /home/mlcuser/venv/mlcflow/lib/python3.12/site-packages/mlc/script_action.py:308 in call_script_module_function
150.2 [2026-05-08 07:07:16,456 main.py        : 178 ERROR] - Failed script: get-git-repo
150.2 [2026-05-08 07:07:16,456 main.py        : 179 ERROR] - To rerun just the failed part: mlcr get-git-repo --print_env=False --print_deps=False --dump_version_info=True --fake_run=True
150.2 [2026-05-08 07:07:16,456 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
150.2 [2026-05-08 07:07:16,456 main.py        : 191 ERROR] - mlcflow 1.2.1
150.2 [2026-05-08 07:07:16,462 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev cc725e1c9
------
nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v5.0-cuda12.8-pytorch25.01-ubuntu24.04-aarch64-Grace-release.Dockerfile:47
--------------------
  45 |
  46 |     # Run commands
  47 | >>> RUN mlc pull repo && mlcr --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_valid,_r6.0-dev_default,_offline --action=docker --quiet=true --env.MLC_USER_RUN_DIR=/home/molli123 --env.MLC_QUIET=yes --env.MLC_HOST_PLATFORM_FLAVOR=aarch64 --env.MLC_HOST_OS_TYPE=linux --env.MLC_MLPERF_IMPLEMENTATION=nvidia --env.MLC_MLPERF_MODEL=resnet50 --env.MLC_MLPERF_LOADGEN_SCENARIO=Offline --env.MLC_MLPERF_RUN_STYLE=valid --env.MLC_MLPERF_SKIP_SUBMISSION_GENERATION=False --env.MLC_DOCKER_PRIVILEGED_MODE=True --env.MLC_MLPERF_SUBMISSION_DIVISION=open --env.MLC_MLPERF_INFERENCE_TP_SIZE=1 --env.MLC_MLPERF_SUBMISSION_SYSTEM_TYPE=datacenter --env.MLC_MLPERF_DEVICE=cuda --env.MLC_MLPERF_USE_DOCKER=True --env.MLC_MLPERF_BACKEND=tensorrt --env.MLC_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.MLC_MLPERF_INFERENCE_VERSION=6.0-dev --env.MLC_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r6.0-dev_default --env.MLC_MLPERF_SUBMISSION_CHECKER_VERSION=v6.0 --env.MLC_MLPERF_LOADGEN_MODE=performance --env.MLC_MLPERF_SUBMISSION_RUN=yes --env.MLC_RUN_SUBMISSION_CHECKER=no --env.MLC_MLPERF_INFERENCE_SOURCE_VERSION=6.0.14 --env.MLC_MLPERF_LAST_RELEASE=v6.0 --env.MLC_MLPERF_MODULARISED_INFERENCE_SUBMISSION_CHECKER=yes --env.MLC_MLPERF_INFERENCE_RESULTS_VERSION=r6.0-dev --env.MLC_MODEL=resnet50 --env.MLC_MLPERF_LOADGEN_COMPLIANCE=no --env.MLC_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.MLC_MLPERF_LOADGEN_SCENARIOS,=Offline --env.MLC_MLPERF_LOADGEN_MODES,=performance --env.MLC_OUTPUT_FOLDER_NAME=valid_results --env.MLC_RUN_STATE_DOCKER=False --const.MLC_USER_RUN_DIR=/home/molli123 --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.coco2014-dataset.tags=_full --add_deps_recursive.igbh-dataset.tags=_full --add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r6.0-dev --add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r6.0-dev --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.MLC_RUN_STATE_DOCKER=True
  48 |     RUN mlc pull repo && mlcr --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_valid,_r6.0-dev_default,_offline --action=docker --quiet=true --env.MLC_USER_RUN_DIR=/home/molli123 --env.MLC_QUIET=yes --env.MLC_HOST_PLATFORM_FLAVOR=aarch64 --env.MLC_HOST_OS_TYPE=linux --env.MLC_MLPERF_IMPLEMENTATION=nvidia --env.MLC_MLPERF_MODEL=resnet50 --env.MLC_MLPERF_LOADGEN_SCENARIO=Offline --env.MLC_MLPERF_RUN_STYLE=valid --env.MLC_MLPERF_SKIP_SUBMISSION_GENERATION=False --env.MLC_DOCKER_PRIVILEGED_MODE=True --env.MLC_MLPERF_SUBMISSION_DIVISION=open --env.MLC_MLPERF_INFERENCE_TP_SIZE=1 --env.MLC_MLPERF_SUBMISSION_SYSTEM_TYPE=datacenter --env.MLC_MLPERF_DEVICE=cuda --env.MLC_MLPERF_USE_DOCKER=True --env.MLC_MLPERF_BACKEND=tensorrt --env.MLC_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.MLC_MLPERF_INFERENCE_VERSION=6.0-dev --env.MLC_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r6.0-dev_default --env.MLC_MLPERF_SUBMISSION_CHECKER_VERSION=v6.0 --env.MLC_MLPERF_LOADGEN_MODE=performance --env.MLC_MLPERF_SUBMISSION_RUN=yes --env.MLC_RUN_SUBMISSION_CHECKER=no --env.MLC_MLPERF_INFERENCE_SOURCE_VERSION=6.0.14 --env.MLC_MLPERF_LAST_RELEASE=v6.0 --env.MLC_MLPERF_MODULARISED_INFERENCE_SUBMISSION_CHECKER=yes --env.MLC_MLPERF_INFERENCE_RESULTS_VERSION=r6.0-dev --env.MLC_MODEL=resnet50 --env.MLC_MLPERF_LOADGEN_COMPLIANCE=no --env.MLC_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.MLC_MLPERF_LOADGEN_SCENARIOS,=Offline --env.MLC_MLPERF_LOADGEN_MODES,=performance --env.MLC_OUTPUT_FOLDER_NAME=valid_results --env.MLC_RUN_STATE_DOCKER=False --const.MLC_USER_RUN_DIR=/home/molli123 --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.coco2014-dataset.tags=_full --add_deps_recursive.igbh-dataset.tags=_full --add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r6.0-dev --add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r6.0-dev --print_env=False --print_deps=False --dump_version_info=True --quiet --env.MLC_RUN_STATE_DOCKER=True
  49 |
--------------------
ERROR: failed to build: failed to solve: process "/bin/bash -c mlc pull repo && mlcr --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_valid,_r6.0-dev_default,_offline --action=docker --quiet=true --env.MLC_USER_RUN_DIR=/home/molli123 --env.MLC_QUIET=yes --env.MLC_HOST_PLATFORM_FLAVOR=aarch64 --env.MLC_HOST_OS_TYPE=linux --env.MLC_MLPERF_IMPLEMENTATION=nvidia --env.MLC_MLPERF_MODEL=resnet50 --env.MLC_MLPERF_LOADGEN_SCENARIO=Offline --env.MLC_MLPERF_RUN_STYLE=valid --env.MLC_MLPERF_SKIP_SUBMISSION_GENERATION=False --env.MLC_DOCKER_PRIVILEGED_MODE=True --env.MLC_MLPERF_SUBMISSION_DIVISION=open --env.MLC_MLPERF_INFERENCE_TP_SIZE=1 --env.MLC_MLPERF_SUBMISSION_SYSTEM_TYPE=datacenter --env.MLC_MLPERF_DEVICE=cuda --env.MLC_MLPERF_USE_DOCKER=True --env.MLC_MLPERF_BACKEND=tensorrt --env.MLC_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.MLC_MLPERF_INFERENCE_VERSION=6.0-dev --env.MLC_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r6.0-dev_default --env.MLC_MLPERF_SUBMISSION_CHECKER_VERSION=v6.0 --env.MLC_MLPERF_LOADGEN_MODE=performance --env.MLC_MLPERF_SUBMISSION_RUN=yes --env.MLC_RUN_SUBMISSION_CHECKER=no --env.MLC_MLPERF_INFERENCE_SOURCE_VERSION=6.0.14 --env.MLC_MLPERF_LAST_RELEASE=v6.0 --env.MLC_MLPERF_MODULARISED_INFERENCE_SUBMISSION_CHECKER=yes --env.MLC_MLPERF_INFERENCE_RESULTS_VERSION=r6.0-dev --env.MLC_MODEL=resnet50 --env.MLC_MLPERF_LOADGEN_COMPLIANCE=no --env.MLC_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.MLC_MLPERF_LOADGEN_SCENARIOS,=Offline --env.MLC_MLPERF_LOADGEN_MODES,=performance --env.MLC_OUTPUT_FOLDER_NAME=valid_results --env.MLC_RUN_STATE_DOCKER=False --const.MLC_USER_RUN_DIR=/home/molli123 --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.coco2014-dataset.tags=_full --add_deps_recursive.igbh-dataset.tags=_full --add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r6.0-dev --add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r6.0-dev --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.MLC_RUN_STATE_DOCKER=True" did not complete successfully: exit code: 1
[2026-05-08 09:07:17,304 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py.
Error : Native run script failed inside MLC script (name = build-docker-image, return code = 256)


[2026-05-08 09:07:17,306 main.py        : 148 ERROR] -   at /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py:418 in docker_run
[2026-05-08 09:07:17,306 main.py        : 178 ERROR] - Failed script: build-docker-image
[2026-05-08 09:07:17,306 main.py        : 179 ERROR] - To rerun just the failed part: mlcr build-docker-image --base_image=nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v5.0-cuda12.8-pytorch25.01-ubuntu24.04-aarch64-Grace-release --os=ubuntu --os_version=22.04 --image_name=mlperf-inference-nvidia-v5.0-common --user=mlcuser --all_gpus=yes --shm_size=32gb --interactive=True --use_google_dns=True --detached=False
[2026-05-08 09:07:17,306 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-08 09:07:17,306 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-08 09:07:17,312 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev cc725e1c9

on the other hand the docker sample job runs fine

(mlc) molli123@spark-7994:~$ docker run --rm  --gpus all ubuntu nvidia-smi
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
2113f8d7eb32: Pull complete
4a7720058461: Pull complete
Digest: sha256:f3d28607ddd78734bb7f71f117f3c6706c666b8b76cbff7c9ff6e5718d46ff64
Status: Downloaded newer image for ubuntu:latest
Fri May  8 07:18:38 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.142                Driver Version: 580.142        CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   35C    P8              4W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
(mlc) molli123@spark-7994:~$

[arjunsuresh]

I'm sorry. I was using a fork - just merged everything here. Can you please try the below command and see if the docker build and test run happens? So far we haven't tested on an aarch64 system but should hopefully work.

mlcr run-mlperf,inference,_full,_r6.0,_performance-only --device=cuda    --model=resnet50    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=valid    --device=cuda    --quiet  --docker --docker_cache=no --docker_rebuild

[michaelfuckner]

OK, now I get one step further- until it struggles with aarch64 (exec format error). Would it be possible to get it running anyway?

[+] Building 1.8s (5/18)                                                                     docker:default
 => [internal] load build definition from nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2  0.0s
 => => transferring dockerfile: 6.55kB                                                                 0.0s
 => [internal] load metadata for nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v3.1-cuda12.2-cudnn8.9  1.5s
 => [internal] load .dockerignore                                                                      0.0s
 => => transferring context: 45B                                                                       0.0s
 => CACHED [ 1/15] FROM nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-u  0.0s
 => ERROR [ 2/15] RUN apt-get update -y                                                                0.2s
------                                                                                                      
 > [ 2/15] RUN apt-get update -y:
0.144 exec /bin/sh: exec format error
------

 1 warning found (use docker --debug to expand):
 - InvalidBaseImagePlatform: Base image nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-l4-public was pulled with platform "linux/amd64", expected "linux/arm64" for current build (line 1)
nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-l4-public.Dockerfile:16
--------------------
  14 |     # Notes: https://runnable.com/blog/9-common-dockerfile-mistakes
  15 |     # Install system dependencies
  16 | >>> RUN apt-get update -y
  17 |     RUN apt-get install -y python3 python3-pip git sudo wget python3-venv systemctl unzip ca-certificates ssl-cert binutils
  18 |     
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c apt-get update -y" did not complete successfully: exit code: 255
[2026-05-09 07:40:57,922 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py. 
Error : Native run script failed inside MLC script (name = build-docker-image, return code = 256)


[2026-05-09 07:40:57,924 main.py        : 148 ERROR] -   at /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py:419 in docker_run
[2026-05-09 07:40:57,924 main.py        : 178 ERROR] - Failed script: build-docker-image
[2026-05-09 07:40:57,924 main.py        : 179 ERROR] - To rerun just the failed part: mlcr build-docker-image --base_image=nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-l4-public --os=ubuntu --os_version=22.04 --image_name=mlperf-inference-nvidia-v6.0-common --user=ubuntu --cache=no --system_site_packages=yes --all_gpus=yes --shm_size=32gb --interactive=True --use_google_dns=True --detached=False
[2026-05-09 07:40:57,924 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-09 07:40:57,924 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-09 07:40:57,930 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev 845df41b6
(mlc) molli123@spark-7994:~$ 

[arjunsuresh]

Are you running this same command?

mlcr run-mlperf,inference,_full,_r6.0,_performance-only --device=cuda    --model=resnet50    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=valid    --device=cuda    --quiet  --docker --docker_cache=no --docker_rebuild

because the console log shows mlpinf-v3.1 which is for an older version. Please do rm -rf $HOME/MLC/repos/mlcommons@mlperf-automations && mlc pull repo mlcommons@mlperf-automations --branch=dev to get the latest changes.

[michaelfuckner]

looks quite similar- still mlpinf 3.1 and still exec format error

(mlc) molli123@spark-7994:~$ rm -rf $HOME/MLC/repos/mlcommons@mlperf-automations && mlc pull repo mlcommons@mlperf-automations --branch=dev 
[2026-05-10 20:25:17,416 action.py      : 117 WARN ] - Warning: /home/molli123/MLC/repos/mlcommons@mlperf-automations not found. Considering it as a corrupt entry and deleting from repos.json...
[2026-05-10 20:25:17,416 repo_action.py : 622 INFO ] - rm command has been called for repo. This would delete the repo folder and unregister the repo from repos.json
[2026-05-10 20:25:17,416 repo_action.py : 665 WARN ] - Repo mlcommons@mlperf-automations was not found in the repo folder. repos.json will be checked for external paths. If any, that will be removed.
[2026-05-10 20:25:17,416 repo_action.py : 673 INFO ] - Unregistering the repo in path /home/molli123/MLC/repos/mlcommons@mlperf-automations
[2026-05-10 20:25:17,416 repo_action.py : 682 INFO ] - Path: /home/molli123/MLC/repos/mlcommons@mlperf-automations has been removed.
[2026-05-10 20:25:17,419 repo_action.py : 363 INFO ] - Cloning repository https://github.com/mlcommons/mlperf-automations.git to /home/molli123/MLC/repos/mlcommons@mlperf-automations...
Cloning into '/home/molli123/MLC/repos/mlcommons@mlperf-automations'...
remote: Enumerating objects: 90443, done.
remote: Counting objects: 100% (966/966), done.
remote: Compressing objects: 100% (450/450), done.
remote: Total 90443 (delta 690), reused 523 (delta 516), pack-reused 89477 (from 2)
Receiving objects: 100% (90443/90443), 20.96 MiB | 37.01 MiB/s, done.
Resolving deltas: 100% (63559/63559), done.
[2026-05-10 20:25:19,041 repo_action.py : 419 INFO ] - Registering the repo in repos.json
[2026-05-10 20:25:19,041 repo_action.py : 181 INFO ] - Added new repo path: /home/molli123/MLC/repos/mlcommons@mlperf-automations
[2026-05-10 20:25:19,041 repo_action.py : 185 INFO ] - Updated repos.json at /home/molli123/MLC/repos/repos.json
(mlc) molli123@spark-7994:~$ mlcr run-mlperf,inference,_full,_r6.0,_performance-only --device=cuda    --model=resnet50    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=valid    --device=cuda    --quiet  --docker --docker_cache=no --docker_rebuild
[2026-05-10 20:25:39,639 script_utils.py:  88 INFO ] - * mlcr run-mlperf,inference,_full,_r6.0,_performance-only
[2026-05-10 20:25:39,644 script_utils.py:  88 INFO ] -   * mlcr get,mlcommons,inference,src
[2026-05-10 20:25:39,648 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/get-mlperf-inference-src_49bff69c/mlc-cached-state.json
[2026-05-10 20:25:39,650 script_utils.py:  88 INFO ] -   * mlcr get,mlperf,inference,results,dir,_version.r6.0
[2026-05-10 20:25:39,650 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/get-mlperf-inference-results-dir_dc6ddadd/mlc-cached-state.json
[2026-05-10 20:25:39,651 script_utils.py:  88 INFO ] -   * mlcr install,pip-package,for-mlc-python,_package.tabulate
[2026-05-10 20:25:39,651 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/install-pip-package-for-mlc-python_a514f465/mlc-cached-state.json
[2026-05-10 20:25:39,652 script_utils.py:  88 INFO ] -   * mlcr get,mlperf,inference,utils
[2026-05-10 20:25:39,657 script_utils.py:  88 INFO ] -     * mlcr get,mlperf,inference,src
[2026-05-10 20:25:39,657 module.py      :1037 INFO ] -          ! load /home/molli123/MLC/repos/local/cache/get-mlperf-inference-src_49bff69c/mlc-cached-state.json
Using MLCommons Inference source from /home/molli123/MLC/repos/local/cache/get-git-repo_inference-src_7b282530/inference
[2026-05-10 20:25:39,681 customize.py   : 273 INFO ] - 
Running loadgen scenario: Offline and mode: performance
[2026-05-10 20:25:39,750 script_utils.py:  88 INFO ] - * mlcr get,mlperf,inference,submission,dir,local,_version.r6.0
[2026-05-10 20:25:39,751 module.py      :1037 INFO ] -      ! load /home/molli123/MLC/repos/local/cache/get-mlperf-inference-submission-dir_1e534c0e/mlc-cached-state.json
[2026-05-10 20:25:39,753 script_utils.py:  88 INFO ] - * mlcr get,dataset,imagenet,validation,original,_full
[2026-05-10 20:25:39,754 module.py      :1037 INFO ] -      ! load /home/molli123/MLC/repos/local/cache/get-dataset-imagenet-val_d89e47d8/mlc-cached-state.json
[2026-05-10 20:25:39,755 script_utils.py:  88 INFO ] - * mlcr get,nvidia-docker
[2026-05-10 20:25:39,755 module.py      :1037 INFO ] -      ! load /home/molli123/MLC/repos/local/cache/get-nvidia-docker_a3c2b143/mlc-cached-state.json
[2026-05-10 20:25:39,756 script_utils.py:  88 INFO ] - * mlcr get,mlperf,inference,nvidia,scratch,space,_version.r6.0
[2026-05-10 20:25:39,757 module.py      :1037 INFO ] -      ! load /home/molli123/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_2113d41f/mlc-cached-state.json
[2026-05-10 20:25:39,758 script_utils.py:  88 INFO ] - * mlcr get,nvidia-docker
[2026-05-10 20:25:39,758 module.py      :1037 INFO ] -      ! load /home/molli123/MLC/repos/local/cache/get-nvidia-docker_a3c2b143/mlc-cached-state.json
[2026-05-10 20:25:39,815 script_utils.py:  88 INFO ] - * mlcr detect,os
[2026-05-10 20:25:39,816 module.py      :1037 INFO ] -      ! load /home/molli123/MLC/repos/local/cache/detect-os_aac747bd/mlc-cached-state.json
[2026-05-10 20:25:39,823 script_utils.py:  88 INFO ] - * mlcr build,dockerfile
[2026-05-10 20:25:39,828 script_utils.py:  88 INFO ] -   * mlcr get,docker
[2026-05-10 20:25:39,829 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/get-docker_ccdcdc91/mlc-cached-state.json
[2026-05-10 20:25:39,886 customize.py   : 478 INFO ] - mlc pull repo && mlcr --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_valid,_r6.0_default,_offline --action=docker --quiet=true --env.MLC_USER_RUN_DIR=/home/molli123 --env.MLC_QUIET=yes --env.MLC_HOST_PLATFORM_FLAVOR=aarch64 --env.MLC_HOST_OS_TYPE=linux --env.MLC_MLPERF_IMPLEMENTATION=nvidia --env.MLC_MLPERF_MODEL=resnet50 --env.MLC_MLPERF_LOADGEN_SCENARIO=Offline --env.MLC_MLPERF_RUN_STYLE=valid --env.MLC_MLPERF_SKIP_SUBMISSION_GENERATION=False --env.MLC_DOCKER_PRIVILEGED_MODE=True --env.MLC_MLPERF_SUBMISSION_DIVISION=open --env.MLC_MLPERF_INFERENCE_TP_SIZE=1 --env.MLC_MLPERF_SUBMISSION_SYSTEM_TYPE=datacenter --env.MLC_MLPERF_DEVICE=cuda --env.MLC_MLPERF_USE_DOCKER=True --env.MLC_MLPERF_BACKEND=tensorrt --env.MLC_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.MLC_MLPERF_INFERENCE_VERSION=6.0 --env.MLC_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r6.0_default --env.MLC_MLPERF_SUBMISSION_CHECKER_VERSION=v6.0 --env.MLC_CUSTOM_SYSTEM_NVIDIA=no --env.MLC_MLPERF_LOADGEN_MODE=performance --env.MLC_MLPERF_SUBMISSION_RUN=yes --env.MLC_RUN_SUBMISSION_CHECKER=no --env.MLC_MLPERF_INFERENCE_SOURCE_VERSION=6.0.14 --env.MLC_MLPERF_LAST_RELEASE=v6.0 --env.MLC_MLPERF_MODULARISED_INFERENCE_SUBMISSION_CHECKER=yes --env.MLC_MLPERF_INFERENCE_RESULTS_VERSION=r6.0 --env.MLC_MODEL=resnet50 --env.MLC_MLPERF_LOADGEN_COMPLIANCE=no --env.MLC_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.MLC_MLPERF_LOADGEN_SCENARIOS,=Offline --env.MLC_MLPERF_LOADGEN_MODES,=performance --env.MLC_OUTPUT_FOLDER_NAME=valid_results --env.MLC_RUN_STATE_DOCKER=False --env.MLC_DOCKER_CACHE=no --const.MLC_USER_RUN_DIR=/home/molli123 --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.coco2014-dataset.tags=_full --add_deps_recursive.igbh-dataset.tags=_full --add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r6.0 --add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r6.0 --add_deps_recursive.mlperf-inference-nvidia-scratch-space.tags=_version.r6.0 --print_env=False --print_deps=False --dump_version_info=True --quiet
[2026-05-10 20:25:39,886 customize.py   : 548 INFO ] - Dockerfile written at /home/molli123/MLC/repos/local/docker/app-mlperf-inference_d775c/dockerfiles/nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-l4-public.Dockerfile
[2026-05-10 20:25:39,889 script_utils.py:  88 INFO ] - * mlcr get,docker
[2026-05-10 20:25:39,890 module.py      :1037 INFO ] -      ! load /home/molli123/MLC/repos/local/cache/get-docker_ccdcdc91/mlc-cached-state.json
[2026-05-10 20:25:39,894 script_utils.py:  88 INFO ] - * mlcr run,docker,container
[2026-05-10 20:25:39,898 script_utils.py:  88 INFO ] -   * mlcr get,docker
[2026-05-10 20:25:39,899 module.py      :1037 INFO ] -        ! load /home/molli123/MLC/repos/local/cache/get-docker_ccdcdc91/mlc-cached-state.json
[2026-05-10 20:25:39,955 customize.py   :  71 INFO ] - 
[2026-05-10 20:25:39,955 customize.py   :  72 INFO ] - Checking existing Docker container:
[2026-05-10 20:25:39,955 customize.py   :  73 INFO ] - 
[2026-05-10 20:25:39,955 customize.py   :  82 INFO ] -   docker ps --format "{{ .ID }},"  --filter "ancestor=localhost/local/mlperf-inference-nvidia-v6.0-common:nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2-cudnn8.9-x8664-ubuntu20.04-l4-public-latest"  2> /dev/null || true 
[2026-05-10 20:25:39,955 customize.py   :  83 INFO ] - 
[2026-05-10 20:25:39,965 customize.py   : 110 INFO ] - No existing container
[2026-05-10 20:25:39,965 customize.py   : 121 INFO ] - 
[2026-05-10 20:25:39,965 customize.py   : 122 INFO ] - Checking Docker images:
[2026-05-10 20:25:39,965 customize.py   : 123 INFO ] - 
[2026-05-10 20:25:39,965 customize.py   : 124 INFO ] -   docker images -q localhost/local/mlperf-inference-nvidia-v6.0-common:nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2-cudnn8.9-x8664-ubuntu20.04-l4-public-latest 2> /dev/null || true
[2026-05-10 20:25:39,965 customize.py   : 125 INFO ] - 
[2026-05-10 20:25:39,976 script_utils.py:  88 INFO ] -     * mlcr build,docker,image
[2026-05-10 20:25:39,978 script_utils.py:  88 INFO ] -       * mlcr get,docker
[2026-05-10 20:25:39,979 module.py      :1037 INFO ] -            ! load /home/molli123/MLC/repos/local/cache/get-docker_ccdcdc91/mlc-cached-state.json
[2026-05-10 20:25:39,983 customize.py   :  96 INFO ] - docker build  --no-cache --build-arg USER=molli123 --build-arg HOME=/home/molli123 --build-arg GID=" $(id -g $USER) " --build-arg UID=" $(id -u $USER) " -f "/home/molli123/MLC/repos/local/docker/app-mlperf-inference_d775c/dockerfiles/nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-l4-public.Dockerfile" -t "localhost/local/mlperf-inference-nvidia-v6.0-common:nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2-cudnn8.9-x8664-ubuntu20.04-l4-public-latest" .
[+] Building 1.1s (5/18)                                                                     docker:default
 => [internal] load build definition from nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2  0.0s
 => => transferring dockerfile: 6.55kB                                                                 0.0s
 => [internal] load metadata for nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v3.1-cuda12.2-cudnn8.9  0.9s
 => [internal] load .dockerignore                                                                      0.0s
 => => transferring context: 45B                                                                       0.0s
 => CACHED [ 1/15] FROM nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-u  0.0s
 => ERROR [ 2/15] RUN apt-get update -y                                                                0.2s
------                                                                                                      
 > [ 2/15] RUN apt-get update -y:
0.140 exec /bin/sh: exec format error
------

 1 warning found (use docker --debug to expand):
 - InvalidBaseImagePlatform: Base image nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-l4-public was pulled with platform "linux/amd64", expected "linux/arm64" for current build (line 1)
nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-l4-public.Dockerfile:16
--------------------
  14 |     # Notes: https://runnable.com/blog/9-common-dockerfile-mistakes
  15 |     # Install system dependencies
  16 | >>> RUN apt-get update -y
  17 |     RUN apt-get install -y python3 python3-pip git sudo wget python3-venv systemctl unzip ca-certificates ssl-cert binutils
  18 |     
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c apt-get update -y" did not complete successfully: exit code: 255
[2026-05-10 20:25:41,208 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py. 
Error : Native run script failed inside MLC script (name = build-docker-image, return code = 256)


[2026-05-10 20:25:41,210 main.py        : 148 ERROR] -   at /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py:419 in docker_run
[2026-05-10 20:25:41,210 main.py        : 178 ERROR] - Failed script: build-docker-image
[2026-05-10 20:25:41,210 main.py        : 179 ERROR] - To rerun just the failed part: mlcr build-docker-image --base_image=nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v3.1-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-l4-public --os=ubuntu --os_version=22.04 --image_name=mlperf-inference-nvidia-v6.0-common --user=ubuntu --cache=no --system_site_packages=yes --all_gpus=yes --shm_size=32gb --interactive=True --use_google_dns=True --detached=False
[2026-05-10 20:25:41,210 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-10 20:25:41,210 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-10 20:25:41,235 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev 430f2f84a
(mlc) molli123@spark-7994:~$ 

[michaelfuckner]

tried the same commands on DGX B300 (AMD64 instead of AARCH64), but it also fails- but different. Do you need the whole long log?

[05/11/2026-10:16:36] [TRT] [W] Dequantize 165 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 169 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 173 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 177 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 181 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 185 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 189 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 193 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 197 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 201 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 205 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 209 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 213 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [W] Dequantize 215 [SCALE] has invalid precision Int8, ignored.
[05/11/2026-10:16:36] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[05/11/2026-10:16:36] [TRT] [I] Compiler backend is used during engine build.
[05/11/2026-10:18:31] [TRT] [W] UNSUPPORTED_STATE: Skipping tactic 0 due to insufficient memory on requested size of 4397306364 detected for tactic 0x0000000000000000.
[05/11/2026-10:18:31] [TRT] [E] IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[Dequantize 218...cast_layer]}. In computeCosts at /_src/optimizer/common/tactic/optimizer.cpp:4143)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_175d0095/repo/closed/NVIDIA/code/main.py", line 803, in <module>
    runner.run_all()
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_175d0095/repo/closed/NVIDIA/code/main.py", line 790, in run_all
    self._run_workload(benchmark, scenario)
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_175d0095/repo/closed/NVIDIA/code/main.py", line 575, in _run_workload
    pipeline.run()
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/debug/debug_manager.py", line 260, in _wrapper
    raise exc_info[1]
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/debug/debug_manager.py", line 247, in _wrapper
    retval = obj(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/pipeline/pipeline.py", line 167, in run
    raise exc
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/pipeline/pipeline.py", line 141, in run
    run_output = instance.run(self.scratch_space, op_inputs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/debug/debug_manager.py", line 260, in _wrapper
    raise exc_info[1]
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/debug/debug_manager.py", line 247, in _wrapper
    retval = obj(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_175d0095/repo/closed/NVIDIA/code/ops/generate_engines.py", line 224, in run
    builder(c_eng.batch_size, eng_fpath, network)
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/nvidia/builder.py", line 404, in __call__
    return self.build_engine(network, builder_config, save_to)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/nvidia/builder.py", line 373, in build_engine
    f.write(serialized_network)
TypeError: a bytes-like object is required, not 'NoneType'
make: *** [Makefile:38: generate_engines] Error 1
[2026-05-11 10:18:35,030 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py.
Error : Native run script failed inside MLC script (name = app-mlperf-inference-nvidia, return code = 256)


[2026-05-11 10:18:35,030 main.py        : 148 ERROR] -   at /home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/mlc/script_action.py:308 in call_script_module_function
[2026-05-11 10:18:35,030 main.py        : 178 ERROR] - Failed script: app-mlperf-inference-nvidia
[2026-05-11 10:18:35,030 main.py        : 179 ERROR] - To rerun just the failed part: mlcr app-mlperf-inference-nvidia --print_env=False --print_deps=False --dump_version_info=True
[2026-05-11 10:18:35,030 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-11 10:18:35,030 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-11 10:18:35,040 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev 430f2f84a
ubuntu@15bfed9c6c13:~$

[arjunsuresh]

@michaelfuckner yes the earlier error on aarch64 was due to the new docker image given only for x86_64. This is fixed now. If you do mlc pull repo, the error should be resolved.

The new error looks like a GPU specific one. Only the below part is relevant. If you have copilot setup, it should be able to solve this for you if you can give the MLC repo as the workspace.

Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[Dequantize 218...cast_layer]}. In computeCosts at /_src/optimizer/common/tactic/optimizer.cpp:4143)
Traceback (most recent call last):

Since DGX is an Nvidia submitted system you can also try the docker run commands from Nvidia if that works. In MLC, so far we have tested only on RTX4090.

https://github.com/mlcommons/inference_results_v6.0/tree/main/closed/NVIDIA

[michaelfuckner]

@arjunsuresh on Spark I now get an error for mlc pull

(mlc) molli123@spark-7994:~$ rm -rf $HOME/MLC/repos/mlcommons@mlperf-automations && mlc pull repo mlcommons@mlperf-automations --branch=dev
[2026-05-11 15:19:05,543 action.py      : 117 WARN ] - Warning: /home/molli123/MLC/repos/mlcommons@mlperf-automations not found. Considering it as a corrupt entry and deleting from repos.json...
[2026-05-11 15:19:05,544 repo_action.py : 622 INFO ] - rm command has been called for repo. This would delete the repo folder and unregister the repo from repos.json
[2026-05-11 15:19:05,544 repo_action.py : 665 WARN ] - Repo mlcommons@mlperf-automations was not found in the repo folder. repos.json will be checked for external paths. If any, that will be removed.
[2026-05-11 15:19:05,544 repo_action.py : 673 INFO ] - Unregistering the repo in path /home/molli123/MLC/repos/mlcommons@mlperf-automations
[2026-05-11 15:19:05,544 repo_action.py : 682 INFO ] - Path: /home/molli123/MLC/repos/mlcommons@mlperf-automations has been removed.
[2026-05-11 15:19:05,546 repo_action.py : 363 INFO ] - Cloning repository https://github.com/mlcommons/mlperf-automations.git to /home/molli123/MLC/repos/mlcommons@mlperf-automations...
Cloning into '/home/molli123/MLC/repos/mlcommons@mlperf-automations'...
remote: Enumerating objects: 90487, done.
remote: Counting objects: 100% (1118/1118), done.
remote: Compressing objects: 100% (545/545), done.
remote: Total 90487 (delta 776), reused 592 (delta 573), pack-reused 89369 (from 2)
Receiving objects: 100% (90487/90487), 21.02 MiB | 26.19 MiB/s, done.
Resolving deltas: 100% (63532/63532), done.
[2026-05-11 15:19:07,552 repo_action.py : 419 INFO ] - Registering the repo in repos.json
[2026-05-11 15:19:07,553 repo_action.py : 181 INFO ] - Added new repo path: /home/molli123/MLC/repos/mlcommons@mlperf-automations
[2026-05-11 15:19:07,553 repo_action.py : 185 INFO ] - Updated repos.json at /home/molli123/MLC/repos/repos.json
[2026-05-11 15:19:07,933 index.py       : 406 ERROR] - Meta validation error: /home/molli123/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/meta.yaml: variations.nvidia-original,r6.0_default.update_meta_if_env has type 'dict', expected {'list'}
[2026-05-11 15:19:07,933 index.py       : 424 ERROR] - Error processing /home/molli123/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/meta.yaml: Meta validation failed for /home/molli123/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/meta.yaml. Fix the above error(s) and try again.
[2026-05-11 15:19:08,807 index.py       : 406 ERROR] - Meta validation error: /home/molli123/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/meta.yaml: variations.nvidia-original,r6.0_default.update_meta_if_env has type 'dict', expected {'list'}
[2026-05-11 15:19:08,807 index.py       : 424 ERROR] - Error processing /home/molli123/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/meta.yaml: Meta validation failed for /home/molli123/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference/meta.yaml. Fix the above error(s) and try again.
(mlc) molli123@spark-7994:~$

[arjunsuresh]

My apologies. Please do mlc pull repo once more.

[michaelfuckner]

now it fails later inside the docker container

/home/ubuntu/venv/mlcflow/bin/python3 /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
[2026-05-11 15:26:46,315 customize.py   : 152 INFO ] -                       Detected version: 4.1.50
[2026-05-11 15:26:46,317 script_utils.py:  88 INFO ] -                 * mlcr get,python3
[2026-05-11 15:26:46,319 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
[2026-05-11 15:26:46,320 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_8e8daaf2/mlc-cached-state.json
[2026-05-11 15:26:46,333 script_utils.py:  88 INFO ] -             * mlcr get,dataset,original,imagenet,_full
[2026-05-11 15:26:46,334 module.py      :1037 INFO ] -                  ! load /home/ubuntu/MLC/repos/local/cache/get-dataset-imagenet-val_d89e47d8/mlc-cached-state.json
[2026-05-11 15:26:46,368 script_utils.py:  88 INFO ] -             * mlcr get,ml-model,resnet50,_fp32,_onnx,_opset-8
[2026-05-11 15:26:46,369 module.py      :1037 INFO ] -                  ! load /home/ubuntu/MLC/repos/local/cache/get-ml-model-resnet50_2c312e25/mlc-cached-state.json
[2026-05-11 15:26:46,369 module.py      :1924 INFO ] -               * Path to the ML model: /home/ubuntu/MLC/repos/local/cache/download-file_ml-model-resnet_0931f4d9/resnet50_v1.onnx
[2026-05-11 15:26:46,386 script_utils.py:  88 INFO ] -             * mlcr get,mlcommons,inference,src
[2026-05-11 15:26:46,388 module.py      :1037 INFO ] -                  ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-src_8edfdb63/mlc-cached-state.json
[2026-05-11 15:26:46,396 script_utils.py:  88 INFO ] -             * mlcr get,nvidia,mlperf,inference,common-code,_v6.0,_mlcommons
[2026-05-11 15:26:46,397 module.py      :1037 INFO ] -                  ! load /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-common-code_9823a353/mlc-cached-state.json
[2026-05-11 15:26:46,445 script_utils.py:  88 INFO ] -             * mlcr get,generic-python-lib,_package.pycuda
[2026-05-11 15:26:46,449 script_utils.py:  88 INFO ] -               * mlcr get,python3
[2026-05-11 15:26:46,450 module.py      :1037 INFO ] -                    ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
/home/ubuntu/venv/mlcflow/bin/python3 /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
[2026-05-11 15:26:46,545 customize.py   : 152 INFO ] -                       Detected version: 2026.1
[2026-05-11 15:26:46,547 script_utils.py:  88 INFO ] -                 * mlcr get,python3
[2026-05-11 15:26:46,549 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
[2026-05-11 15:26:46,549 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_efef0797/mlc-cached-state.json
[2026-05-11 15:26:46,599 script_utils.py:  88 INFO ] -             * mlcr get,generic-python-lib,_package.nvtx
[2026-05-11 15:26:46,602 script_utils.py:  88 INFO ] -               * mlcr get,python3
[2026-05-11 15:26:46,604 module.py      :1037 INFO ] -                    ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
/home/ubuntu/venv/mlcflow/bin/python3 /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
[2026-05-11 15:26:46,696 customize.py   : 152 INFO ] -                       Detected version: 0.2.14
[2026-05-11 15:26:46,698 script_utils.py:  88 INFO ] -                 * mlcr get,python3
[2026-05-11 15:26:46,700 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
[2026-05-11 15:26:46,700 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_f66487b4/mlc-cached-state.json
[2026-05-11 15:26:46,753 script_utils.py:  88 INFO ] -             * mlcr get,generic-python-lib,_package.cuda-python
[2026-05-11 15:26:46,756 script_utils.py:  88 INFO ] -               * mlcr get,python3
[2026-05-11 15:26:46,757 module.py      :1037 INFO ] -                    ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
/home/ubuntu/venv/mlcflow/bin/python3 /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
[2026-05-11 15:26:46,847 customize.py   : 152 INFO ] -                       Detected version: 13.2.0
[2026-05-11 15:26:46,849 script_utils.py:  88 INFO ] -                 * mlcr get,python3
[2026-05-11 15:26:46,851 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
[2026-05-11 15:26:46,851 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_e175c65c/mlc-cached-state.json
[2026-05-11 15:26:46,856 script_utils.py:  88 INFO ] -             * mlcr get,nvidia,mitten
[2026-05-11 15:26:46,856 module.py      :1037 INFO ] -                  ! load /home/ubuntu/MLC/repos/local/cache/get-nvidia-mitten_e839336f/mlc-cached-state.json
[2026-05-11 15:26:46,905 script_utils.py:  88 INFO ] -             * mlcr get,generic-python-lib,_onnx-graphsurgeon
[2026-05-11 15:26:46,908 script_utils.py:  88 INFO ] -               * mlcr get,python3
[2026-05-11 15:26:46,910 module.py      :1037 INFO ] -                    ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
/home/ubuntu/venv/mlcflow/bin/python3 /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
[2026-05-11 15:26:47,005 customize.py   : 152 INFO ] -                       Detected version: 0.3.25
[2026-05-11 15:26:47,007 script_utils.py:  88 INFO ] -                 * mlcr get,python3
[2026-05-11 15:26:47,009 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-python3_f5503fed/mlc-cached-state.json
[2026-05-11 15:26:47,009 module.py      :1037 INFO ] -                      ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_44de4830/mlc-cached-state.json
ln -sf /home/ubuntu/MLC/repos/local/cache/download-file_ml-model-resnet_0931f4d9/resnet50_v1.onnx /home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_2113d41f/models/ResNet50/resnet50_v1.onnx && make preprocess_data BENCHMARKS='resnet50'
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_f48fdddb/repo/closed/NVIDIA/code/resnet50/tensorrt/__init__.py", line 13, in <module>
    from code.ops.harness import LWISExecutableHarness
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_f48fdddb/repo/closed/NVIDIA/code/ops/__init__.py", line 16, in <module>
    from .generate_engines import (EngineBuilderOp,
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_f48fdddb/repo/closed/NVIDIA/code/ops/generate_engines.py", line 26, in <module>
    from ..common.workload import EngineIndex
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_f48fdddb/repo/closed/NVIDIA/code/common/workload.py", line 18, in <module>
    from code.common.systems.system_list import DETECTED_SYSTEM
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_f48fdddb/repo/closed/NVIDIA/code/common/systems/system_list.py", line 317, in <module>
    DETECTED_SYSTEM = KnownSystem.get_runtime_system()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_f48fdddb/repo/closed/NVIDIA/code/common/systems/system_list.py", line 264, in get_runtime_system
    sys = System.detect()[0]
          ^^^^^^^^^^^^^^^
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/system/system.py", line 45, in detect
    cpu = CPU.detect()[0]
          ^^^^^^^^^^^^
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/system/cpu.py", line 76, in detect
    process_field(d)
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/system/cpu.py", line 70, in process_field
    _t[keyspace] = data
    ~~^^^^^^^^^^
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/tree.py", line 396, in __setitem__
    self.insert_value(keyspace[-1], value, keyspace=keyspace[:-1])
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/tree.py", line 303, in insert_value
    curr.add_child(new_node)
  File "/home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/nvmitten/tree.py", line 272, in add_child
    raise ValueError(f"Cannot insert node with duplicate name {child_node.name}")
ValueError: Cannot insert node with duplicate name model name
make: *** [/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_f48fdddb/repo/closed/NVIDIA/Makefile.data:38: preprocess_data] Error 1
[2026-05-11 15:26:50,604 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py.
Error : Native run script failed inside MLC script (name = app-mlperf-inference-nvidia, return code = 256)


[2026-05-11 15:26:50,605 main.py        : 148 ERROR] -   at /home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/mlc/script_action.py:308 in call_script_module_function
[2026-05-11 15:26:50,605 main.py        : 178 ERROR] - Failed script: app-mlperf-inference-nvidia
[2026-05-11 15:26:50,605 main.py        : 179 ERROR] - To rerun just the failed part: mlcr app-mlperf-inference-nvidia --print_env=False --print_deps=False --dump_version_info=True
[2026-05-11 15:26:50,605 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-11 15:26:50,605 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-11 15:26:50,610 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev 0547f4bb3
ubuntu@74ea62471983:~$

[arjunsuresh]

@michaelfuckner I added a fix as suggested by Claude here Since you're trying on a new system more such issues are expected. It took me about 2 days fixing 10+ bugs thanks to Claude to get the Nvidia MLPerf inference run on RTX4090.

[michaelfuckner]

Hi,

do I need a config file for my system?

rgeon
[2026-05-12 19:08:59,538 script_utils.py:  88 INFO ] -             * mlcr get,python3
[2026-05-12 19:08:59,540 module.py      :1037 INFO ] -                  ! load /home/ubuntu/MLC/repos/local/cache/get-python3_5cfa8c15/mlc-cached-state.json
/home/ubuntu/venv/mlcflow/bin/python3 /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
[2026-05-12 19:08:59,651 customize.py   : 152 INFO ] -                     Detected version: 0.3.25
[2026-05-12 19:08:59,658 script_utils.py:  88 INFO ] -               * mlcr get,python3
[2026-05-12 19:08:59,659 module.py      :1037 INFO ] -                    ! load /home/ubuntu/MLC/repos/local/cache/get-python3_5cfa8c15/mlc-cached-state.json
[2026-05-12 19:08:59,659 module.py      :1037 INFO ] -                    ! load /home/ubuntu/MLC/repos/local/cache/get-generic-python-lib_93ece761/mlc-cached-state.json
make generate_engines RUN_ARGS=' --benchmarks=resnet50 --scenarios=offline  --test_mode=PerformanceOnly  --log_dir=/home/ubuntu/MLC/repos/local/cache/get-mlperf-inference-results-dir_dc6ddadd/valid_results/buildkitsandbox-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/offline/performance/run_1 --use_deque_limit --num_warmups=10 --no_audit_verify  '
[2026-05-12 19:09:02,147 system_list.py:279 ERROR RANK=0] Runtime system not found in built-ins. Please specify a system name using the SYSTEM_NAME environment variable.
[2026-05-12 19:09:02,921 font_manager.py:1639 INFO RANK=0] generated new fontManager
[2026-05-12 19:09:03,030 main.py:800 INFO RANK=0] Detected system ID: UNREGISTERED_aarch64_NVIDIA_GB10x1
[2026-05-12 19:09:03,030 main.py:136 INFO RANK=0] Config file /home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/configs/UNREGISTERED_aarch64_NVIDIA_GB10x1/Offline/resnet50.py not found. Loading minimal configs as default.
[2026-05-12 19:09:03,031 main.py:494 WARNING RANK=0] Config not found for current system. Attempting to load minimal config for resnet50 Offline (cp990)
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:45: SyntaxWarning: invalid escape sequence '\+'
  "res3a_branch2a \+ res3a_branch2a_relu.*": "0x266a23a6dae5e9dd",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:47: SyntaxWarning: invalid escape sequence '\+'
  "res3a_branch2b \+ res3a_branch2b_relu.*": "0x11e97a6f7b62ebde",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:48: SyntaxWarning: invalid escape sequence '\+'
  "res3b_branch2a \+ res3b_branch2a_relu.*": "0x7b6ce4f355d80159",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:49: SyntaxWarning: invalid escape sequence '\+'
  "res3b_branch2b \+ res3b_branch2b_relu.*": "0x11e97a6f7b62ebde",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:50: SyntaxWarning: invalid escape sequence '\+'
  "res3c_branch2a \+ res3c_branch2a_relu.*": "0x7b6ce4f355d80159",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:51: SyntaxWarning: invalid escape sequence '\+'
  "res3c_branch2b \+ res3c_branch2b_relu.*": "0x11e97a6f7b62ebde",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:52: SyntaxWarning: invalid escape sequence '\+'
  "res3d_branch2a \+ res3d_branch2a_relu.*": "0x7b6ce4f355d80159",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:53: SyntaxWarning: invalid escape sequence '\+'
  "res3d_branch2b \+ res3d_branch2b_relu.*": "0x11e97a6f7b62ebde",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:54: SyntaxWarning: invalid escape sequence '\+'
  "res4a_branch2a \+ res4a_branch2a_relu.*": "0x743ccad8b4fb4cdc",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:56: SyntaxWarning: invalid escape sequence '\+'
  "res4a_branch2b \+ res4a_branch2b_relu.*": "0xdaed974d12d937ad",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:57: SyntaxWarning: invalid escape sequence '\+'
  "res4b_branch2a \+ res4b_branch2a_relu.*": "0xd470dbb02604d85e",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:58: SyntaxWarning: invalid escape sequence '\+'
  "res4b_branch2b \+ res4b_branch2b_relu.*": "0x5d63ba6b5cfeb06f",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:59: SyntaxWarning: invalid escape sequence '\+'
  "res4c_branch2a \+ res4c_branch2a_relu.*": "0xd470dbb02604d85e",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:60: SyntaxWarning: invalid escape sequence '\+'
  "res4c_branch2b \+ res4c_branch2b_relu.*": "0x5d63ba6b5cfeb06f",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:61: SyntaxWarning: invalid escape sequence '\+'
  "res4d_branch2a \+ res4d_branch2a_relu.*": "0xd470dbb02604d85e",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:62: SyntaxWarning: invalid escape sequence '\+'
  "res4d_branch2b \+ res4d_branch2b_relu.*": "0x5d63ba6b5cfeb06f",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:63: SyntaxWarning: invalid escape sequence '\+'
  "res4e_branch2a \+ res4e_branch2a_relu.*": "0xd470dbb02604d85e",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:64: SyntaxWarning: invalid escape sequence '\+'
  "res4e_branch2b \+ res4e_branch2b_relu.*": "0x5d63ba6b5cfeb06f",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:65: SyntaxWarning: invalid escape sequence '\+'
  "res4f_branch2a \+ res4f_branch2a_relu.*": "0xd470dbb02604d85e",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:66: SyntaxWarning: invalid escape sequence '\+'
  "res4f_branch2b \+ res4f_branch2b_relu.*": "0x5d63ba6b5cfeb06f",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:68: SyntaxWarning: invalid escape sequence '\+'
  "res5a_branch2a \+ res5a_branch2a_relu.*": "0x8d507197d88f65f7",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:69: SyntaxWarning: invalid escape sequence '\+'
  "res5a_branch2b \+ res5a_branch2b_relu.*": "0xdaed974d12d937ad",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:70: SyntaxWarning: invalid escape sequence '\+'
  "res5b_branch2a \+ res5b_branch2a_relu.*": "0xdaed974d12d937ad",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:71: SyntaxWarning: invalid escape sequence '\+'
  "res5b_branch2b \+ res5b_branch2b_relu.*": "0xdaed974d12d937ad",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:72: SyntaxWarning: invalid escape sequence '\+'
  "res5c_branch2a \+ res5c_branch2a_relu.*": "0xdaed974d12d937ad",
/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/resnet50/tensorrt/builder.py:73: SyntaxWarning: invalid escape sequence '\+'
  "res5c_branch2b \+ res5c_branch2b_relu.*": "0xdaed974d12d937ad",
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/main.py", line 803, in <module>
    runner.run_all()
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/main.py", line 790, in run_all
    self._run_workload(benchmark, scenario)
  File "/home/ubuntu/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_631dad6c/repo/closed/NVIDIA/code/main.py", line 574, in _run_workload
    with power_context:
TypeError: 'ABCMeta' object does not support the context manager protocol
make: *** [Makefile:38: generate_engines] Error 1
[2026-05-12 19:09:03,347 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/ubuntu/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py. 
Error : Native run script failed inside MLC script (name = app-mlperf-inference-nvidia, return code = 256)


[2026-05-12 19:09:03,347 main.py        : 148 ERROR] -   at /home/ubuntu/venv/mlcflow/lib/python3.12/site-packages/mlc/script_action.py:308 in call_script_module_function
[2026-05-12 19:09:03,347 main.py        : 178 ERROR] - Failed script: app-mlperf-inference-nvidia
[2026-05-12 19:09:03,347 main.py        : 179 ERROR] - To rerun just the failed part: mlcr app-mlperf-inference-nvidia --print_env=False --print_deps=False --dump_version_info=True
[2026-05-12 19:09:03,347 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-12 19:09:03,347 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-12 19:09:03,359 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev 7048f0f1a
ubuntu@d2a6818054a4:~$ 
ubuntu@d2a6818054a4:~$ 

[arjunsuresh]

Minimal config should work even without an explicit config - I have added a fix from Claude for power_context failure. You can give a try with mlc pull repo - can do it within the container too and then redoing the same mlcr command but without --docker

[michaelfuckner]

[2026-05-14 17:50:38,557 script_utils.py:  88 INFO ] -         * mlcr get,python3
[2026-05-14 17:50:38,557 module.py      :1037 INFO ] -              ! load /home/molli123/MLC/repos/local/cache/get-python3_09388c6a/mlc-cached-state.json
/home/molli123/mlc/bin/python3 /home/molli123/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
[2026-05-14 17:50:38,594 customize.py   : 152 INFO ] -                 Detected version: 13.2.0
[2026-05-14 17:50:38,598 script_utils.py:  88 INFO ] -           * mlcr get,python3
[2026-05-14 17:50:38,598 module.py      :1037 INFO ] -                ! load /home/molli123/MLC/repos/local/cache/get-python3_09388c6a/mlc-cached-state.json
[2026-05-14 17:50:38,599 module.py      :1037 INFO ] -                ! load /home/molli123/MLC/repos/local/cache/get-generic-python-lib_4677a0f0/mlc-cached-state.json
[2026-05-14 17:50:38,600 script_utils.py:  88 INFO ] -       * mlcr get,nvidia,mitten
[2026-05-14 17:50:38,601 module.py      :1037 INFO ] -            ! load /home/molli123/MLC/repos/local/cache/get-nvidia-mitten_2296c283/mlc-cached-state.json
[2026-05-14 17:50:38,604 script_utils.py:  88 INFO ] -       * mlcr get,cuda,_cudnn
[2026-05-14 17:50:38,605 module.py      :1037 INFO ] -            ! load /home/molli123/MLC/repos/local/cache/get-cuda_b63280d4/mlc-cached-state.json
[2026-05-14 17:50:38,605 module.py      :1924 INFO ] -         * ENV[CUDA_HOME]: /usr/local/cuda
[2026-05-14 17:50:38,605 module.py      :1924 INFO ] -         * ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: yes
[2026-05-14 17:50:38,605 module.py      :1924 INFO ] -         * ENV[MLC_CUDA_VERSION]: 13.0
[2026-05-14 17:50:38,606 module.py      :1924 INFO ] -         * ENV[MLC_CUDA_VERSION_STRING]: cu130
[2026-05-14 17:50:38,606 module.py      :1924 INFO ] -         * ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2026-05-14 17:50:38,607 script_utils.py:  88 INFO ] -       * mlcr get,tensorrt
[2026-05-14 17:50:38,615 script_utils.py:  88 INFO ] -         * mlcr detect,os
[2026-05-14 17:50:38,616 module.py      :1037 INFO ] -              ! load /home/molli123/MLC/repos/local/cache/detect-os_aac747bd/mlc-cached-state.json
[2026-05-14 17:50:38,620 script_utils.py:  88 INFO ] -         * mlcr get,python3
[2026-05-14 17:50:38,621 module.py      :1037 INFO ] -              ! load /home/molli123/MLC/repos/local/cache/get-python3_09388c6a/mlc-cached-state.json
[2026-05-14 17:50:38,632 module.py      :4104 INFO ] -             # Requested paths: :/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu:/usr/local/lib:/lib:/usr/lib
[2026-05-14 17:50:38,647 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/molli123/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py. 
Error : Please envoke mlcr "get,tensorrt,_dev" --tar_file={full path to the TensorRT tar file}
[2026-05-14 17:50:38,649 main.py        : 148 ERROR] -   at /home/molli123/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py:285 in preprocess
[2026-05-14 17:50:38,649 main.py        : 178 ERROR] - Failed script: app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_valid,_r6.0_default,_offline
[2026-05-14 17:50:38,649 main.py        : 179 ERROR] - To rerun just the failed part: mlcr app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_valid,_r6.0_default,_offline --automation=script --print_env=False --print_deps=False --dump_version_info=True
[2026-05-14 17:50:38,649 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.
[2026-05-14 17:50:38,649 main.py        : 191 ERROR] - mlcflow 1.2.1
[2026-05-14 17:50:38,653 main.py        : 196 ERROR] -   mlcommons@mlperf-automations: dev d5be4725d
(mlc) molli123@spark-7994:~$